/hydrus/ - Hydrus Network

Archive for bug reports, feature requests, and other discussion for the hydrus network.

Index Catalog Archive Bottom Refresh
Name
Options
Subject
Message

Max message length: 12000

files

Max file size: 32.00 MB

Total max file size: 50.00 MB

Max files: 5

Supported file types: GIF, JPG, PNG, WebM, OGG, and more

E-mail
Password

(used to delete files and posts)

Misc

Remember to follow the Rules

The backup domains are located at 8chan.se and 8chan.cc. TOR access can be found here, or you can access the TOR portal from the clearnet at Redchannit 3.0.

Uncommon Time Winter Stream

Interboard /christmas/ Event has Begun!
Come celebrate Christmas with us here


8chan.moe is a hobby project with no affiliation whatsoever to the administration of any other "8chan" site, past or present.

(4.11 KB 300x100 simplebanner.png)

Hydrus Network General #9 Anonymous Board volunteer 01/03/2024 (Wed) 19:10:11 No. 21127
This is a thread for releases, bug reports, and other discussion for the hydrus network software. The hydrus network client is an application written for Anon and other internet-fluent media nerds who have large image/swf/webm collections. It browses with tags instead of folders, a little like a booru on your desktop. Users can choose to download and share tags through a Public Tag Repository that now has more than 2 billion tag mappings, and advanced users may set up their own repositories just for themselves and friends. Everything is free and privacy is the first concern. Releases are available for Windows, Linux, and macOS, and it is now easy to run the program straight from source. I am the hydrus developer. I am continually working on the software and try to put out a new release every Wednesday by 8pm EST. Past hydrus imageboard discussion, and these generals as they hit the post limit, are being archived at >>>/hydrus/ . Hydrus is a powerful and complicated program, and it is not for everyone. If you would like to learn more, please check out the extensive help and getting started guide here: https://hydrusnetwork.github.io/hydrus/ Previous thread >>>/hydrus/20352
Edited last time by hydrus_dev on 01/20/2024 (Sat) 18:36:21.
>>22209 zoomerspeak for internet janitor. It's hip with underageB&s to add "oid" to the end of things.
>>22159 >Hydrus checks if a double click is occuring ONLY WHILE CTRL is pressed? Yeah I think this is probably the solution. I should update the logic to detect a ctrl+double-click and infer what happened and ensure the result is a select event no matter what the first click did. I'll have a think. I ultimately want to make it user-customisable through the normal shortcut system, so perhaps a bunch of hardcoded rules is a bad idea. >>22160 >>22172 Ah, numpy is the problem? Interesting. numpy used to be a real pain years ago since it didn't distribute a pre-compiled 'wheel', and trying to compile this giant math library locally would often fail with random -1 errors after ten minutes, but these days it tends to be ok. It is still a dense C++ library though, and I bet it uses a lot of CPU optimisations that can conflict with a slightly odd setup. If you activate your venv and do this: pip show numpy (this will spam a ton of licence bullshit, but scroll up to see version) python import numpy what happens? Does it segfault there? My guess is your pip was able to get a partial numpy somehow, but it is conflicting with something in your local OS for whatever crazy environment reason. EDIT: Aha, I remember that numpy released a 2.0 recently and it fucked a bunch of stuff up! Hydrus enforces v1.26.4 at the latest, but if you somehow have 2.0, that could be your problem here. If Fedora 40 uses Python 3.13 (new) or something, I wonder if the allowed versions are all whack and my requirements.txt file is failing. You can try to install other versions with this line (with your venv activated): pip install numpy==1.25.2 As per the versions here: https://pypi.org/project/numpy/#history Maybe try an older version? Let me know what you discover! >>22161 >>22165 >>22174 I'm not enough of a Linux expert to know why, but sometimes when I call xdg to ask the OS to do the 'default' open action, there's something in the hydrus environment that fucks the call up. You can try to do an 'open externally' with help->debug->report modes->subprocess report mode on, and we'll see a bunch of your env stuff, but I'm not sure what the solution would be, unless it gives a nice error message or it is otherwise obvious to you. Sorry for the trouble, but specifying the specific launch path in that options dialog is usually the safe fallback. >>22162 Thanks, I'll check this!
>>22167 I am sorry to say I have made the technical decision not to support it! tl;dr is that I want tags to be for searching, not describing, and upper case would not be worth the programming complexity involved in adding it, but some fuller thoughts are here: https://hydrusnetwork.github.io/hydrus/faq.html#tags File notes support upper case, but there are a little too long for filenames, and there aren't nice ways to convert them to a filename on export (yet). I've long wondered/wanted a middle-ground, tbh, in which the 'title:' tags would live along with filenames, which would be a 'label' or 'name' for a file outside of tags (which are for searching) and notes (which are a paragraph), but my thoughts aren't complete. We want to generally get away from filenames in hydrus, but they are still fun and useful at times. >>22170 >>22171 Thanks. I did not have time for this last week, but I will try again this week, and check pixiv. >>22173 The logic can get complicated, but in general if: - A URL is mapped to more than one file. - A file has more than one URL in the same domain. Then hydrus will not trust the given URL-file mapping to deliver an 'already in db' result and will proceed with downloading the html or json from the 'Post' URL. It sounds like hydrus is doing this, in your case, and then seeing that the actual direct image URL is indeed different and downloading it. It seems like hydrus is concluding that the URL mappings here are mixed up or duplicated and so isn't trusting the 'already in db' result to be true (which, since it is then downloading a duplicate, appears to be technically true). I don't know enough about VK and this album/feed/wall distinction to talk cleverly about it--is the Post URL getting duplicated somehow? How is it the same and yet delivering two different Image URLs? Does the 'wall' post also link to the original 'photo' post? (that's what your screenshot seems to suggest) If the wall post points to the photo post, then I think it is too late, since at this stage hydrus has already fetched the html/json and it sees a new direct Image URL and is going to go ahead with the download. I don't think there is a way to trick it into ignoring the new URL atm. If 'wall' posts always point to a higher quality image on the 'photo' post, then I think the downloader should, instead of downloading the smaller 'wall' post and attaching the photo post as a source URL, redirect to the 'photo' post instead. Let me know if I have misunderstood something or explained myself badly. >>22176 Thanks, I'll check it out! >>22177 Strange! When I unload a video, I basically say 'hey, mpv, please load this black square .png file' and then hide the video panel. My first guess here is that your libmpv.so has a bug or other certain behaviour where if the buffer has already rendered the end of the file, it doesn't move to the black png until the buffer is clear. This may be related to me hiding the mpv due to my own technical bullshit. I can't promise an excellent solution here since this is a tricky area of the program and very platform dependant, but I am planning to add some DEBUG options to the overall mpv pipeline, stuff that is probably going to cause crashes but which may fix some odd situations like yours. I don't know when this will happen, but I do want to do it, so please wait for then and let me know how it works for you. I don't know enough about Linux (or Arch) but does your apt-get or AUR or whatever have a nice way to roll back to an older libmpv? The newer python-mpv should be able to talk to libmpv1.so going back years, so if you can locate the file, you should be able to just slot it in place and it'll all work. e.g. for Windows, I can just grab any of these dlls and drop them in the install dir and they load: https://sourceforge.net/projects/mpv-player-windows/files/libmpv/ I know your package manager wants to handle things, but can you force a rewind in any way, rather than having to build your own older version (I'm no expert, but I do know how pain in the ass this would be).
Is there any way to move namespace values for a number of files to another namespace? I have a bunch of files with the 'filename' namespace (all with different values, naturally), but I want to migrate those namespace values to another namespace, 'title', for just those files. Is that possible?
>>22179 >>22191 No worries, and thanks for the feedback. Actual feedback on real-world workflows is always useful. Glad it is working better now. >>22180 Glad you like the program! >>22181 How are you generating a random filename? I half-remember adding this somewhere but cannot find it now. Btw just as a side thing, if you are worryied about fingerprinting, the hydrus sha256 filenames are the same for all hydrus users (they are based only on file byte content, no salt or anything), so the info that leaks, if someone recognises your filenames, is that you are a hydrus user (or an 8chan user, it uses the same filenames), not that you are a particular user. >>22184 Great idea, I will add this to the options. Can you also check out help->debug->profiling? This wasn't originally my code, but I think if you turn that mode on and do some test searches, I think the profiling will catch it. We can see what the hell is taking so long. >>22185 Although I don't like the idea, if you are desperate and just want to paste some ephemeral example of a 28MB png or something to discord, you should have the option to copy a 1024x1024 bitmap under 'share->copy source lookup bitmap'. >>22192 >>22193 >>22194 >>22195 I didn't know myself, either (I didn't write this code originally), but yeah it does a background right-click on the current page and then harvests that menu for actions to fire. So, if you have a selection (maybe by ctrl+a or something), you can now type 'my files' and you'll get 'from my files (delete)'. On any empty selection you might be able to typing 'in' for 'inbox (select/remove)' and such. A bit dangerous, and basically prototype, which is why it was behind advanced mode before. Let me know if and how you use it, and what if anything should be filtered out. Sounds like the first step is to label it better in the UI. >>22196 There is no good solution for this right, now, I'm sorry to say. In future we'll have clients dialing into each other via the Client API and this will be possible, but for now you have to go with a jank solution to do this sort of thing. Best answer at the moment, I think, is to have something like freefilesync sync the database out of the central network store to a local location with a local hydrus install, run the client there, and then commit back if you make changes. >>22200 Pic related in one way or another, it seems. Many such cases.
This thread is finally anchoring, so I'll aim to make a new one for next Wednesday's release. >>22214 Not yet, but we want this tech. We've wanted it for a long time as 'namespace siblings' or similar, but I think the math will just be too complicated. I'll probably just write a new panel that hard-replaces the tags manually. We want this for the PTR where we are faced with several hundred thousand artist:->creator: siblings.
>>22213 >We want to generally get away from filenames in hydrus, but they are still fun and useful at times. Yup. The point of preserving file names rises to the top when exporting files comes up, then they become a must.
(254.96 KB 811x690 twilight - yes.jpg)

>>22214 >Is there any way to move namespace values for a number of files to another namespace? >>22216 >Not yet, but we want this tech. A highly needed and expected feature.
>>22212 >You can try to do an 'open externally' with help->debug->report modes->subprocess report mode on, and we'll see a bunch of your env stuff, but I'm not sure what the solution would be, unless it gives a nice error message or it is otherwise obvious to you. No worries, thanks for the debug tip: I got lost looking for one but now I should have a thread to unravel. Will let you know if I solve it!
>>22213 >mpv unloading, libmpv Arch, being a terminal updooter (or bleeding edge if you want to be accurate) distribution intentionally doesn't have any intended way to downgrade libraries. Worse, libmpv.so is bundled together with mpv, and there don't appear to be any legacy mpv repositories, so it's compiling unsupported past versions from source if I want to try those, which is highly painful. However, I think I found the problem. Disabling the "af=lavfi=[loudnorm=I=-16:TP=-3:LRA=4]" mpv.conf option solves the issue entirely. It may be a mainline mpv bug where pulling in ffmpeg for the audio filter chain messes up subprocess handling? Or maybe it's because the loudnorm filter upsamples everything to 192kHz which might demand a longer buffer? The only other plausible culprit would be the mpv autoload script which automatically adds all playable files in a directory to a playlist, but adding load-scripts=no to mpv.conf didn't really do anything.
>>22218 More than that, I want some way to operate on all tags in a view using some script. Like a Hydrus database sed util. Something to convert all the unnamespaced underscored tags into whitespace tags. Even just a full regex (\1 capture group substitution included) would do the job.
>>22219 (me) >>22161 (me) Solved, environment issue (not a bug). I was trying to figure out which version it affected but it turns out it was only affecting source builds. Reinstalled Python requirements and it looks like some were missing, opening in external programs now works fine. I probably uninstalled the dependencies by accident when trying to regain disk space after some other Python tool tried to download 10GB of libraries.
Enter something in a tag search to see suggestions. Right-click one or more suggestions. The "search" submenu does not contain an exclude option.
>>22223 Actually, you can just enter the search with a "-", and press Enter on the suggestion or selected suggestions.
>>22223 You can add it to your search and then immediately exclude it by ctrl+double clicking it or a selection of tags in your search. I never right click a tag unless I want a quick glance at parents and children.
(10.33 KB Nord.qss.txt)

I've set up a Nord theme stylesheet, feedback welcome. I've noticed almost all stylesheets don't handle disabled widgets (e.g. menu 'undo' button if nothing to undo) and push buttons (many in the downloaders dialog), which can be annoying.
Dumb feature request, something to click to count up orgasms on a given image/file.
>>22216 >This thread is finally anchoring I was here!
>>22227 The "rating" window feels superfluous with just the is favorite or not star, honestly. It shouldn't be hard to add an arbitrary user counter/int to it.
>>22227 Already possible: Go to Go to services->manage services->add->local inc/dec ratings https://hydrusnetwork.github.io/hydrus/getting_started_ratings.html#incdec Then when you open an image, click on the rating to add +1.
>>22173 >>22213 I think I've fixed that, but I changed several settings, no idea which one(s) helped. I ticked off the "neighbour spam" checking, increased the priority of post urls to 100, and meddled with some other settings in url classes.
>>22230 Oh, neat! Thanks!
In the tag view dialog, is there any way to only display the tags you are petitioning to be added/removed/whatever to the PTR? It sure would help against copypasted trash if I could actually review what I'm pasting in a timely manner.
>>22230 The fabled 'nut counter'.
So, I have a tiny problem, where I've imported a bunch of pictures with improper creation/mod times. I've changed them to match the correct times, but when importing those, hydrus seems to prefer the initial import times. Is there any way to temporarily change that preference, or an import option that specifies this. I tried looking for it but couldn't find anything. I could just delete and reimport but figured it was worth asking first. Thanks :D
I had a good, simple week. I added some commands to the Client API and cleared some bug fixes and UI improvements. An issue that was causing redundant file downloads on Pixiv is fixed.
Is there any way to re-run the metadata merge process that happens when you assign duplicates/alternates, on a selection of files with those relationships already set? For example say I have a set of 10 images that are alternates. I then import one image form a booru that is a duplicate of one of those 10 images. This duplicate has a bunch of new tags that get added to that one image once I tell hydrus it's a duplicate. I want those tags to propagate to the rest of the alternates according to the rules I have set up for alternates. As far as I know it only happens when you actually set the alternate relationships. So okay I could do that manually again, but... Now if I could select 1000 files with perhaps 100 sets of alternates and rightclick and select merge metadata between alternates and have this done for all of them... that would be useful.
>>22235 >I've changed them to match the correct times Bit hard to follow where you changed the times (Operating System or within Hydrus), but also not that important. It seems you cannot 'reimport' files that are 'already in db' to update their times without deleting them first. probably you have to not leave a deletion record also. BUT you can just change the import/modify times by right-clicking all the files you want to change -> manage -> times. there you can change the modify time on the top and the import times for the file domains in the lower box. it works also for several files at once with cascading steps, but make sure to be on the latest version. if that is any help for you. Also one thing to notice: hydrus doesnt seem to handle creation time that you see in Windows, which makes sense, because at the time of import, the file gets copied to your database and creates a creation time that would be identical to the import time pretty much, whereas modify times in windows can persist over many locations/devices and so on.
Accidentally deleted this from my client. Pulled it out of the trash bin and reimported it. This results in a failure to import with the import log citing the file is previously deleted. Normally, I can then open this deleted file in a new page to see a default Hydrus thumbnail with a red background from which I can clear the deletion record and then successfully reimport the file. This time however, when I opened the deleted file in a new page, it was still there, thumbnail and fullsize file. The duplicate processor won't detect it despite there being an alternate. I have one other file displaying this anomaly. How do I remedy this weird behavior? Additionally, is there a way to make Hydrus clear the deletion record of previously deleted files when manually importing files should any come up? It's a little tedious to restore them, and now I'm running into this weird behavior too. I'm on Hydrus 583.
>>22239 Read the following paragraph from Hydev and the according quotes. It could have to do with caches. Maybe you need to restart the client. >>22095 >I'm a little loose with some of my caching tech, so I think what's happening here is that the 'image cache', which basically stores bitmaps of rendered files, doesn't get told the file was deleted >Additionally, is there a way to make Hydrus clear the deletion record of previously deleted files when manually importing files should any come up? It's a little tedious to restore them, and now I'm running into this weird behavior too. I'm on Hydrus 583. Maybe this helps: Import a file that has a deletion record -> press 'file import options (default)' button directly above the green 'import now' button -> press button on next little window that opens and change to 'set custom file import options just for this importer' -> deactivate the 'exclude previously deleted files' checkbox -> apply -> import now That way you don't need to go the additional steps with logs and deleting deletion records manually in new pages. It will import everything previously deleted for this import.
>>22240 Thanks. I wasn't sure if there was an option. I should have looked harder. You were also on the money about the caching issue.
>>22222 Great, thanks for letting me know! >>22220 Thanks, this is useful. I'll add this to the help text around here as a debug thing to try. >>22221 Yeah this is the dream. My model is something advanced and powerful but clean like tags->migrate tags. We'll see how it goes. Otherwise, Client API is the best solution for now. >>22223 Thanks, I'll add it! >>22226 Thanks, I'll check this out! >>22233 Not at the moment, but this is a good idea. I'm going to make a push on 'deleted tags' right now, which you can view in an awkward way with the cog icon there. There should be a better way to filter all the visible states. >>22235 >Is there any way to temporarily change that preference, or an import option that specifies this. I am not sure if this will work (I wrote it if the modified time was missing, so I don't know off the top of my head if it will overwrite an existing value), but try right-clicking a thumbnail and going manage->maintenance->regenerate file modified time. I'm not sure if deleting (and re-importing) will clear a previously recorded modified time. Sounds like we could do with a 'clear all these times' button somewhere. >>22239 Just as another option, when you see a file import come out as previously deleted, you should be able to open the 'file log' and I think you'll be able to right-click on the 'previously deleted' rows and say something like 'try again and override deleted', or you just hit 'try again' and it pops up a dialog saying 'hey you want to clear deleted records before you try again?', I forget exactly but it should be simple, and then all the updates happen behind the scenes for you and it'll just work. Sorry for the weird caching thing, let me know if it keeps happening.
New thread here >>>/t/15721 I'll post v584 to it later today, and this thread should be migrated to >>>/hydrus/ soon. Thanks everyone!
>>22210 Thanks for this. I've found two files on the btrfs with the most errors that are only partly readable, and then Input/output error happens. So it reports errors for smaller parts. There is an option that sounds like it could help recover some files before ditching the filesystem, but it seems dumb. [quote] --init-csum-tree create a new checksum tree and recalculate checksums in all files WARNING: Do not blindly use this option to fix checksum mismatch problems. [/quote]
How do I set up auth in sankaku beta from the user repo? I'm fine with repasting api keys every 48 hours if it works, but the ones I grab from browser don't work at all and return error 401 I know sankaku support is officially dead and I've been using parsers posted in this thread that require constant babying and switching between hydrus and browser just to download files one by one, but recently sankaku updated their tags layout and I messed up my setup trying to fix it.
>>22245 Never mind, got it to work. The token first appeared in POST request, not in GET.


Forms
Delete
Report
Quick Reply