/hydrus/ - Hydrus Network

Archive for bug reports, feature requests, and other discussion for the hydrus network.

Index Catalog Archive Bottom Refresh
Name
Options
Subject
Message

Max message length: 12000

files

Max file size: 32.00 MB

Total max file size: 50.00 MB

Max files: 5

Supported file types: GIF, JPG, PNG, WebM, OGG, and more

E-mail
Password

(used to delete files and posts)

Misc

Remember to follow the Rules

The backup domains are located at 8chan.se and 8chan.cc. TOR access can be found here, or you can access the TOR portal from the clearnet at Redchannit 3.0.

Uncommon Time Winter Stream

Interboard /christmas/ Event has Begun!
Come celebrate Christmas with us here


8chan.moe is a hobby project with no affiliation whatsoever to the administration of any other "8chan" site, past or present.

(4.11 KB 300x100 simplebanner.png)

Hydrus Network General #5 Anonymous Board volunteer 09/14/2022 (Wed) 22:07:14 No. 18264
This is a thread for releases, bug reports, and other discussion for the hydrus network software. The hydrus network client is an application written for Anon and other internet-fluent media nerds who have large image/swf/webm collections. It browses with tags instead of folders, a little like a booru on your desktop. Advanced users can share tags and files anonymously through custom servers that any user may run. Everything is free, privacy is the first concern, and the source code is included with the release. Releases are available for Windows, Linux, and macOS. I am the hydrus developer. I am continually working on the software and try to put out a new release every Wednesday by 8pm EST. Past hydrus imageboard discussion, and these generals as they hit the post limit, are being archived at >>>/hydrus/ . If you would like to learn more, please check out the extensive help and getting started guide here: https://hydrusnetwork.github.io/hydrus/
(1.61 MB 550x550 77c0.gif)

>>18930 >It is Hydrus's 11th birthday today! Happy Birthday and thanks for this awesome program.
Here's a thought. Parents that are conditional upon multiple children. Say I have a pictures from a certain manga. I'd want the creator of the manga tagged only on images that he actually drew from the manga, not on fanart of the characters. It'd be useful to have a way that the parent tag creator:X applies only if both child tags series:Y and origin:source material are present. This way you could automatically tag all media taken from it's source material with the artist that made it using the tag "source material". I'm sure there's other examples where multi-conditional parents could be used too. No idea how this would mesh with how parents are currently displayed and managed though.
>added an option 'mouse wheel can "scroll" through menu buttons' to options->gui Just noticed this Thank you so much, that's been annoying me for years
>>18933 Having to switch on my IME, type whatever I want, select the correct option from the list, and hit enter for each Japanese tag was too much of a pain for me so I just set up siblings to turn the Japanese tags I care about into their English counterparts.
Hi, Is there a downloader for Deviantart (in Hydrus). I see one for Deviantart artist lookup, but not one for tag. If not, would it be possible to change the Artist one to tag?
>>18887 Thanks, I'm glad you like it! >>18889 >>18890 As a side thing here, I've been talking with a guy recently who has good evidence to show that CF are also checking which ciphersuite you used to clear the captcha (i.e. the original permitted range your client declares when TLS is negotiated), at least in the stricted protection modes. I'm going to investigate this more, as if I can plug into this better, it may make our CF-solver, cloudscraper, work better. In any case, if we see the normal Hydrus Companion solution (i.e. copy cookies and User-Agent) not work, it may be this third variable that also needs to line up. >>18894 If you find out, and you think it is appropriate, let me know and I'll roll it into the defaults for future users in your situation. >>18895 Yeah, check the 'file import options' of the importer, under 'import options' button somewhere on its UI. I don't like how awkward this is to set up right now, since you only have the option of the defaults (set under options->importing) or a custom set you have to set up every time, so I'll be moving this to a favourites system (multiple named defaults) in time.
>>18896 I'm afraid I don't know anything about their implementation, so you may need to ask them. I haven't heard of this problem from anyone else yet, either. Do you have an error 'traceback' you can post, or any other information? Does it just do nothing? >>18897 >Why do subscriptions and gallery pages download things in reverse orders from each other? Gallery downloaders 'get everything', subscriptions 'catch up with the new'. They use different logic when they decide what to add, and the way that naturally figured itself out meant that I ended up reversing the parse-order when I wrote the 'add import objects to the log' routine for subscriptions. EDIT: I wrote a whole thing describing the differences in technical behaviour, but it just sounded muddy. The basic thing to answer your question is that it is nice to download in a clean specific order, either new-to-old or old-to-new (and not doing a zig-zag or jumping place when new pages are added), and for downloaders it is difficult to get old-to-new conveniently. >>18898 >>18912 For your EXIF, I know some users have had luck with some automatic EXIF export programs, but I have no experience myself with this. I know some programs--although I don't know their names--can export to JSON files, which you can now import to hydrus with the improved sidecars system. (this was one of the reasons I added it) >>18899 If you hit services->review service->all local files, you can clear the deleted files record, but that clears it for every file you ever deleted. You want just for a selection that you just deleted. I'm not sure if we have a nice way to do that yet. If the files are in your trash still, then undelete them, turn on the 'advanced file deletion dialog' under options->files and trash, and then re-delete them and pick the option to physically delete with no deletion record. If they aren't in trash, I'm sure this will come up again, and we now have nice ways to search deleted files, so I'll make a new menu option. If you aren't in help->advanced mode, turn it on and click your 'my files' button on a normal search page. You should have 'multiple locations', and in advanced mode that lets you specifically search deleted files. I'll add a thumbnail menu or something to say 'clear deletion record'.
>>18900 A 'hard replace' siblings will likely come in future. The original way I wrote this system tried to do both at the same time. I tried to satisfy people who wanted hard replace, and those who wanted clean undo, and of course I failed at both. I think last year or the year before I moved it all to a completely 'virtualised' system, which is why there is the storage/display dichotomy. We still want to do some hard replace for other classes of 'sibling' and some one-time jobs, but I am ruminating on it all. Behind the scenes, the math for this stuff gets complex fast. Namespace siblings (your ip->series) are something I've wanted to do for a long time, but it'll have to wait for the next significant expansion of the system. The algebra is hellish enough atm, and expanding it again gives me shivers. >>18901 No, but I want to make it easier! Same for >>18902 , a lot of the UI here has to accomodate technical reality, so when I have the timeβ„’, I will dig back in here and asynchronise and cache everything so it loads quick and I can add some 'quick-add' interstitials and all that. >>18908 Damn, wow. Ok, I'll think about this situation. You are right, I absolutely do not handle multiple hashes well atm. I still have to add options for the logic, too. >>18923 If you can find these tiff files, I'd be interested in seeing them. Either email or point them to me. Overall this isn't a huge worry though, I think. They might not be corrupt in the sense you have a hard drive problem. These are just 'warning' level problems, which are generally not super serious, and many programs that make files either make invalid headers by accident, or they make headers so advanced that PIL (python image library) doesn't quite understand them. The truncated file read is more worrying. Unfortunately I do not control those warnings and it isn't trivial for me to figure out which file caused them without fucking up other stuff. I'd say don't worry about it too much though, but if you only have like three tiff files, you may be able to figure out which is/are the bad ones, and I'd be interested in checking them out.
>>18924 >>18926 Yeah, hydrus-dd is good. I know several users working on related tech, especially since training models recently got easier, and I hope the next year or two lets me add some new project links to the Client API help page. >>18927 >>18928 We have several incidences of this now. We want to pull retroactive notes, URLs, modified times, and tags. It is all the more important that I write a proper slow 'in the background' maintenance routine that does a drip-feed retroactive fetch of all this data. Please hack the solutions you like in the meantime for the most important files, and I'll figure out a proper solution in time. >>18929 Thank you for this report. I just went through it in debug mode and I don't know what the hell is going on. I'll check it and see if I can fix it. Ideally it sorts nicely. This system is legacy and difficult to update. It is another thing put on the back-burner that is planned to be completely overhauled. Eventually, it will leapfrog onto the normal download system, and it'll get all the tags the normal gelbooru/whatever downloader gets. >>18931 Thank you for this feedback. I don't like how clunky the taglist is, so this is useful. I'll copy it all down and try and action it early in the new year. Let me know how it works for you as I roll it out.
>>18933 >>18937 Sorry for the trouble. If the 'IME mode' as in >>10977 doesn't help, I know one Linux user just couldn't use their IME and we couldn't figure it out other than 'Qt python weirdness'. iirc, we were hoping Qt 6 would help, but if you are still getting it, perhaps not. If your IME works on a 'normal' Qt widget in hydrus, like options->gui->application name, but doesn't work in my custom autocomplete tag lookup, then that's likely my fault. If it works in neither, then it is something about Qt, python, or PyInstaller. Let me know if it is my fault! If it seems to be Qt/python/pyinstaller, then you might like to try running from source. That brings hydrus one step closer to your system .so files and fixes a bunch of weird stuff like this. https://hydrusnetwork.github.io/hydrus/running_from_source.html Let me know how you get on. I want nice IME and unicode support, but some of this is outside of my hands/expertise. >>18938 No, but this is an interesting idea. I don't want to go crazy with specific options, but I'll have a think what I could do. >>18939 Yes. >>18940 πŸŽ‰πŸŽ‰πŸŽ‰ >>18932 >>18934 >>18941 Sorry I am running short on time for the day, but just a quick thing that siblings and parents are a logical and mathematical nightmare even in their relatively simple state. I thought they would be simple, and we'd be moving on to more complicated relationships and tag algebra (and I am keen on the basic idea, or I was), but instead the whole thing spun into an endless vista of counting and display bugs and human disagreement. Just as a general attitude, I would be very very careful about ever implementing more advanced tag relationship algebra since I realise I am not clever enough to foresee the consequences. I'm open to doing it as suggestions though, so human eyes can figure out the truth of it. Just not iron rules. >>18944 Unfortunately I had to remove it fairly recently! DA changed how their site works, more phone friendly, and the 'tag search' they now provide is now mostly a borked 'popular-first' search. I think gallery-dl dropped support too. Maybe a clever download maker can figure out a way to resurrect the old new-to-old tag ordering, but I failed.
>>18947 >and those who wanted clean undo, and of course I failed at both I think you succeeded in the latter and find great merit in soft tagging with clean undoability. My real issue is ultimately with presentation and the longstanding inability to "edit" a tag while maintaining it's relationships, and the latter seems to be a very unusual way to use the program that is only needed if you're autistic like me and don't like typo siblings and other bad tag siblings remaining in the database. The underlying system is good and I was wrong earlier. >>18948 >Let me know how it works for you as I roll it out. I'll be ready and waiting. Thank you for your time. >>18949 >Yes. Awesome.
I've been using Hydrus for like a year and a half, now, and only recently made the mistake of exiting out of the main gallery tab. I'd mostly been ripping boorus with the artist tag and navigating it that way, never really using the search functions. Is there a way I can have Hydrus bring up a tab with all of the query:artist tags listed as they were before?
tried to commit to the ptr, got this mess v510, win32, source ServerException 500: The server's error text was too long to display. The first part follows, while a larger chunk has been written to the log. Traceback (most recent call last): File "D:\hydrus\hydrus\core\HydrusThreading.py", line 401, in run callable( *args, **kwargs ) File "D:\hydrus\hydrus\client\gui\ClientGUI.py", line 326, in THREADUploadPending service.Request( HC.POST, 'update', { 'client_to_server_update' : client_to_server_update } ) File "D:\hydrus\hydrus\client\ClientServices.py", line 1270, in Request network_job.WaitUntilDone() File "D:\hydrus\hydrus\client\networking\ClientNetworkingJobs.py", line 1973, in WaitUntilDone raise self._error_exception hydrus.core.HydrusExceptions.ServerException: 500: The server's error text was too long to display. The first part follows, while a larger chunk has been written to the log. The repository encountered an error it could not handle! Here is a dump of what happened, which will also be written to your client.log file. If it persists, please forward it to hydrus.admin@gmail.com: Traceback (most recent call last): File "/usr/lib/ The repository encountered an error it could not handle! Here is a dump of what happened, which will also be written to your client.log file. If it persists, please forward it to hydrus.admin@gmail.com: Traceback (most recent call last): File "/usr/lib/
>>18896 >>18946 Having a similar issue with Lolisnatcher. A change to the hydrus client api appears to have broken things on the 3rd party app side. I've tested both AnimeBoxes and Lolisnatcher with new client api setups. Both fail in different ways. Animeboxes simply can no longer tell the api exists. It's either not getting a response from hydrus or it can't interpret it. Lolisnatcher seems to be only partly broken, but still enough that it's unusuable currently. Lolisnatcher can see hydrus, and can even request a new client api from in the app. That all works fine. But it fails it's network test when trying to finish the setup in app. Most likely it's the same issue as AnimeBoxes and Hydrus is either not responding or sending a response Lolisnatcher doesn't understand. Unfortunately I don't have any kind of logs to provide. If Hydrus is logging these client api interactions somewhere let me know where and I can send it over.
>>18953 They finally fixed the issue with Lolisnatcher a few days ago but haven't released a new build yet
>>18946 >for downloaders it is difficult to get old-to-new conveniently Okay. Well then in that case just a simple reverse order button in the context menu or something like that for the file url log will work fine for now. Thanks!
>>18949 >If your IME works on a 'normal' Qt widget in hydrus, like options->gui->application name, but doesn't work in my custom autocomplete tag lookup, then that's likely my fault. If it works in neither, then it is something about Qt, python, or PyInstaller. Doesn't work in that box either, so I guess it's out of your hands. >I want nice IME and unicode support Incidentally, emojis also don't display. Some, like 😭, display as the monochrome equivalents, while others like πŸ† are just white boxes. It doesn't matter much since I don't plan on using them as normal tags, but there are a few cases where artists use them in usernames or something. On the bright side, at least the Compose key does work
I am setting up shortcuts for the first time and am having trouble. I want a shortcut that toggles the favorites rating while I am in the archive/delete filter page (this is explicitly described in the help page but I can't figure it out). I was also hoping to make a shortcut that opens the archive/delete filter for the selected files while I am at main window. Can someone walk me through this? Thanks and happy birthday to hydrus!
>>18952 Sorry for the trouble--should be fixed now! I messed something up and we had to patch the server to a v510a hotfix. Let me know if you have any more trouble.
>>18956 >emojis also don't display That might be a font issue. I've had to deal with monochrome emoji and squares when setting up terminal emulators, it was a font issue then. My guess is that whatever font Hydrus chose to use doesn't have proper emoji support, so you get lots of squares and colorless emoji.
>>18951 You can try Pages / Sessions / Append / Last Session. There are also some other options under sessions, like Append Session Backup / Last Session / *last ten sessions are so listed here.
Hi, small question about duplicates: I mark a lot of pictures as "same quality". What is the simplest way to delete all dups, so only one of "same quality" pictures stays?
Looking for an anon that has the images saved from ourobooru. There's a new site to dump the art into now: https://ourobooru.art/
I noticed that when I make a gallery search for a certain artist on pixiv, it shows around 800 files in total, but when I made a subscription query, it only showed about 200 and those 200 seem to be the most recent 200 posts by that creator. It looks like this is messing up the results of the quality info button because it's not counting all the files from that subscription that got cut off. Is there a way to stop that from happening so the good percentage can stay accurate? I tried resetting the query but after it ran to completion it went back to around 200 again, like it's getting cut off somehow. I then did it again, and this time, it worked and the whole queries files are accounted for in the subscription. I also noticed that it did something like this for 1 other artist before, also on pixiv. It doesn't seem like it's happening for most of the queries though, but it's hard to tell.
>>18963 The query that I reset to get the full number of files back just ran again for a few seconds and seemed to do nothing, but then I checked it and now it's again back down to around 200. That makes me think that whatever is happening is something intentional that the subscription is doing.
>>18957 >I want a shortcut that toggles the favorites rating file > shortcuts. open the section "media actions, either thumbnails or the viewer", click "add", then in the command section change the "simple command" dropdown to "like/dislike command". then you can choose whether to set or swap the rating or whatever. >while I am in the archive/delete filter page do you want it to only work while you are in the archive/delete filter? you can't do that. the shortcut will work in the normal viewer, the archive/delete viewer, or on selected thumbnails. >a shortcut that opens the archive/delete filter for the selected files while I am at main window file > shortcuts. in the section "media actions, either thumbnails or the viewer", click "add", then in the command section in the dropdown beneath "simple command", choose "launch the archive/delete filter". if you don't have any files selected, it will launch the filter for every file in the current page. >>18961 open a new file search page. in the dropdown, click "system:file relationships". click the box at the top that says "system:IS NOT the best quality file of its duplicate group". this will show you all your files that are same quality duplicates without the main "king" of their duplicate group. if there's a lot of files, it may be a good idea to make a backup first in case you make a mistake. then you can delete them. protip: next time you're in the duplicate filter, instead of selecting "they are the same quality" select "this is better, delete the other". then you won't have to do this! proprotip: enable the advanced file deletion dialog in options so that when you delete a bunch of files from the main gallery, the you from the future will know why. advanced note: when you tell hydrus that multiple files are "they are the same quality", hydrus arbitrarily selects one of them as being the "king" of the group. this "king" is the one that appears if you search "system:IS the best quality file of its duplicate group". so "they are the same quality" can perhaps more accurately be understood as telling hydrus "consider these files as interchangeable". read more about this here. https://hydrusnetwork.github.io/hydrus/duplicates.html#duplicates_advanced
I had a good week. I fixed some bugs, added some new import and tag search options for advanced users, and made thumbnails look crisp at high UI scale. The release should be as normal tomorrow. We are bumplocked, so I will try and hustle tomorrow, catch up on the thread, and make a new global for the release post.
>>18966 You're superhuman. I don't know how you do it. Not every anon posting in the thread is necessarily giving good feedback, you know?
(783.27 KB 2030x1689 rtg81.png)

>>18967 >I don't know how you do it. Simple and unadulterated 100% anon's autism.
(960.28 KB 1926x1898 5262.png)

>>18967 >good feedback KEK Welcome to this /board/ newfag. Anons don't need any patting. What they do is mostly for the lulz and eventually to smash the very foundations of the normies' fake reality.
>>18965 Thank you so much anon that did the trick
Is there a way to search for files by their hydrus file id (the internal one that counts up, not sha-256) and sort pages by that id too?
>>18962 I'm uploading the files to LetsUpload, the mass booru uploader doesn't seem to work for me. I'll post the link when it's done, may take a while.
>>18953 >>18954 Sorry for the trouble here, it sounds like I caused some of this. I've been doing some API reshaping recently--basically I made some dumb decisions when I first made the API, and now cleaning it up and obsoleting the old way. I also messed up how I did that obsolescence in a couple cases, doing it too fast with too little warning. The main change has been in how I present tags and services. I'm moving towards the 'service_key' being the primary API id for services, and tags are now showing both storage and display contexts. I'm trying hard to move to data structures that we can expand in future without breaking anything else, so I'm hoping there won't be much more of this, and we'll have a lot more backwards compatibility going forward. I have a tight work loop and regularly make a lot of changes, which is fine when I am only devving my own stuff, but I need to be more careful with the systems used by third parties like the API. I need to do one more change in the new year, on the '/get_services' call, and I hope that'll mostly be it. If it helps, you can have the Client API log its requests under services->manage services. It gets spammy in normal usage, but is useful for quick debugging. I don't know the animeboxes guy(s), but I am happy to help if they or you have any technical questions. >>18956 >>18959 Yeah, I think the emoji stuff is based on the font your OS supplies. There may be additional Qt issues there (I remember back in the wx days, there was different emoji support depending on whether I was drawing in software or hardware, so I bet Qt has some similar issues), but afaik Qt says 'hey, how do I show this unicode?', and your OS reponds with some kind of svg instructions. πŸ† shows ok here on Win 10, as does ethiopic α•α‹˜αŒŽ and cuniform π’Š•, with I'm pretty sure default fonts. My guess is this support will generally increase as OSes roll out newer font tech generally. There may be a way to improve clever font support by specifically selecting a different font in QSS, but what to actually choose is beyond my expertise. As for IME, that sucks, sorry! I'll keep my eyes open in case there are flags I can ever set to make this work better, but ideally Qt just updates one day and it all starts working. Actually, now I think of it, if you decide to run from source and PySide6 doesn't do IME, maybe PyQt6 will. They talk to the C++ layer in different ways, despite doing ostensibly the same job. Let me know if you discover anything!
>>18963 >>18964 Yeah, I'm afraid there are two things going on here: - Subscription queries 'compact' themselves down to 250 items as long as the 251st item is 'done' and old (twice the death velocity age). This is because subscriptions are a slightly weird bloaty object in order to be smarter and I have to keep them lean so they load and save fast. - Pixiv have an unusual API that gives hundreds of results in one request. The hydrus downloader engine has logic tuned for boorus, where you'll get 20-75 per page. I have some hacky pixiv-specific hardcoded subscription logic that takes the first 200 from a giganto-page like that in order to avoid other logic getting confused about how every new subscription check suddenly found 800 new files. This isn't helped by the fact that pixiv post pages can have multiple files, which frustrates some of my recheck logic. The 'quality info button' functionality is limited by the 200/250 file compaction. I think the real answer here is for me to write a proper 'this file was downloaded by x sub' metadata type and consult that, rather than the sub query's semi-ephemeral file log. I'm also thinking about expanding the statistics I offer and generalising the search domain to any file search. Then you'd be able to look up how 'good' a creator: tag was, or system:was downloaded by x sub. >>18967 I'm just a normal Anon in the lucky position to put in the work, but I appreciate the praise. I'm a little older, been around these parts since the start, which I think has helped callous my soul a little, but I'm also grateful that the hydrus feedback has been typically very kind, given how we often go about things. I try not to bullshit people or drop drama bombs, and I really do care about and want to add tools for the community, which I think has helped in that regard. I'll also say there's a certain momentum to it--you start a routine and before you know it, it feels normal. This is just what I do now. Just keep on pushing that boulder up the hill. >>18971 Not right now. I was quite opposed to this for a while, but the Client API guys convinced me to expose file ids there, so I've been drip-feeding it to the UI too. I expect you've seen you can copy the file id from the thumbnail menu if you are in advanced mode. I could add a hash_id type to 'system:hash', I expect. Can you talk about why you'd like this feature, so I can understand the workflow you are aiming for better? The file_id/hash_id is pretty obscure as a specific value and not as simple as a 'this file was imported before this one' measure, so it often confuses when people actually start working with it.
New thread here >>>/t/11075 I'll post v511 to it later today, and this thread should be migrated to /hydrus/ soon. Thanks everyone!


Forms
Delete
Report
Quick Reply