I had a mixed week. I was a bit short on work time, but there are some bug fixes, including some more logic/label fixes in my new date calculation code, and quality of life improvements, such as your recent petition 'reasons' now being remembered in the manage tags/siblings/parents dialogs and easier column-size reset for multi-column-lists. The release should be as normal tomorrow.

https://www.youtube.com/watch?v=2_fxNRtmyAE windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v526/Hydrus.Network.526.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v526/Hydrus.Network.526.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v526/Hydrus.Network.526.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v526/Hydrus.Network.526.-.Linux.-.Executable.tar.gz I had a mixed week. I was a little short on time, so this is a simple bugfix and quality of life release. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html highlights When you petition in the manage tags/siblings/parents panel on a repository, it typically asks you for a reason for the change. There are some fixed reasons, but now the system will remember the last five custom reasons you enter too. I hope this saves some typing/pasting work for people with clever petitions to make. You can change the number of recent reasons saved (including setting to 0 to disable) under options->tags. Every multi-column list in the program now has a 'reset the column widths for this type of list' entry in their header right-click menus. It also resizes immediately (an old debug routine that did this for all lists needed a restart). If your thread watcher list goes crazy wide, it should be fixable now! Let me know how it goes. I added the basic media keys you find on many keyboards to the shortcut system--previous, next, play/pause, volume up, volume down, and mute. On Windows, these keys seem to have a global hook that fires regardless of how I capture it too, so they may not be so useful yet, but feel free to play around with them. I fixed a stupid logical typo in my new date calculating code that was breaking system time predicates with the same number of months as the current month (like "system:modified time before 2 years 5 months ago" in May). Sorry for the trouble! next week There is going to be an important update next week! Last week, I tried to update OpenCV for everyone, but it caused a dll conflict in the updated build and I quickly rolled it back in a hotfix. We are going to try again next week, which will require 'extracting' users to perform a clean install. Since we are making that change, I have decided to also roll in a long-planned renaming of the program executables. 'client' will become 'hydrus client' and 'server' 'hydrus server'. This part will require updating of custom shortcuts and launch scripts. I will reiterate all this with specific instructions for all platforms in next week's release, and I'll drop reminders for several weeks after. Next week is also a 'medium size' job week. If I have time and energy, I'd really like to see if I can get the Qt media player working as an mpv fallback.

Is it possible, at this stage, to have a directory namespace be half constructed from the directory above it? For example, I'm importing files from a directory structure that has the following: mastodon/<instance name>/username, with several different instances each with their own folder. The instance name is the first directory namespace, and I'd like the next one down, the username tag, to have the instance as a part of the tag, like <instance> id: <username>. Is there any way to do this?

Does each file domain each have a deleted file service attached to it? So I could set a watcher for nsfw art to ignore files only deleted from the nsfw watcher while the sfw watcher can still download files deleted from nsfw but also ignore files deleted by sfw watchers? In case one file is better for one domain than the other.

Would it be possible to maintain the same progress on a video file when swapping between the preview window and full view?

>>19646 Checked. +1

I was away from my PC for a few days and when I came back Hydrus had crashed, checking the logs, it seems like Hydrus crashed due to the internet going out for a time I assume this is a bug? v525, 2023/05/04 17:00:37: Exception: v525, 2023/05/04 17:00:37: ConnectionException: Could not connect! socket.gaierror: [Errno -3] Temporary failure in name resolution During handling of the above exception, another exception occurred: urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fb6f70ad570>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution During handling of the above exception, another exception occurred: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='a.4cdn.org', port=443): Max retries exceeded with url: /v/thread/635742337.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb6f70ad570>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) During handling of the above exception, another exception occurred: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='a.4cdn.org', port=443): Max retries exceeded with url: /v/thread/635742337.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb6f70ad570>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) During handling of the above exception, another exception occurred: hydrus.core.HydrusExceptions.ConnectionException: Could not connect! This is the absolute last error, but there were about 5 hours of continuous errors.

Sorry, couldn't post earlier! >>19644 Not super easily, I don't think. The regex area of the advanced path tagging dialog can't do clever grouping and substitution yet. I am planning to overhaul that whole system and integrate the 'String Processing' system from the parsing engine into it, and that should let you do much cleverer stuff. For now, I think your best bet is to figure out something with sidecars. If you can export the desired tags to some .txt files, you could set that up to import automatically with an Import Folder. https://hydrusnetwork.github.io/hydrus/advanced_sidecars.html >>19645 Each file service has its own deleted file record, which is searchable, but if I understand you right, you are talking about the 'exclude previously deleted files' check in importers' 'file import options'. This check only filters out files that have been physically deleted or are currently in trash (technically, this is the deleted file store of the 'all my files' domain). Extending file import options to allow for cleverer filtering sounds like a good idea though, and something to consider in future. Maybe even something like the normal file search interface. You could say 'dis-allow anything that matches these search predicates' and have much more flexibility than the current fixed 'min/max size' stuff. >>19646 Yeah I want this. I actually have some of the tech ready, since I support starting a vid at x% in, although for weird videos scanning can be imprecise. I'm going to write a simple object that holds the play status and yeah, pass it back and forth between the search page and media viewer. Such a system could also support bookmarking too, if we wanted to go that way.

>>19648 Hmm, normally hydrus is fine with network stuff going out. And, when it can't connect to something x times within y minutes, it is supposed to set a 'this domain sucks right now' status on your erroring a.4cdn.org and it'll not try again until that y has run out. Can you double check your settings for this under options->connection? For the five hours, was it taking ten minute breaks or was it going full bore over and over? Maybe watchers, which are always checking and re-checking, don't work so great with this tech and need some different handling. Unfortunately, in almost all cases, crashing happens in UI code, usually me touching things from the wrong place or the wrong order. Now it could be that all these network errors were stacking up 6,000 error popups or something mad, and that caused some UI events to get stuck or delayed too long, but I am not very confident, and I'd normally suspect something else. Have you had many crashes before? Also unfortunately, because of the way python works, it won't log any info about the actual crash itself (as it happens), and getting crashdumps from the OS is a real technical pain. Do you know any more about the nature of this error? that initial socket line is odd--was your DNS down or something similar? It looks like these things eventually resolved as HydrusExceptions.ConnectionException, so this stuff is supposed to be handled 'correctly', but it looks like something more complicated than just a down internet connection was happening here--maaaaybe that somehow overloaded some OS buffer. (some OSes like macOS treat network connections as file handles and get pissed if you try to have more than 256 open at once etc...)

>>19649 >I'm going to write a simple object that holds the play status and yeah, pass it back and forth between the search page and media viewer. Such a system could also support bookmarking too, if we wanted to go that way. This! Video bookmarks sound like my wish coming true. Jumping among Tabs to watch different videos at different time indexes would be great.

>>19649 Yeah, I tested importing a file to the nsfw file domain, then deleted it from the client and also physically deleted the imported file. Then I tried importing it again but this time to the sfw domain and with exclude deleted files checked. It didn't import as it seems exclude deleted files doesn't just check the import domain's deleted file list, but all domains deleted file lists.

I had a good week. There are two things happening tomorrow, and nothing else: there's a prototype new media player that I would like some people to test, and there are long-delayed library and build updates that will require users to update in a special way. Users who extract will be making a ‘clean install’, and almost all users will be updating their shortcuts to point to the new executable, which is renamed from 'client' to 'hydrus_client'. I'll walk everyone through what they need to do in the post. The release should be as normal tomorrow.

Is downloading from atf via hydrus working for anyone else or have I messed up by accidently exceeding their bandwidth? The regular site still works for me.

>>19650 >connection options Yeah I'm in advanced mode so they're the really low settings, sidenote, could we get a way to set the timeouts to the suggested normal mode even in advanced mode? I also might have lowered the 4chan timer for some really fast threads a while ago. I'll attach the pictures. >have you had many crashes before Not too many, I've had a couple from "Unexpected end of file" errors usually from internet dropping when downloading a file, I can actually see one in the logs from when the internet dropped. I sync files to my desktop to auto-import into hydrus, the sync was interrupted before the file finished I guess. >nature of this error The ISP had an outage, so all network wasn't working, I use my router as my DNS so that should have still been up. I do also see a segfault in the Python library libshiboken2 at around the same time, is that something Hydrus uses? I shouldn't have to contend with resource limits, I use Linux and I've set limits.conf to an extraordinarily high number.

https://www.youtube.com/shorts/xK5SJ6CuezY windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v527/Hydrus.Network.527.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v527/Hydrus.Network.527.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v527/Hydrus.Network.527.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v527/Hydrus.Network.527.-.Linux.-.Executable.tar.gz I had a good week working on important updates to the build and a new media player prototype. This build has special update instructions! All users who update from 526 or earlier to 527 or later will have to do this once! Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html build update A couple of weeks ago, I tried updating OpenCV (our image library) for everyone, but it caused a dll conflict and the program wouldn't boot. I am trying again this week, and since we have to do this change, I am also rolling in a long-planned change to the main exectuable names. 'client' is now 'hydrus_client' and 'server' is now 'hydrus_server'. Some users will have to do a 'clean install', and most will have to update their shortcuts or launch scripts. As always--but particularly this week--please make a backup before you update. The following instructions are simple, but if anything goes wrong, you can always roll back to a safe backup. Ok, the instructions, by situation, are: - If you use the Windows installer, update as normal. It will do the clean install for you. Your start menu 'hydrus client' shortcut should also be overwritten with one to the new executable, but if you use a custom shortcut, you will need to update that yourself. - If you use one of the normal extract builds, do a clean install. You also need to update any program shortcuts. - If you use the macOS app, there are no special instructions. Update as normal. - If you run from source, git pull as normal. If you haven't already, feel free to run setup_venv again to get the new OpenCV. Update any custom launch scripts to point at the new hydrus_client.py file. A 'clean install' is as here: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs You are basically going to be deleting everything except your database (to clear out old dll files) and then updating as normal. To update your shortcut or launch script, simply edit it, and where it says 'install_dir\client.exe', just rename it to 'install_dir\hydrus_client.exe'. For Linux users, that'll be 'install_dir/client'. If you have a custom firewall rule for the Client API, you'll need to update it for the new executable name too. And that's it! Let me know if you have any trouble. QMediaPlayer This is for advanced users only. Also only users who run from source and use Qt6! I integrated Qt's built-in media player this week. I hope that in future it may be a fallback for users who cannot run mpv. Unfortunately, while it worked surprisingly well when it worked, it was also very crashy. I've got it stable when running from source, but it seems to insta-crash when in a build, so that's something to figure out later. If you are an advanced user who runs Qt6 from source, you will now see the new experimental Qt Media Player as a selectable viewer under options->media for audio, animations, and video. It has a number of limitations (check the changelog for the full list), but it does work in most cases. Have a play with it and let me know how you get on. next week This week was a lot of work for what felt like little outcome. I was disappointed that QMediaPlayer crashed in the build. In any case, I'm short on getting small things done, so I'll return to that. Just some little fixes and quality of life. I am taking my vacation week on the 8-14th of June, so I have four releases left before then. I won't go for anything huge, but try, if I can, to tie up some loose ends.

>>19656 I jumped from 523 to 527, and now my rightclick menu is hueg.

>>19656 >>19657 Actually, it seems like text in a lot of places is bigger.

>>19656 Qubes anon here, decided to try the Qt viewer for both preview and media viewer for the hell of it. Here's my experience, in an unorganized mess of thoughts. It's more "just werks" than mpv on Qubes, but not perfect. Just turn it on and it works. It takes a half-second or so to show the media in general, whereas mpv load times were unnoticeable. It also seems to be less efficient than mpv, some ridiculously hi-res files I keep for archival that had around 10 fps with mpv have 1~2 fps with Qt. Regular videos also made my fans run harder than with mpv. GIFs don't work, they get to the second frame and stop. Like you mentioned, it really doesn't want anything to be on top of it. It covers the file info at the top and the tags popout in the media viewer as well. No crashes so far, but I have gotten two freezes. Both involved changing files being previewed quickly; one with all videos and waiting under a second between switches, and another with very quickly changing from previews of images to a video. I think that some files don't work when they are the first to be previewed (unselecting a file "resets" the preview in this case). Other files work fine, but some specific files consistently don't. If the first file opened in the media viewer is a video, it always blacks out and plays the first second of audio. Changing files fixes this until the media viewer is closed. My log has a few lines saying `Failed to start video surface due to main thread blocked.` but there are fewer of these lines than there were videos failing to play. I got this error a few times (the number changes): Trying to dispose element qgstvideorenderersink109, but it is in READY instead of the NULL state. You need to explicitly set elements to the NULL state before dropping the final reference, to allow them to clean up. This problem may also be caused by a refcounting bug in the application or some element. Also it seems that the issues I had with mpv no longer working as of v525 are gone, I can use it as I used to now after doing the ritual.

>>19652 Yep, sorry, I just don't think I have this tech yet. >>19655 Thanks, this is interesting. shiboken is a helper library for Qt, so yeah this sounds as expected like an UI thing, but perhaps it is, say, an emergency network error trying to report itself in the UI and I am not handling it correctly. I'd really like to have some good 'reset this page to defaults' button in the options dialog, but the backlog of work to get this done has held me up. For now, here's the defaults of that page, pic related. EDIT: I think I misunderstood, your pictures seem to show the default settings, both pages. >>19657 >>19658 Hmm, I rolled in a new Qt version in 526, I wonder if it is doing font UI scaling better. What's your UI scale--150%? I don't control this stuff in general--Qt is supposed to do it automatically according to your OS settings, and I know they have been improving support in recent versions. >>19659 Thank you very much, this is useful. I regret this isn't really ready for prime time, but I think and hope that it will get better with new versions of Qt. I think I know what might be going on with that READY/NULL error, so I'll see what I can do there, and I'm sure some other related code cleanup I am planning for the media viewer will help some of the jank here (I'm amazed it even worked in some ways, given how my media canvas layout system works).

Does anyone here rank his files beyond a simple "favorites" on/off rating? I'm considering adding something like a 5 star ranking system but wondering if there's a point. I guess it could help find files I don't really like, could delete them to save some space.

>>19661 Yeah I do 11 point (0..10) rating. I use them in searches all the time so it's definitely worth it for me. One of the things I try to do is make sure that files I like more are properly tagged before files I like less, so I use the ratings to prioritize maintenance. In fact, what I really wish Hydrus had was something like a 0 to 1 rating system, where the ratings were decimal numbers, so that ratings could be arbitrarily precise. If that's too niche of an idea, then just having 0 to 100 would be cool. 0 to 10 worked for a while, but after so many files, there's many with the same rating that I don't really like the same.

I just updated from v523 to v527 about an hour ago. I'm on Fedora, which uses Python 3.11, so when I was rerunning the venv setup, I used the test opencv. It looks like everything is working fine, and Hydrus seems to work on Python 3.11 now. Why is the test version of opencv called that? Is it an experimental branch or something?

>>19660 >What's your UI scale 125%. I can change it, but that fucks up everything outside of Hydrus.

Anyone know how to view full messages? I have a message about some network context that is cut off and can not find any way to view the entire thing.

>update hydrus >have to fix the pixiv downloader so it doesn't add 1 to every page count again dev please

>>19664 Same for me. My Windows 10 UI scaling is at 125% because it's comfier for me. Since the qt6 upgrade, hydrus has become incredibly big, too, which makes some windows almost inaccessible.

Anyone working on Hydrus-dd? Is there any way to change the model to something better? The program itself is good, it's just the dd model isn't very good. I would love to be able to change it over to one of Smilingwolf's models, as is shown here. - https://huggingface.co/spaces/SmilingWolf/wd-v1-4-tags DD is especially bad as it doesn't seem to be able to detect loli. I've fed it several pics that were extreme loli, and it didn't detect one of them. Also, it's pretty basic on tags.

>>19668 A couple of us asked the developer about 3 months ago, and he said he might try it, but we haven't heard anything since.

Minor complaint: Numpad delete key (i.e. the "." with Num Lock off) doesn't delete files. Only normal Delete does. I use a left-side numpad and I've gotten in the habit of generally using that delete instead of the normal one. That's pretty niche though.

After updating to 527, I get the following. v527, 2023/05/16 11:23:48: hydrus client started v527, 2023/05/16 11:23:49: hydrus client failed v527, 2023/05/16 11:23:49: Traceback (most recent call last): File "/home/anon/Pictures/hydrus/hydrus/hydrus_client_boot.py", line 244, in boot from hydrus.client import ClientController File "/home/anon/Pictures/hydrus/hydrus/client/ClientController.py", line 43, in <module> from hydrus.client.gui import ClientGUI File "/home/anon/Pictures/hydrus/hydrus/client/gui/ClientGUI.py", line 92, in <module> from hydrus.client.gui.pages import ClientGUIManagement File "/home/anon/Pictures/hydrus/hydrus/client/gui/pages/ClientGUIManagement.py", line 39, in <module> from hydrus.client.gui.canvas import ClientGUICanvas File "/home/anon/Pictures/hydrus/hydrus/client/gui/canvas/ClientGUICanvas.py", line 42, in <module> from hydrus.client.gui.canvas import ClientGUICanvasMedia File "/home/anon/Pictures/hydrus/hydrus/client/gui/canvas/ClientGUICanvasMedia.py", line 7, in <module> from qtpy import QtMultimediaWidgets as QMW File "/home/anon/anaconda3/lib/python3.7/site-packages/qtpy/QtMultimediaWidgets.py", line 13, in <module> from PyQt5.QtMultimediaWidgets import * ImportError: /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2: undefined symbol: k5_sname_compare, version krb5_3_MIT

>>19671 This was with Python 3.7.16 Using Python 3.11.2 instead, it works.

>>19665 If it's in a file log, you right click the entry and click "copy note". >>19670 options > shortcuts you'll want the "media actions, either thumbnails or the viewer" section. note that "delete (on numpad)" and ". (on numpad)" are considered separate keys. >>19661 at first, i didn't really want to do a multi-point ranking system because what i like depends on my mood and i would feel like i would have to justify every rating by comparing it to every other rating to ensure the system is accurate. so i just used a simple "favorites". but then i realized my favorites is almost 2500 files so i made a new "super favorites" rating service... i'm just headed towards a multi-point ranking system in the end, aren't i?

I recently updated from 523 to 526, and I've noticed that opening a new page (whether middle-clicking a tag or using the new page picker or whatever) takes significantly longer. Has this happened for anyone else?

>>19674 Noticing the same as you on 527, I think. I don't know how fast it was before, but I can tell that searching a tag on an existing page with system limit:1000 takes a very short time, maybe 1.5 seconds or less. Middle clicking the tag to run the same search in a new page takes 2-3 more seconds. However making a page by right clicking the page ribbon and selecting new page is instant. So much so that it's faster to open a new page this way and then run the a search than it is to middle click a tag.

I had an ok week that unfortunately had less work time than I had hoped. There's some quality of life improvements with snappier search updating/cancelling and more flexible system predicate parsing. The macOS App is also fixed. The release should be as normal tomorrow.

It's a new thread, so I'll modify and repost some of my feature requests. Show all parents in tag manager You can show all parents already with "expand parent rows", but it's messy looking and parent tags are duplicated if multiple children with one parent are present. Modify either one of the experimental dropdown options in the tag manager, "switch to 'mulitple media views' tag display" or "switch to 'display tags' tag display" so it is a permanent toggle, rather than reverting upon closing the tag manager. I do not know what the difference between these two options are. This already displays all parent tags that are applied only through their children as regular tags in the tag manager, but lacks permanence. Add decorators to the parent tags denoting they are only present as a parent tag if they are not already hard tagged on a file. archive thumbnails When a .zip, .rar, .7z, etc. file contains images, the first image should be used as a thumbnail. This makes browsing a collection of manga/comics or hentai much more feasible as once you narrow down the search pool to a reasonable amount you can still browse the files by eye and decide to look at something based on often untaggable art styles or visual memory. This can be simulated by turning an archive into a .png archive with the cover page as the png image, but that's of course quite tedious and impractical. Grouping namespaces By color in tag manager You've said this is on the back burner. Hopefully it can be done one day, as it will vastly improve the presentation of tags in the tag manager and increase the speed of viewing them.

>>19671 >>19672 Sorry for the trouble. I've wrapped this in some basic error handling for today's release. It shouldn't moan again. I'll brush it up later and have it say in help->about whether it imported ok. >>19674 >>19675 I have a couple of reports like this. I regret it--it seems the Qt 6.3.1 to 6.4.1 jump messed with some people, and these guys above >>19667 >>19664 with the UI scaling mess. I walked one guy through getting a source install working and rolling back to 6.3.1 and it relieved his mega-long session load (which I'm pretty sure was because each page had the 2-second lag), so now I have to think about what to do. The 6.4.1 update has useful updates in, so I want to keep it for the most users who have no problem with it, but I want to figure you guys out too. I'm interested in what versions of Windows (or whatever) you are running. I'm not sure what the common theme is here yet. If you feel brave, please check out running your own source version of hydrus, as here: https://hydrusnetwork.github.io/hydrus/running_from_source.html I recommend just doing it on your desktop to start, not your real install. Once it is up and it boots, go shift+right-click on the folder to open the install folder in your Windows terminal, then if you are in powershell type 'cmd' to take you to the old terminal, and then type: open_venv.bat pip install PySide6==6.3.1 Then try running it again. Ideally, that fixes you. If you like, you can migrate your real client to the same setup, and you'll just stay on 6.3.1 for a bit. You might like also to try 6.5.0, which is the next version I've been testing. Maybe they fixed your regression there?--in which case, I could hurry the next update.

>>19678 I'm on wangblows 10 as well as the other guy, and while the text is larger, it's not a huge issue. I prefer to wait for 6.5.0 and see if that fixes it.

https://www.youtube.com/watch?v=I2lgy4_Quko windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v528/Hydrus.Network.528.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v528/Hydrus.Network.528.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v528/Hydrus.Network.528.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v528/Hydrus.Network.528.-.Linux.-.Executable.tar.gz NOTICE! Version 527 had special update instructions. If you are updating from 526 or earlier, please consult the post here: https://github.com/hydrusnetwork/hydrus/releases/tag/v527 I had an ok week. A bit short on work time, but there's some simple quality of life and bug fixes. The macOS App is also fixed. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html highlights If you regularly use a custom slideshow time, you can now edit the quick-select times shown in the media viewer menu under options->media. It is hacky comma-separated numbers, like the zoom intervals, but have a fiddle and you'll get it. I improved a bunch of the file search cancel tech. If you put in a search with 'system:imported since 6 months ago', and you then update it before it finishes, it should cancel that ongoing search and start the new one very quickly. This _should_ also vastly improve Client API browsing latency for these situations, since the cancel tech is supposed to fire when a search request is dropped, but let me know how it works IRL. The time-based system predicate parsing is also more flexible. You can type 'since' or 'before' rather than having to figure out which '<' or '>' you mean, and putting a colon in or 'ago' on the end should parse more reliably. My ideal is to get the text that a system predicate copies to clipboard perfectly parseable. I'd also, fairly soon, like to plug system predicate parsing into the normal file search autocomplete so more users can play with this. The macOS App had a couple problems last week, with the client->hydrus_client change, and it wasn't booting correctly. Thanks to the users who helped figure this out. Please just install as normal, you still have no special update instructions for the v527 step (it'll just work now!). next week Mostly more simple work like this. I'm toying with the idea of a 'paste file/bitmap onto the client, it then does a similar-files search without importing the file' process, basically a 'sauce?' command from your clipboard, which might actually be real easy to add, but we'll see.

>select multiple alternate versions of a file >right click, manage times >change modified time to what I want, apply >only one file has its modified time changed >have to manually change times of each file Could the ability to change multiple files' times at once be added?

>>19681 What is modified time even supposed to mean. I'm not sure how I'm supposed to interpret those. Is it supposed to be the time the file was created?

>>19682 File created, last modified, and time imported. He's editing just the second of the three. Hydev recently added support for editing these to times before computers existed so people can manually set their creation date to the time when the art was actually made, like say, a photo of the Mona Lisa. I don't think it's a great idea, since you're mixing metadata with non-metadata, and much prefer myself to just use "time period:" tags.

>>19682 I use it like "time created", yeah. Setting lots of alternates as having the same modified time also makes them show up together when sorting by modified time. In this case >>19681 I had imported some files from disk with correct modification times, but a few had previously been imported from a downloader and had the wrong times (the time they were posted to the site I downloaded them from, a few months after it was actually made).

I'm running into an issue where I'll have a tag that I'm typing, but it's 1 character longer than a tag that got siblinged to another tag, so before I'm finished typing it, the suggestion box autocompletes to the ideal sibling, and then when I type the final letter, instead of staying at the first result, which would've been the tag I typed, it jumps down the list to where the ideal sibling from before is, and keeps that highlighted, so when I press enter, it enters that tag instead of the one I want. >pic related

I made a url class for 8chan file pages, since it didn't already have one. The original reason I made it was simply because not having one kinda bugged me a bit, but somehow it looks like adding it also made it so that url downloaders properly grab the correct file instead of the 8chan placeholder image. I don't know how that fixed it, but it's working now with this, which is great!

I've been using yt-dlp to download videos from youtube as well as a few other websites and then putting them in a directory to import later into hydrus, and I've been thinking: do you have any plans to integrate a feature that would allow hydrus to delegate the actual file-downloading portion of a downloader to some external program, so that you could have something like yt-dlp download files, but hydrus will still be able to do things like add tags or other metadata from the post page, and associate known urls and things like that? I think that would be a smart way to leverage all the tools that already exist to download stuff so that it doesn't have to all be NIH-ed into hydrus manually. although since yt-dlp is written in python like hydrus, a simpler solution if you don't like that one could be to just integrate yt-dlp directly into hydrus.

>>19686 Bless you, anon.

>>19674 >>19675 >>19679 Hey, I have confirmed with a running-from-source user that this problem is caused by Qt 6.4.1. 6.3.1 and 6.5.0 do not have it, so I'd like to step us up to 6.5.0 as soon as reasonable. That said, I have made a decision to do these updates more sensibly. There are enough hydrus users now that shit like this should be done more careful, so I'm going to start a new branch on the github that does the same build but with my test library versions, as a beta. This will let more users test things before I roll them into the master and provide more options for people who need them but can't run from source. I'd love to have it out for this week's release. It'll have Python 3.10 (up from 3.9) and this Qt 6.5.0 update. Please try it out, and fingers crossed it'll fix your problem. >>19682 Just as an aside here, I mostly based this on your file system modified time. The cool thing about that is it will normally pass from one computer to another. The 'creation time' may mean 'this file was created on this disk 5 minutes ago', and will update if you move it to a new disk, but the modified time will persist, and for read-only media like we handle it'll stay as a nice 'this was "finished" in 2015-09-03' kind of thing. It isn't perfect as a creation time, but it pushes in that direction (especially since I now aggregate modified time from post times on boorus that hydrus sees, showing you the earliest it has seen for a particular file). >>19681 Yeah it sucks. The current state is really just a proof of concept of me having finished every damn content update path for every timestamp, and that's neat, but it isn't good to use yet. I added the copy/paste buttons to try and help here, but I really need UI support for multiple files. Unfortunately this is going to be a bigger job than it sounds, since I need to deal with some pain in the ass logic, but there we go. I've been flirting with the idea of a 'stepped/cascade' timestamp set too, so you can force a set of files to have timestamp n, n+1, n+2, etc... in order to force a certain import order, but maybe I should just get my finger out and get a simple multi-file dialog started. Oh, another thing--I still have to add proper full timestamp editing to the Client API. This'll be another way for users to do clever stuff without having to wait on me. >>19685 Ah, thank you, this is a neat bug. There's an override for that behaviour, I think, if it helps, under options->tags, the 'by default, select the first tag...'. I think I disable that selection-hopping behaviour when that is selected. This is an interesting thing, in that that ideal sibling must have un-ideals of both 'closed eye' exactly and something like '1 closed eyes'. In the former case, I can make the positive match on what you typed exactly, and in the latter, it isn't exactly what you typed, but it still matches what is selected, so the selection is sticky. I will look at my logic and see if I can recognise and stop this. >>19687 Yeah, search the old thread >>>/hydrus/18976 for 'exe manager' and 'executable manager'. The dream is to let you plug in any exe to hydrus, and as long as it takes simple parameters and spits out a file, we can import it and/or convert it and/or tag-analyse it and/or any other lego-brick pipeline we can think of.

I add tag relationships very often, and one thing that's a bit annoying about that is that the window for managing relationships when I click to add a new one to a tag loads slowly. It takes around 5 to 10 seconds for it to open. Do you think you could add some way to make it not load everything up or whatever makes it take so long, and just let you add some parents or siblings then apply and that's it? Like for when you don't wanna actually manage the existing relationships, just add new ones?

Oh, and another unrelated suggestion. It'd be cool if there was a way to do a sort of mass tag addition on a search, but in a way that's treated as an "automatic" or "downloader" addition, such that it won't re-add tags that are already marked as deleted.

Hello. I just realized that the version of Hydrus Companion I'm using is really old, so I tried to update it but it looks like it's not working in Firefox anymore because the xpi files are unsigned. Does anyone have a solution for this, or do I have to stop using Firefox or Hydrus Companion? I don't know what to do. I tried using Librewolf briefly since that lets you use unsigned extensions, but no matter what settings I changed, too many websites just refused to work properly on Librewolf. I like Firefox and I don't wanna have to switch to something like ungoogled-chromium, but Hydrus Companion is too useful to give up :(

When I do duplicate processing, I generally use it for things I haven't tagged yet that I just put in the inbox, because I don't want to waste time tagging duplicates. However when I find a duplicate and have the duplicate processor delete it, it automatically archives the superior version regardless of whether the deleted version was archived or in the inbox. Is there a way to make it only do this if a duplicate is replacing an already archived file? For now I've just been jumping to the latest archived files and putting them back into my inbox until they're tagged proper.

When you eventually introduce a way to have proper ordered collections, I think a great idea would be to have an option so that files can't be deleted when they're marked as being part of 1 or more collections, so that you don't accidentally delete files and make the collections incomplete now. It could work in such a way that you have to remove the files from any collections they're in, and then it lets you delete the files. I know that implementing that feature hasn't even been started yet, but maybe that could be written as a todo or supplemental addition when you do finally get to implementing it. Just an idea.

I'm running into an issue with file relationships. I have a file that I deleted in the duplicate filter. I wanted to see its duplicate group and the best file in that group, but for some reason, it won't show me the options to see the other files in the duplicate group. I know that the file has one, because it gives me the option to mark it as the best file in the group, but it won't actually let me see the group. Is it because it's no longer part of "my files" or something? I just wanted to see what the best version was.

After I run a search and have X number of files show up, is there anyway to filter this down to all files that have alternates with them grouped by their alternates? Or to then open a new page containing all the sets of alternates that have at least one of their files in the initial search, regardless of whether they had the tag searched? It's very cumbersome and inefficient to discover sets of alternate files by going >Right click >manage >file relationships >view alternates OR there are no alternates for every file. So much so I feel like I might have to start tagging things as "has alternates" for future reference, but that would require finding all the alternate relationships I've ever applied. I'd also like to suggest moving the "view alternates" dropdown option from being under "file relationships" to directly under "open".

According to Hydrus I have 201,362 files, but when I check the properties of my hydrus files folder it says there's 216,525 files. How do I go about figuring out what these extra files are? The only thing I can think of is that Hydrus forgot a bunch of files when I repaired my database last year, but I don't know if that's possible.

Is there anyway I can save pages? I'd like close some pages that have a bunch of search terms already and run the same search again later without having to memorize or manually save all the tags and search predicates I used.

>>19698 The star next to the search box lets you save searches.

>>19699 Sweet, thanks.

>>19697 > I check the properties of my hydrus files folder it says there's 216,525 files I would apply the brute force method. 1- Re-import the 216,525 files 2- From the log dialog select those files NOT already in the database, then right click on them and select "Open in new Tab" to tag them.

>>19701 It's still importing (probably going to take the rest of the day) but 15,117 ended up being unsupported file types. I'm guessing these won't show up in the file log?

>>19693 on the duplicates page, click "edit default duplicate metadata merge options" and then "...for 'this is better'". it sounds like you have "sync archived status?" set to "always archive both" instead of "if one is archived, archive the other". >>19696 >is there anyway to filter this down to all files that have alternates on the search dropdown, click "system:file relationships" and you can search for files with greater than 0 alternates. >with them grouped by their alternates? Or to then open a new page containing all the sets of alternates that have at least one of their files in the initial search, regardless of whether they had the tag searched? i wish. i could go on a whole tangent about how hydrus should allow you to turn a group of images into a single "gallery item" where instead of each picture having tags, the entire gallery shares tags. and any search that matches at least one of the files brings up all of them as one thumbnail in the main window. basically combine boorus with a gallery like e-hentai. one "item" can be a single image or a gallery of images. >I'd also like to suggest moving the "view alternates" dropdown option from being under "file relationships" to directly under "open". oh, that's a good idea. and then add that option to shortcuts so you could just press a key to open all the alternates.

>>19699 Is there a way to make a favorite search that appends to the current search instead of replacing it?

>>19703 >it sounds like you have "sync archived status?" set to "always archive both" instead of "if one is archived, archive the other". >system:file relationships Thanks.

I had a good week. There's some misc small work, a new build for advanced users to test out, and neat similar-files 'get source' tech that looks up files quickly from your clipboard, without having to import them. The release should be as normal tomorrow.

What determines a files modified time exactly? If I have a file with a modified time, but then a downloader sees the same exactly file later from some other source, and the other source reports an earlier source time for the file, does the file now get that earlier source time as its modified time? What about if the source reports a later source time? Does Hydrus go with whichever one is earliest?

Is there any good booru software I can feed my hydrus into that supports random file order? I still use a modified Hyve because as far as I'm aware nothing else does.

>>19702 >but 15,117 ended up being unsupported file types. I'm guessing these won't show up in the file log? Correct, unsupported files, for example TXT, HTML, etc; are not imported and therefore not shown in the log.

>>19702 So I ended up copying all of my files to another drive so I could use the delete after import option. All of the extra files have no extension and when I use the file command it says they're zlib compressed data. I'm not sure what do do with them.

>>19710 Oh, and 46 of them did get imported. In hydrus they're listed at either application/hydrus-update-definitions or application/hydrus-update-content.

small feature suggestion: If you select files on a page, and open those files on a new page, the new page should inherit the sort order of the page the files came from, instead of having the normal default sort.

https://www.youtube.com/watch?v=zIY17xqTzmk windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v529/Hydrus.Network.529.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v529/Hydrus.Network.529.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v529/Hydrus.Network.529.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v529/Hydrus.Network.529.-.Linux.-.Executable.tar.gz NOTICE! For everyone but macOS, Version 527 had special update instructions. If you are updating from 526 or earlier, please consult the post here: https://github.com/hydrusnetwork/hydrus/releases/tag/v527 I had a good week. There's a new quick lookup system that lets you search for a file's source without importing it. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html sauce Every now and then, I am looking at a file outside of hydrus and I can't remember who the artist/character is, or I want to know if I have the artist subbed. When it isn't something the common online source platforms support, I usually download the file, import it to my client, and then do a 'open->similar looking files' on it to see everything I have that looks like it to get more info. I'm basically doing SauceNAO but on my own client. I wanted a way to do this faster. So, this week, check out the renamed 'system:similar files' predicate on a fresh search page. It now has two tabs. The latter is the normal 'system:similar to' that takes file hashes, if you need to do some manual lookup between imported files, but the first panel is now essentially a paste button. If you copy image data or a file path to your clipboard and paste it there, it'll calculate the similar files info and embed it into the search predicate, letting you search for anything that looks similar to what you have in your clipboard. Give it a go! I also added a search cache to the main 'similar files' search system. Repeat searches should be massively faster, and searches you re-run with increased distance, like 0 to 4 to 8, and the background similar files search should also accelerate naturally as the cache populates. My 10k file test client's similar files search sped up about 3-4x! I'm not sure what a million+ file client will do, but let me know what you see. other (advanced) highlights The v527 update went ok, no massive problems, but I wish I had done a bit more testing. Some Win 10 users are getting a two-second delay on opening any page due it seems to a Qt bug that got fixed in the next update, and putting these new libraries in front of more eyes would have caught this. Therefore, I have made a 'future' beta build that is the same code as the normal release, but it uses the newer library versions I am planning to next update to, for instance Python 3.10 instead of 3.9 and Qt 6.5.0 rather than 6.4.1. I am not sure how often I will put these future previews out--maybe once a month, maybe every week--but for now, if you had any weird issues with the recent update like the two-second bug, please check out the 'future' version of last week's 528 here: https://github.com/hydrusnetwork/hydrus/releases/tag/v528-future-1 . I'd also like to hear from anyone who has an older or unusual system and just wants to try it out. I particularly want to hear about problems--if you need to perform a clean install to update, if you have an older OS and it straight up won't boot, just any feedback on what is good and bad, and I can tweak it before I roll the updates into the master branch. Thanks to a user, the HTML parsing formula now lets you search the previous or next siblings of a tag (instead of just descendants or ancestors). If you know you need the 2nd 'p' tag after the 'div' you have, it should be pretty easy now! next week I'll keep at small jobs and cleanup. I realised just now that my new similar files paste button should probably take direct file URLs too, so I'll give that a go. It'd be nice to hammer out some more Client API stuff too, and I really need to catch up on github issues, but we'll see. I am not firing on all cylinders right now, so I am keeping it simple.

I'm an idiot and I'm locked out of my database. I know the password is stored in plain text, can someone give me the steps to fish it back?

please make command-bar searches case insensitive. I keep thinking that I don't have a certain page open, and then I find out later that I do, but I capitalized the first letter of the page name.

I modded the visuabusters parser to add the "creator comment" to a note. It's my first time dipping my toes into really modifying downloader components so let me know if anything's messed up. I just kind of fiddled with it and it worked but maybe there's a better way idk.

>>19716 Oh, and if someone's motivated enough, feel free to put it in a PR to Presets-and-Scripts, I don't want to make a GH for one single pull request.

Is there anything to help me tag my fuckass huge reaction folder or do I just have to go through every picture one by one? Can you remove tagged images from a list so you know what is left to get cropped?

>>19718 >I just have to go through every picture one by one Decide on categories of reaction images. Ctrl+click on all image of a category you can find by eye, mass tag them at once, then add -"that catagory" to your search to filter out what you've gone through. >so you know what is left to get cropped? What do you mean? Are these reaction images, or random images you need to crop the faces of to turn into reaction images?

>>19719 >What do you mean Sorry, I meant to say tagged, not cropped. I wanna get rid of the tagged images and keep only the non-tagged ones so it's easier to see what needs tags.

>>19720 Also, I added two import folders, how do I make the pictures in there fill the Files tab?

>>19721 Go to the files tab. Double click "inbox". For personal tagging, I make a point of not archiving a file until I'm sure I'm completely done tagging it. Do you plan on manually tagging everything? Have you looked into the PTR yet? It may be of use to you if you don't develop tagging autism like me, and also have an SSD.

>>19721 Also, several thousand files is a pretty big import for personal tagging. Could you have broken up those imports by subfolders, or were those just a couple really big folders? I generally import around 300 or so files at a time at most, but my folders are also half-decently organized.

>>19723 One big folder and one that had subfolders, but the import folders thing put them all in the same Art folder category.

thank you for creating hydrus, i used hydrus for a year and ended up liking it. i pretty much access all my images with hydrus now. For how long is this software going to be supported? and most importantly is it safe to delete my normal collection (they have already been imported to hydrus)? i am planning on deleting my normal collection with folder structure to go full hydrus because having a separate collection of the same data takes too much space. i don't want to having to switch back again to folder structure again in case hydrus goes EOL

>>19725 >is it safe to delete my normal collection (they have already been imported to hydrus)? Read the getting started guide. Whether you nuke your old folders or not is up to you. Common sense says keep backups. >For how long is this software going to be supported? Probably as long as hydev doesn't get tired of actively developing it. Probably even longer, given the high level utility, someone will pick up wherever he leaves off.

It'd be really cool if the file history graph could let you zoom in so that you could look for closely at just the recent history of your db. With the whole view, it's difficult to see changes that are big on a smaller scale and would be visible, but tiny in the context of your whole db's history.

>>19725 >For how long is this software going to be supported? Never underestimate dev-anon's weaponized autism.

>>19654 This was resolved by deleting all downloader components and reimport them from the github page. Something must've got messed up over time.

>>19690 Yeah, I'm sorry, this really sucks. Especially when you just want to add a quick 'my tags' sibling and you have to wait for the PTR stuff to load before the UI unjanks. You are correct, basically I have to rewrite the guts of the dialog to load and cache what it needs on demand rather than loading everything. It needs the relevant pairs so it can detect loops and figure out if something should be un-pended or petitioned or whatever, but it doesn't strictly need everything, and it should be able to load instantly. >>19691 Thanks, this is an interesting idea. I should be able to add this logic as a selectable thing somewhere in the manage tags dialog UI. >>19692 I am no expert in Firefox add-ons, but can you override it to allow unsigned addons? It seems like you can do it on a temporary basis under 'about:debugging', and maybe the 'Developer Edition' of firefox will let you set that setting permanently. Might also need to set 'xpinstall.signatures.required' in 'about:config' to false. >>19694 Yeah, I would be totally willing to add some protection layers, an additional yes/no dialog on a delete or similar, but my general idea is to show this info in the UI clearly, so I hope you'll see when an individual file is in a collection. On the thumbnail view, it'll be a little icon, and in the media viewer I want to let you actually browse the collection 'perpendicularly' to the main carousel and without losing your place in the carousel. There may be a simpler way to do it though, but in any case yes, I agree, I think when a file is part of a larger thing--and related, images compiled in any clever virtual or real way into a cbr file--the user should know what they are doing when they change that file. >>19695 This is interesting. I think what's happened here is the best file of the group has been completely deleted and physically removed from the trash. If it were still in trash, that menu would find it, but it won't look for group members that have been completely removed. Now, since it was the king of the group that was removed, it looks weird. I suppose I should catch this situation. I could be as limited as hiding the dumb-looking UI here and just say it is the king, or if you physically delete a king, I could promote a remaining lesser file to be the new king. To answer your actual question, you can't really see the king. Even if you finagle the right search here in the 'all known files' domain, you'd just see the hydrus icon (for files you don't have on your disk) in the place of the king. I'll think about this a bit, I think promoting other files when the king is deleted is a good idea. Maybe if you delete a king, it should also offer to delete all the other files in the group? Maybe it asks you what to do? I have some background work to do with thumbnails and how much they know about their duplicates, which these issues are urging on.

>>19696 Not yet. Thumbnails don't 'know' their duplicate relations yet (it all has to be fetched from db when you right-click), but they will at some point, including stuff like alternate group id, and then I'll be able to sort by it. >>19697 >>19701 >>19702 >>19709 >>19710 >>19711 If you check services->review services and look at your local file domains, there should be one called 'repository updates'. When you sync with the PTR, you'll download a bunch of zipped up JSON files--that's what these are. They are what is processed when you process the PTR. They are mixed in with your media files in normal file storage. You can ignore them. No worries if you don't import them, or accidentally delete them or something, your client will heal itself and redownload anything it needs again. There's a diagram here with the overall structure of these domains and why they would have their respective file counts: https://hydrusnetwork.github.io/hydrus/advanced_multiple_local_file_services.html#meta_file_domains Note the client has terrible automatic update recycling/clearing atm, so if you delete the PTR from your services, these files will probably still hang around for now. If you turn on help->advanced mode you can search the 'repository updates' domain just like 'my files' and see them all. >>19704 No, but good idea! >>19707 Yeah, earliest. Basically, I store two types of modified time atm: what the file has in your disk, the actual file modified time, when you import; and for every website you download/see that file from, it notes the current time or the parsed post time, if there is one, and stores "safebooru was 2021-08-09 blah blah". You can see all these when you right-click a file with some URLs and explore the menu top line a bit. Then, the single modified time you see in most of the UI and what is used when you search or sort files is the 'aggregate' modified time, which is basically 'the smallest stored modified time that isn't in the 24 hours after midnight 1970'. I assume any timestamp of 0 or so to be some parse error and hence discard them. Otherwise, the minimum here is an attempt to asymptote towards a fairly decent 'this was created by some human around this time' time. >>19712 Thanks, good idea. >>19714 First of all, back up your database so if this goes wrong, you can roll back and try again. Then: Download SQLite Studio or another free SQLite database browser. Load up your 'client.db' in your 'db' folder and look up the 'options' table. This table has one row in its data. Copy that whole data to an external editing program (the one in SQLite studio has no ctrl+f). It should be nicely formatted. Ctrl+F for 'password', then replace the shit that looks like this: password: !!binary | ungWv48Bz+pBQUDeXa4iI7ADYaOWF3qctBD/YfIAFa0= with this: password: null Save it and paste it back into that single row in the 'options' table. Save/commit any changes. Try to boot. Let me know how it goes.

>>19715 Thanks, I'll see what I can do. >>19725 >>19726 >>19728 Great, I am really glad you like it. If you have been at it a year and haven't needed your files in their old folder structure, I think it is safe to delete the original copies. To keep a backup, make sure you backup your hydrus client, as here: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#backing_up I have been at this about eleven years now, and I feel generally good to keep going. I'm a weird guy and don't know what the hell I am doing, but I've got a schedule I like, I'm just comfortable enough on money, and I haven't burned out yet. This past year has probably been close to my worst, but I got through it and I'm finally sailing on open seas again. I generally foresee myself working on hydrus in some way for another five to ten years. There's enough work to keep me occupied for thirty at the current rate, and I get self-esteem from it, but AI is going to change many things hydrus related, so I won't make any firm predictions. Hydrus collections may be really useful for training various AIs to do cool stuff, but if a future image-gen AI model can make really good stuff on demand, then features like hydrus subscriptions are of less importance. Loading up mid quality images from your collection is possibly of near zero importance. On my end, if an AI can clean up hydrus's code faster than I ever could, and even sketch out new features, then my contributions may simultaneously accelerate in productivity and also become less important as people fork hydrus and do their own stuff more easily. Hydrus is completely free, no strings attached public domain work. If I get hit by a bus, it would be completely possible for others to continue. I know a couple of programmers who might jump in if that happened, but I also know that several other guys have tried to make hydrus alternatives and their work generally fell off as the grind sucked out their soul. It takes an unusual person to work on Anon-tier projects. I can't promise anything, and I have no firm fallback plans (I'm a turbo sperg who rarely talks to anyone behind the scenes), so it'll be up to the community to figure it out if I go 'last online 319 days ago'. Ultimately, as long as there is interesting work to be done, I would like to keep pushing the boulder up the hill. I wrote the program for myself, and I don't want to stop seeing new features every week either. >>19727 Great idea. I'm going to add some file search tech to it at some point, so you can limit the search domain, and having a date range too would let us figure out a mickey-mouse zoom.

>if an AI can clean up hydrus's code faster than I ever could That demonic AI spawn might be good for refactoring, but to make new features plus to plug them to Qt is a totally alternate universe. Only autistic fags are up to the task.

I've encountered a bug. When I have "filename:" tags and choose to automatically rename the file to it's "filename:" tag upon exporting via drag and drop, if the "filename:" tag is shared by any other files with the same filetype, one in particular will be chosen every time that replaces whatever I had selected upon exporting. This can be remedied by changing or removing the filename tag. This does not occur with regular exporting to a folder. Second pic is what I had selected and first pic is the result of drag and dropping. The file chosen does not have to be in the current page.

>>19734 Oh, and I'm Hydrus version 527.

What happened to the local tags tab? Using v529

Small would-be-nice feature: Mr. Bones also showing the current stats on files in the trash

>>19735 This could be a serious issue, so don't change anything big or overwrite your backup. I haven't changed anything in that UI in a long time, and I can't think of anything in 529 that might have 'hidden' the service there for you, and I've never heard of this before, so I think that service is truly missing or damaged. Please check your services->review services and services->manage services panels. What local/remote tag services do you have there? What do you expect to see? Is the 'hydrus network' service something you expected to see, or is that apparently new--is that the Public Tag Repository, or something else? Have you been in manage services recently and edited or deleted anything? If the service has seemingly completely disappeared from your services panels, then it looks like A) You deleted it unknowingly by accident or B) Your database has suffered some damage and the critical line (and perhaps more) in your services table has been removed. If it could be B, check your other services in review services. Is anything else missing? A rating service maybe? If you have a backup, I recommend rolling back to it, even just temporarily, and seeing if you have the service there. If it shows in the UI all good, then something removed it from your newer database. If this has happened, there is a tiny chance that I can help you recover it on your newer database, if this was due to database damage, but if this was an accidental user deletion, then I do not think I can, and you should roll back to the backup completely. In any case, I can give you some manual SQLite commands to run through here if and when we need them. As a related thing, if you think you might have had database corruption, check out 'install_dir/db/help my db is broke.txt' as background reading. It will tell you how to check your database for errors so we can know more here.

>>19738 I think this post was meant to reply to >>19736, not me.

Was image caching improved in v529? Viewing large files used to always have about a second of load time for me, but I'm now noticing that I can now switch between loaded images instantly.

>>19734 This always happened when I drag and droped any file that would have the same name as another file I dropped before. I just added {file_id} to the drag and dropped pattern

Thanks anon, that fixed it. Does anything go wrong with particularly long filenames?

>>19738 >Please check your services->review services and services->manage services panels. See attached. If there are any sub-tabs you want to see just ask. >What local/remote tag services do you have there? What do you expect to see? Local tags and "Hydrus Network" (PTR) >Is the 'hydrus network' service something you expected to see, or is that apparently new--is that the Public Tag Repository, or something else? That's the PTR. I had it a long time before I knew it was called the PTR >Have you been in manage services recently and edited or deleted anything? No. Updated and it was gone. I'll restore a backup and see how that fairs. Luckily I haven't been downloading anything for a few weeks before I first saw this issue, but will lose some progress on archive filtering (<500).

>>19743 Ok, phew, looks like I panicked. It still exists in review and manage services, so this is some very odd display issue. I now think this is a dumb code error I made. You have a ton of tags in your 'my tags', which will be useful. Can you select 'my tags' in your normal file search page autocomplete dropdown (instead of 'all known tags'), or is it hidden there as well? If you can select it, try doing so and typing some regular tag autocomplete searches on it--do you get the tag counts back ok, and can you do searches, or are there any errors? If you do a search, do you get the expected sorts of tags in the 'selection tags' box on the left side of the file search page? If you set the tag domain of a search page to something specific instead of 'all known tags', it should limit what you see on the left to just what is in that domain. If everything works ok there, then this is even better news. I will examine the manage tags dialog initialisation code closely today. I don't know what the hell is going on here.

I had a good week. I mostly worked on quality of life improvements--highlighting large downloaders is now smooth and cancellable, and the file search tag autocomplete can now take pastes of multiple tags and will also parse typed system predicates. The release should be as normal tomorrow. >>19743 >>19744 I can't really see what is going on, so I wrote one thing in a clearer way and figured out some debug statements. When it is convenient, please turn on help->debug->report modes->gui report mode in tomorrow's v530 and then open a manage tags dialog. It'll make some popup statements about which services it sees, how it adds them, and how many pages it thinks it has. We should be able to see if we have a hidden page or overwritten service ids or something. Let me know what you get, and/or if this magically fixes itself somehow. A side thing: how are your tags->manage tag siblings/parents dialogs? Same missing service, or fine there? You aren't the only person with this issue btw--I just got it from someone else, so this is definitely something stupid that I did. Sorry for the trouble!

>>19744 >>19745 >Can you select 'my tags' in your normal file search page autocomplete dropdown (instead of 'all known tags'), or is it hidden there as well? It's there and autocomplete works if it's selected. Using a test tag there are more results from the PTR than local tags as expected. Following your comments and what I can see here it does indeed look like the UI is wrongfully not displaying the local tags tab only in the manage tags dialog. I'll keep my backups around just in case. Thanks dev-san

https://www.youtube.com/watch?v=Gp1dzdFmwmc windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v530/Hydrus.Network.530.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v530/Hydrus.Network.530.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v530/Hydrus.Network.530.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v530/Hydrus.Network.530.-.Linux.-.Executable.tar.gz NOTICE! For everyone but macOS, Version 527 had special update instructions. If you are updating from 526 or earlier, please consult the post here: https://github.com/hydrusnetwork/hydrus/releases/tag/v527 I had a good week. You can now type system predicates in a normal search, and highlighting big downloaders is nicer. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html highlights The tag autocomplete text inputs on search pages get a couple of neat improvements this week: First off, you can now type most system predicates, and they'll appear in the dropdown. You type the whole complete predicate, like 'system:imported before 2021-03-09'. This is the same system predicate parser that has been available in some advanced places, now for everyone. Note that it can't do every predicate yet, and while I try to cover several grammatical variants, some it is still quite inflexible. Let me know how you get on, and I'll keep working on it! Second, the autocomplete now has a paste button. It accepts multiple newline-separated tags and will understand -negation, wildc*rds, namespace:* wildcards, and the new system predicates. For downloaders, I made the 'highlighting' when you double-click a downloader or thread watcher asynchronous--it now loads the files in the background, only presenting them when the results are ready. This means if you highlight a big 6,000 file downloader, it won't hang the UI for several seconds. You can cancel a load by un-highlighting or highlighting something else, and a file-load that takes more than a few seconds will make a popup with its progress. next week I've just got one more week before my vacation, so I'll just do some code cleaning and little fixes again. Thanks everyone!

I've been processing a lot of duplicates lately, and it seem like I keep running into this situation >Compare A & B set as alternates >Compare B & C set as alternates >Commit actions >Compare A & C set as alternates Shouldn't it be that if A & B are alternates, and B & C are alternates, that A & C should automatically be set as alternates? When I start running into large sets of similar images, like Squilliams for E3 posting, I find it much easier to just drop out of the duplicate filter, look up a tag that covers 90% of those similar images, and then mass set as alternates by eye. But even then, should I miss one or two, I seem to end up seeing it in the duplicate filter a dozen times comparing it to a bunch of different images from the same set of alternates. My brain is fired from processing hundreds of potential dupes. Am I imagining this? >>19747 >you can now type most system predicates, and they'll appear in the dropdown. real fuckin neato. I felt that's how it ought to work intuitively and kept trying to do it for a while.

>>19748 >Shouldn't it be that if A & B are alternates, and B & C are alternates, that A & C should automatically be set as alternates? There is the case where A and C are the same image but C is a lower resolution version. >A is no cum version >B is cum version >C is a low res no cum version You want to mark B & C as alts as to not move tags from C to B, but then trash C when comparing it with A.

>>19749 >There is the case where A and C are the same image but C is a lower resolution version. Oh yeah, fuck, I'm retarded. Guess brute force is the only solution. I've gone up to search distance 14 with a few thousand potential duplicates, and I'm still marking about 10-15% of the comparisons I go through as alternates.

Is there a way to white/blacklist tags from being added to a file? I know there's a way to prevent files with certain tags from being downloaded, but I want to ignore certain tags. I find that lots of booru tags are useless unless you're looking for an extremely specific file out of many thousands, and even then levels of tagging are inconsistent. I have tag clutter, in my client there are over 1700 unnamespaced tags that are only on one file. Take porn for example. I do want tags like "fellatio", "cum", "erection" and so on but things like "meta:highres", "hoop earrings", "too many sex toys", or the hundreds of "[color] [thing]" tags will never be searched for. Ideally I could come up with a list of tags I want to have based on what I search for, then add them to a whitelist and only tags from that list can be added to files (either manually or automatically via downloader etc). Also there would be an option to whitelist namespaces, if I have one image by a certain artist I still want the creator: tag for example. Like the tag filter except instead of filtering shown tags it restricts what tags any given file can have.

Might just be me but the pixiv downloader/parser seems to be broken

Two questions Is there any way to fix this? The drive is just fine Also, how can I export my subcriptions to just toss them into some working backup?

>>19753 Yes, following the instructions helped me a great deal. Backup your stuff, follow the .txt help files.

>>19752 Windows.v529 here. Broken for me, too. Accepts an URL in the downloader window but does not even return any text.

I ran the duplicate filter at distance 16, but only processed duplicates up through most of the potential duplicates for distance 14. I want to reset potential duplicates as I'm not touching distance 14 and up again, but I'm not sure I should because of the "files' search status". What does that mean that that will be reset and does it matter for me?

>>19752 Looks like they removed the original size url from the API ("url_big"), unless you're signed in.

Copying my login cookies from my browser to hydrus makes pixiv work again.

>>19756 >"files' search status" That just means whether or not a file has been checked for duplicates. A file can have potential duplicates. Separately, a file can have been checked for duplicates or not. It's just saying that both of these will be reset.

>>19751 >Like the tag filter except instead of filtering shown tags it restricts what tags any given file can have. But why delete them? Why not just remove them from view with the tag display options?

>>19745 >When it is convenient, please turn on help->debug->report modes->gui report mode in tomorrow's v530 and then open a manage tags dialog. "my tags" is mentioned so it looks like it's seeing it but not displaying it

I'm trying to connect the api thing but the way to do it seems different since I don't see anything along the instructions description Any idea of how to do it on current version?

>>19758 How?

Is there an easy way to sync the db's client images and tags to the server's images and tags? I can bang out some SQL to copy what look like the relevant tables from one to the other but I'd prefer not to if there's an easier way. The reason I'm asking is that I'd like to try running Hydrus as a server on my NAS. I know the docs have a vague warning about server performance not being good enough for use on LAN but I'd like to at least try. It's that or migrating to an actual self-hosted booru.

>>19763 Install and configure the Hydrus companion extension, then click the designated button once you're logged in.

>>19765 I keep getting error there, activated the api, created one, added the key to hydrus and yet doesn't work Isn't there a way to manually add it? Since I can download them with the helper but the add button will not work Ths used to work before updating to 530, something broke along the way

Also, any idea of where can I set the time it takes to override bandwith settings? The default seems to be 30 seconds and I have 300 tasts for it so just checking takes forever

is there a way to export a selection of images along with their tags in a separate txt file for training LoRAs?

Anyone know of an up-to-date fork/alternative to iqdb_tagger? I get a ModuleNotFoundError when I try to run it now. I assume because the package is named hydrus-api instead of hydrus? I have a lot of images to tag and doing it automatically would be preferable.

>>19748 >>19749 >>19750 Yeah, this situation is a real pain. I really hoped that alternate could be fully transitive, but as I devved it I realised I have to do this second layer of comparison. For large alternate groups it causes a rats nest of relationships. The good news is that most alternate groups in normal files are small, just 3-4 WIPs or costume alts, and if you compare more closely-related pairs first, you tend to clear up the true dupes first and save yourself confirmation work. When I eventually get to the alternates expansion, I'm going to have to re-examine the logic here and the UI workflow and see what I can do. I suspect there are many many rules I should apply to the order in which I present the duplicate files queue too, and give you options around that. Like if I seeded these annoyingly related but seemingly necessary decisions apart from each other, they wouldn't be so jarring to have pop up. Or, if you wanted, I could rigorously present every decision on a particular group, with a count down of '8 decisions left', and you could happily clear the whole thing. And of course showing you more closely related pairs before less related pairs. Unfortunately the duplicate search code is a nightmare hellscape of a like 11-table join, so I need to spend some time gathering my chi before I can think about clever sort orders in the follow-up. At the moment, if you have a 100-file CG set of alternates (e.g. five thousand autistic clean/messy variants), I recommend selecting them and right-click->fileshit->set all as alternates, and it'll spam the full network graph against them. >>19746 >>19761 Thanks, and damn. Another user who has this says they were recently updated to python 3.11. How are you running the program--from source, from one of my builds, or an external package like from the AUR? What is your python and Qt version in help->about? Also is it PySide6 or PyQt6? I'm wondering if a new version of Qt is being added or something and because I am not doing x y z, it is hiding the new tab. >>19751 Yeah like >>19760 says you can either right-click->hide the tags you don't like, if there are just a few, or you can (WARNING, advanced autism UI) go into network->downloaders->manage default import options and then to the import options for 'file posts' up top, edit the tag import options, find where it says '[x] get tags' and for that service click where it says 'adding: all tags'. This spawns the default tag filter for the downloader parser. You can add any tags you don't want to be parsed to the blacklist, or just what you want to whitelist, whatever. If you have multiple default tag import options set for different domains already and need to set this up a bunch, sorry for the current awkwardness, but tag filters have easy import/export buttons if you need to spam. In the tag import options UI, there are also very complicated actions under the 'cog' menus. You can say 'don't add a tag if it doesn't already exist', if that works for you, or just for a certain subset of tags, like unnamespaced.

>>19763 I used a cookie manager extension to export my cookies as a "cookies.txt" and then added it under network > data > manage cookies

>>19752 >>19755 >>19757 Thanks. That's a shame, I'll see what I can figure out as a fallback for guest logins. >>19753 If you decide to migrate your subs, you can do import/export from the manage subs panel, including to a png file, which is probably your best bet. I have heard this can fuck up any custom tag options though (might be ok to a backup though), so if you do it, check out a couple of the subs once imported to make sure all the settings worked out how you want. Also yeah check out the .txt file if you haven't yet, it explains everything. If the drive is healthy, it was probably a rough power cut during a database commit (save). >>19756 Thanks I'll reword this. >>19764 Your best shot here is to run a Hydrus Client on your network and connect to its Client API. I know a guy who does this somehow with the Docker build, but I don't know much about that tech so I can't talk cleverly. The server can't host booru style html, it can only talk to clients. Hyshare and Hydrus Web are good, as here: https://hydrusnetwork.github.io/hydrus/client_api.html This may be related, as written by some users: https://hydrusnetwork.github.io/hydrus/youDontWantTheServer.html For the db stuff if you want to play around anyway: client to server is probably ok. You are copying from the table in client.mappings.db to server.mappings.db, and mappings via the hash and tags tables in xxx.master.db (although for the server you'll have an additional map table to go through to map service_hash/tag_id to master_hash/tag_id. Server (or anything) to client is too difficult to do via SQL--the caches and stuff that mirror and optimise search and sibling and parent calculation are too complicated to do manually. >>19766 You can paste a cookies.txt (which some browser add-ons can generate for you) to network->data->review session cookies, if it helps. For the Client API, what happens if you put 'http://127.0.0.1:45869' (different port if you set a diff port in your manage services, ideally keep it on http too for now) in your browser? Do you get a pretty lady telling you about the Client API? >>19767 I think that's hardcoded. I made it to ensure that a gallery will finish its search even if a domain is throttled. If you want them to run faster, hit network->data->review bandwidth usage and edit rules to give your domains (or global) some more bandwidth. There's some in-built delays in normal downloading, too; hit up options->downloading to see the 'fixed time' between gallery page fetches. >>19768 I haven't got an easy one-click system yet, but check out the highly configurable 'sidecars' system in normal file export (right-click thumbs->share->export->files) to see what you can do. https://hydrusnetwork.github.io/hydrus/advanced_sidecars.html >>19769 I don't know much about this, sorry! These are the only links I have: https://github.com/softashell/iqdb_tagger https://github.com/nostrenz/hatate-iqdb-tagger Is the second one a fixed fork of the first?

>>19770 >Thanks, and damn. Another user who has this says they were recently updated to python 3.11. How are you running the program--from source, from one of my builds, or an external package like from the AUR? What is your python and Qt version in help->about? Also is it PySide6 or PyQt6? I'm wondering if a new version of Qt is being added or something and because I am not doing x y z, it is hiding the new tab. I'm using the AUR package Versions in pic related

I dont suppose there has been any progress on getting twitter/nitter subscriptions to work again somehow? Or is gallerydl still the best alternative

how do I make the local tag autocomplete default to all known rather than local tags only?

>>19773 Ok I put a bunch of time into this, reproduced it, and figured it out somewhat. It is 'fixed' for v531 so let me know how it goes. The problem appears to be: Python 3.11 PyQt6 6.5.0 or 6.5.1 Two tabs Total tab character count > ~12 Hit setCurrentWidget or setCurrentIndex on index 1 during the same event that created the tabs (in your case, the default tag service is PTR) = First tab (index 0) disappears This is such a fucking weird combination of things (e.g. PyQt6 6.5.0 on Python 3.10 does not do this) that I have no earthly idea what is actually going on here. Since the character count comes into play, I think a during-init size calculation is overflowing when many sizes are tiny or (0, 0) and thus the tab bar is being scrolled one to the 'right' somehow, so the first tab is in the -1 drawing position. But why would it only hit Py 3.11? Is there a C object conversion difference in PyQt6-sip for Py3.11? Who the hell knows. I don't really want to rebuild the 50-odd places I do tab selection to handle this specific situation, so I will hack a solution for the three tag dialogs and filename tagging, just making the selection event happen in the subsequent event. Now we know what this is, please let me know if it occurs elsewhere and I'll fix them too. Thanks for your help and patience! >>19774 Yeah I think twitter is locked up pretty tight now. Their policy explicitly does not want scraping--you have to sign up and pay to get clean API access now. Gallery-DL is good. yt-dlp has been excellent for video, but I noticed it has started failing on some vids recently though. >>19775 In manage tags dialog? Hit up tags->manage tag display and search, then the local tag service tab you want to edit and then set the default tag service for manage tags there. Note you can't have 'all known files' and 'all known tags' at the same time, so set 'all known files' to 'my files', 'all my files', or 'all files ever imported/deleted'.

I have a lot of images featuring a certain character, and I need those higher search distances to catch all the alternates. However, since preparation and filtering are separate, I have to run preparation for potential dupes between all files, and then filter down to the character's tag before duplicate processing, and afterwards reset potential duplicates because of the tons of false positives that were found that don't involve this character. Is there a way to run potential duplicate prep comparisons on just files with a certain tag? This would save a ton of time processing potential dupes, and skip the step of resetting them. If I can't do this, then that would mean to best ensure I don't miss any alternates/dupes, I would have to use the former process repeatedly whenever I do a large import of files likely to have dupes, and that's a lot of redundant potential dupe preparation.

>>19776 >The My Tags problem This is going to sound dumb in that I never tested this, but I scrolled over the PTR tab and the My Tags tab suddenly appeared. It stays there if I have My Tags selected when I close the dialog, but it goes away if it's not selected. It puts the urgency down a bit

What is the best Gallery-DL command to download images from twitter with the tweet commentary? I want to import them later to Hydrus and add the commentary to the notes. right now i'm using this: gallery-dl --write-metadata -u "myusername" -p "mypass" "https://twitter.com/username/status/000000000000000"

I had a good week. I fixed some bugs--including the recent problem with editing favourite searches--improved some UI quality of life, and added a new services object to the Client API. The release should be as normal tomorrow.

https://www.youtube.com/watch?v=7ACn4nCY4lQ windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v531/Hydrus.Network.531.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v531/Hydrus.Network.531.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v531/Hydrus.Network.531.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v531/Hydrus.Network.531.-.Linux.-.Executable.tar.gz NOTICE! For everyone but macOS, Version 527 had special update instructions. If you are updating from 526 or earlier, please consult the post here: https://github.com/hydrusnetwork/hydrus/releases/tag/v527 I had a good week working on simple fixes and cleanup before my break. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html I also made another future build for v531 here: https://github.com/hydrusnetwork/hydrus/releases/tag/v531-future-1 all misc this week I fixed editing favourite searches, which I accidentally broke last week! If you right-click a tag or a file and it has a billion siblings/parents/children/known URLs, the menus now truncate the giga list. Previously, child-heavy tags like pokemon would make a menu that would laggily take up your entire screen. Now it lists about ten items and says '438 more...'. I fixed the Pixiv parser for guest logins, but it no longer gets the 'original' quality. It seems like they hide this for non-logged-in users now. If you do a lot of Pixiv downloading, figure out a login solution with Hydrus Companion or cookies.txt imports. If you have some downloaders that were inexplicably halted on 'waiting for a work slot', even after client restart, check them again this week--I cleaned up some bad status reporting. The Client API gets an overhaul to service info this week. '/get_services' and '/get_files/file_metadata' are affected. If you do Client API work, please check the changelog and the updated help here https://hydrusnetwork.github.io/hydrus/developer_api.html#services_object . The old method of giving service names and types still works, so nothing breaks today, but it is now deprecated and will be deleted in early 2024. Please plan to move over! next week I am now on vacation! I'm off for a week to shitpost not-E3. I'll be back at it in a week, and I want to start pushing on my larger projects again. v532 should be out on the 21st. Thanks for the support everyone!

Is it possible to revise the way maintenance operations are done so Windows doesn't mistake Hydrus for being unresponsive? You know, that thing where if you click inside a program with the hourglass cursor one too many times it turns light gray.

When I am syncing the PTR, what is hydrus *actually* doing? Like it will say "content rows xxxxxx / yyyyyy; processing at zzzzz rows/s", but I'm not sure I understand what that means.

>>19783 As I understand it, it downloads sqlite database files from the PTR, then processing is when it takes those files, and inserts the table rows into your actual database. I haven't synced to it in about 2 years so I can't check, but I think that's what happens. I remember that the processing took far longer for me than downloading.

Hey hydev, I have 2 requests/suggestions: 1st, could we get an "open this tag in the dupe finder" button? E.g. right click on tag "blue eyes" search>add to dupe filter. I think making a new dupe processor would be better but maybe add a toggle in the settings to change the behavior? 2nd, could we get an auto-favorite tag system? For instance, I use a favorite namespace to describe why I like a certain piece, artstyle, artist, subject, etc. Having all my favorite namespaces automatically show up would be pretty nice. I think at it's simplest it could just automatically add a namespace but I guess you could make it really complicated with a bunch of rules? I don't know if other users would get much use out of these, the first probably more than the second, but I figured I'd suggest them anyways.

Has anyone found a solution for Pixiv yet? I downloaded the cookies with Hydrus Companion but it still doesn't download the original sized images.

Hey guys. Does booru.plus have a parser yet? they have a real surprising amount of content on there I'd like to archive. I tried looking around on the github and searched the previous threads but couldn't find any mention of it.

>>19786 Works on my machine. Did you add your user agent as well?

hi. how to disable autoplay of video? i'm little retarded and can't find that feature. there's videos that crash your player, so i need to delete it, but i can't just click on them.

>>19789 i found

>>19790 There is something wrong with that video. It plays 1 second and then freezes, then it freezes my machine. It makes me wonder, can WEBMs have malicious payloads?

>>19791 https://youtu.be/UbhrGYZTO3Y

>>19788 >Did you add your user agent as well? How do I do that?

>>19793 I dunno if Hydrus Companion can do it for you, but go to network > data > manage http headers, then click add, set the web domain to pixiv.net, set the key to be "User-Agent" and for the value you can google "what is my user agent" and put that in. Then you should probably re-log in too pixiv and copy the cookies over again.

>>19792 Thanks anon. Now I know.

Is there a way to save a path as a tag during import, but only from, let's say, third folder onward in a variable length folder structure? For example from these: D:\Pictures\Common Folder\Animation\filename1.jpg D:\Pictures\Common Folder\Animation\Frames\filename2.jpg D:\Pictures\Common Folder\Random\A\B\filename3.jpg To these tags: folder:animation folder:animation\frames folder:random\a\b I know there's regex, but I'm missing some kind of find/replace feature, where I can exclude a part of the match.

>>19796 Nevermind, figured it out: (?<=^D:\\Pictures\\Common Folder\\).*(?=\\.*$)

I have two ancient hydrus databases (~450) that have been out of use for some time. I want to get this going again but trying to start them with the newest client or version 509 fails. What would be the best way to go about reimporting these? Should I try to just import the images into a new client? Is there a way to avoid importing the thumbnails? The tags are mostly just pulled from e621 so will the community-tag-sync-thing regenerate those well?

>>19798 >I have two ancient hydrus databases Here is the recipe: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#updating

tldr for >>19799 All that said, and while updating is complex and every client is different, various user reports over the years suggest this route works and is efficient: 204 > 238 > 246 > 291 > 328 > 335 > 376 > 421 > 466 > 474 ? 480 > 521

The Sankaku downloader seems to no longer be working because it's fetching the page-urls rather than individual posts (?). Could someone verify this issue and maybe suggest a fix? I just want to liberate my favorites from this hellhole of a booru. https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Downloaders/Sankaku

This failed Twitter download (https://twitter.com/ingi_r0303_2/status/1368974442731171841) Crashes hydrus. Don't care about the video itself, but I'm scared of Hydrus crashing.

I'm trying to use a remote file service and it all seems to work, except when I right-click a file and "remote services -> download", it pops up this message and an "Initializing downloader" that never completes: v531, linux, source TypeError Population must be a sequence. For dicts or sets, use sorted(d). Traceback (most recent call last): File "/opt/hydrus/hydrus/core/HydrusThreading.py", line 427, in run callable( *args, **kwargs ) File "/opt/hydrus/hydrus/client/ClientDownloading.py", line 265, in MainLoop hash = random.sample( hashes_still_to_download_in_this_run, 1 )[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/random.py", line 439, in sample raise TypeError("Population must be a sequence. " TypeError: Population must be a sequence. For dicts or sets, use sorted(d). The server (running v531) doesn't log anything about it, and I tried a few older versions but it all seemed the same.

>>19802 I'm using v528 and the download fails, but it does not crash. Perhaps because it needs to login first to show the post?

>>19804 Sometimes the twitter downloader will output botched or incomplete videos, I don't mind that, but videos without thumbnail and really low filesizes(<100 kb) will come out and crash the cllient if clicked on.

How is system:similar files supposed to work? It seems like it's working sometimes, but most of the time it doesn't, or am I doing something wrong? >copy image from browser >open the similar files window >data tab >click the paste image button, which fills the hashes >click ok >no results >download the image >drag and drop it into hydrus >image already in db What gives?

>>19806 It found the image when I put the search distance to 8, but the images are identical, so why doesn't it work with lower numbers?

Is there a way to blacklist specific tag service tags from showing up in all known tags in search? Or maybe a way to select multiple services like with file domains? I use multiple different tag services for different stuff and when browsing I want the column on the left to show tags from all services except one or two. I know I can stop tags from a service from showing up, but that turns them off even for that service itself, which I don't want.

Hey, catching up after my break. I had a good time, did nothing productive all week. >>19777 I don't have a really nice solution for you, but if you know this is just on a certain subset of files, you could try something like this: - see a file you know has alts you want to clear - right-click->open->similar looking files->whatever - in the new page, shift-double-click the search predicate and edit the distance up to 12 or whatever you want - then, on the big mess of alts that appear, select them and ''right-click->manage->file relationships->set relationship->set all possible pairs as potential for the duplicate filter [sic]' That'll essentially manually force them to queue in the duplicate filter. I'm not sure how spammy/tangled it will be, so if you try it, let me know how it works. >>19778 Thanks, what a weird thing. I keep hearing bad things about the new Qt updates, it is a shame. 6.3.1 is the last nice one I know about for Windows 10, maybe other (older?) OSes. >>19779 I don't know any of the clever stuff here, so if you figure a nice solution out here and want to do it, I'd be interested if you wrote a little guide that I can slap into the help somewhere for other people to check out. I assume that --write-metadata basically spams everything gallery-dl knows to a json file somewhere? >>19782 It depends on the maintenance operation. The program used to be much worse about this sort of thing, but most of the common big CPU hitters like tag repository processing generally only hit you for 0.5s at a time and don't cause this any more. What maintenance do you find locks up your client, do you know which one it is? Or can you walk me through how this most often happens to you? >>19783 Yep, pretty much >>19784 except it is JSON files. I won't go into super super detail, but basically the server exports giant lists of: tag 123456 = 'character:samus aran' file 555666 = (file hash) NEW mapping 123456 x 555666 The client eats up these JSON files and fills several master tables with the raw data (client.master.db and client.mappings.db). There are also a handful of subsidiary tables (client.caches.db) that have copies of the data in different views, with clever indices, that allow for various sorts of fast searching. This all takes a ton of CPU and HDD work to figure out, and there's like 1.5 billion mappings now and tens of millions of file and tag definitions, which is why it has the big background maintenance job.

>>19785 1 - sure! 2 - probably a bit too niche, I'm afraid. Are you always making new tags in this namespace, or is it rare? I'm sure you've seen you can add tags to your favourites from their right-click menu, so is there anything I can add to that to make the process of adding them manually more easily? There's also two locations for tag favourites, it is bullshit design and I should rename them or something, but there's favourites beneath the autocomplete and also in the tag suggestions column you can add to manage tags dialog--which set are we talking about here? >>19787 I've never been here before--can you tell me, is it essentually a reddit aggregator? I had a quick search, and it seemed like the sources it had were all /r/whatever. Would a parser appropriately add the correct reddit URL as a known URL, therefore? And should I expect there to be no tags to parse? And what would be the typical ways to sort--what's the difference between 'newest' and 'timestamp' sort? (I don't know anything about reddit) >>19801 God damn, I thought this got fixed recently too. I'll look into it, thanks for reporting. >>19802 >>19804 >>19805 Yeah I'm afraid I get 404 when I try to download that with my normal guest access. If you have movies that crash the client, that may be an mpv problem. I've had a couple of reports, usually on Linux, about that recently. Can you try sending me those vids? You usually have to zip them up and post the zip, since the borked movie won't upload to the normal places or attach to your post correctly. Since you can't click on the file in hydrus, exporting is tricky. What you do is get the bad file loaded and isolated in a page, then in a different page find some jpeg. drag and drop the jpeg to the dangerous tab, then hit ctrl+a to select both files and then drag and drop export the selection of two files to your desktop by clicking on the jpeg. That's safe since at no point are you asking the client to load the dangerous vid. I fucking hate it when a library crashes on a media load. OpenCV used to do it on certain kinds of PNG, it sucked. It has happened before with mpv, too, but things have been getting better. Btw, if you don't know, yt-dlp downloads most twitter videos very well (although I guess for these nsfw you'd have to figure out some login solution). Try: yt-dlp -f '(bestvideo+bestaudio/best)', '-o', '%(uploader).36s - %(title).96s.%(ext)s' URL This and variants can be easily scripted into a neat downloader you can paste a bunch of twitter URLs into.

>>19806 >>19807 That's definitely not working right! Obviously if you have the exact same file, down to the byte, it should match on both pixel and perceptual hashes. Can you try something for me? Is the file a normal jpeg/png, nothing clever or weird like a tiff or single-frame gif or something? 1) If you have a file like this, what happens when you right-click->open->similar looking files->exact/speculative? Does it show itself there? 2) Hit up database->file maintenance->manage scheduled jobs. Do you happen to have like 150,000 'regenerate pixel hashes' jobs hanging around in there? 3) Hit database->regenerate->similar files search tree. Do these queries suddenly start to work correctly? >>19808 Not yet. But when I did multiple local file services last year, I figured out some code that would do the 'multiple locations' bit you see on the file side, which is an OR of multiple file domains. I want to bring that tech to the tag side too, eventually, and then you'll be able to cross reference whatever slices of the venn diagram you like.

>>19809 >That'll essentially manually force them to queue in the duplicate filter. I'm not sure how spammy/tangled it will be, so if you try it, let me know how it works. That would work if I know something has alts, but the thing is, I can't always be sure, which is why I use the duplicate filter to find them in the first place. This will help me find alts for files I suspect of having them though.

>>19799 >>19800 I ended up just taking the old SQLite database deleting all the thumbnails then importing that folder. Appreciate that all the files are in the filesystem rather than inside some MySQL database. It seems to have worked and since they were wall e621 imports the PTR seems to have it covered. Thank you for the advice regardless. I did have two questions when doing this. How are updates to tags on files handled? If a file gets uploaded to e621 then the PTR gets those tags (how?) then 3 months later someone on e621 fixes a mistake in the tags on e621, does that ever make it to the PTR? Are the PTR jannies manually doing this? The other question I had was specifically for our gracious developer and that was have you ever considered any kind of in-place conversion to save space? I was mostly thinking of jpeg to jpeg XL since from what I understand it has a nearly(?) lossless re-encode function and the savings are significant (around 30% afaik) but you could also do lossless png compression and stuff. I would just keep the hash of the file and all the records, tags, etc. then have a redirect for the path of the file in SQlite to the re-encodeded one. Update: I decided to check my sources and found this: https://jpegxl.io/articles/faq/#doesjpegxlsupportlosslesscompression https://jpegxl.io/articles/faq/#losslesstranscodingfromjpeg https://jpegxl.io/articles/faq/#canirecompressmyjpgimagestojxllosslessly tl;dr; You can convert from jpeg to jpeg XL with no generational loss for around 20% space savings and because jpeg xl supports actual lossless encoding but is more efficient you can convert any lossless file like png or bmp to also get significant savings. So "make DB 20% smaller for free" button when?

>>19811 Yeah, the file is a jpg. >1) If you have a file like this, what happens when you right-click->open->similar looking files->exact/speculative? Does it show itself there? Yeah it finds itself. The thing I noticed though is, that when I copy the image bitmap from hydrus using share > copy > image (bitmap) and paste that into similar file search, the hashes are different compared to if I find the file on disk using open > in file browser, open that file in browser/photoshop/imageglass and copy the bitmap from those. >2) Hit up database->file maintenance->manage scheduled jobs. Do you happen to have like 150,000 'regenerate pixel hashes' jobs hanging around in there? The window is completely empty. >3) Hit database->regenerate->similar files search tree. Do these queries suddenly start to work correctly? Nope. Also the thing finished pretty much instantly, but I also have only like 6k files.

>>19813 I couldn't find anything about efficiency easily but ffmpeg now supports animated jpeg xl encoding which must be way smaller than a gif so that might be another big savings.

>>19810 The site doesn't allow .zip files but I uploaded it here. Just importing it, and click on the seekbar will crash the client https://files.catbox.moe/g1nasz.mp4

>>19810 >probably a bit too niche That's a shame, but thanks for thinking it through anyways I don't add new ones too often but I just know I'll add a new namespace and completely forget to favorite it. >which favorites I was thinking of the autocomplete section, but I didn't even realize there were 2 separate spots lmao.

>>19779 That's part of --write-metadata by default. Have you tried updating gallery-dl?

>>19817 If you want a list of all tags inside a namespace to select from, you could create a separate tag service and in the "tags > manage tag display and search" options for it enable "allow "namespace:"", so every time you type "namespace:" the autocomplete will show all possible tags for it. You don't even need to make a separate tag service for it, but it could make the autocomplete slow if your main tag service has a lot of namespaced tags.

Pic related takes forever at search distance 8 because I have so many files. Would it be possible to select some files in a tab and have hydrus do a higher duplicate search distance on only those files, because i only want to find duplicates within that group? For example a certain artist. In my case I have downloaded an artist's images from multiple sources, some with watermarks, and hydrus can't recognize watermarked/non-watermarked duplicates at search distance 6 which is what I usually use for my entire database. So I'd like to do a deeper search only within those files. But it seems I have to "prepare" my entire database at search distance 8 to do that.

>>19820 I had the same question. Hydev already answered. Top reply here. >>19809

>>19811 >>19814 Sometimes I also get this error when pasting a bitmap copied from hydrus, but if I paste that into photoshop and copy the bitmap from there, it works.

How are collections supposed to work? It bunches files together all right, but when I double click it, it opens just one file, is that the intended behavior? I would expect it to open a new tab with those files inside it or something. I guess I could do that manually with open > new page, but it's kind of annoying having to turn the collection filter off there.

>>19809 >shift-double-click the search predicate Woah. This edit predicates function is neat and useful, especially for various system predicates and OR predicates. Was this function explained anywhere obvious that I missed?

>>19824 Yeah, just found that out randomly when trying stuff. There are also other useful ones like ctrl double clicking switching between include and exclude or shift double clicking on a selection of tags making a quick OR statement out of them.

I guess this is technically a gallerydl question rather than a hydrus question but it feels relevant. Is there any way to make it so the link to where i downloaded the image from is attached to the image when i import it into Hydrus from my gallerydl folder? So that i can easily just open the source through Hydrus like i can with most other gallery downloads rather than having to manually go through my entire twitter likes for example.

I had a good week. I fixed some bugs and improved program stability in multiple places--with luck, I have eliminated several crashes. There's also a couple neat updates to 'open these files in a new page'. The release should be as normal tomorrow.

>>19827 >I had a good week. The best part about these posts.

>>19794 Unfortunately it's still not working for me. Hopefully someone comes up with another solution because it sucks having to manually download images and add urls/notes.

I just got LoliSnatcher on my phone and wonder if i could use it to access my personal Hydrus Network gallery that i got in my computer But another issue is that being the newfag that i am, i don't even know shit about setting a live server from my Hydrus I even wonder if it's possible to access my personal Hydrus server from my phone's LoliSnatcher while my PC is turned off or in sleep mode I just want to view my content and even with settings like randomized order While the actual tagging and uploading new images is still left on the computer

https://www.youtube.com/watch?v=eclMFa0mD1c windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v532a/Hydrus.Network.532a.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v532a/Hydrus.Network.532a.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v532a/Hydrus.Network.532a.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v532a/Hydrus.Network.532a.-.Linux.-.Executable.tar.gz The 532a is a hotfix, I messed something up with the new pixiv parser first time around. I had a good week. I've improved stability and added some little features. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html stability Some damaged videos were causing mpv to dump out in a way Qt was unhappy with, and it lead to black video screens that would crash after a few clicks. I have written a new error-catching system that should recognise these specific files and immediately unload them along with a warning to the user. Also, all other general errors (like missing metadata or chapter marks, which are common for our sort of files) are now printed to your log for future investigation. As we learn more, I expect to add more 'unload this immediately' and 'don't print to log, this error is nbd' hooks here in future. The macOS release has unfortunately been unstable for some/all users since v530. Clicking on lists can simply not work and then suddenly cause a crashe. With some user help, I have learned an important potential crash in Qt and tightened up all the respective areas in hydrus code. If you are a macOS user and have had trouble, let me know how you get on with today's release. If you still get crashes from simple list interaction (e.g. in manage subscriptions), please revert to v529, which does not appear to have the problem. There have been no big database changes since then, so all you have to do is re-install v529 and you should be good to go. The above macOS error handling improvements may have fixed some other unexplained crashes. What once caused a crash should now appear as a normal popup error. Let me know what you see. other highlights When you hit one of the 'open these files in a new page' commands, the new page is now search-enabled and starts with a 'system:hashes' of the files it opened with. This lets you quickly add more search predicates to narrow down the selection! When you right-click a tag, the 'search' menu now lets you open the tag(s) in a new duplicate filter page! I updated the pixiv downloader again. A couple weeks ago, Pixiv decided that if you aren't logged in, you can't get the 'original' quality of an image. I updated the downloader before my vacation by telling it to get the resized version, but as this was importing low quality files silently, it was not a great idea. I have given it another go, and now, if you aren't logged in to pixiv, it sets those files as 'ignored' and the 'note' on the file log mentions you need to be logged in to get the original quality. If you want the previous 'get the medium quality' version, it is now selectable as a different parser for 'pixiv file page api' under network->downloader components->manage url class links. next week I'd like to sink my teeth into a medium size job. It'd be nice to finally catch up on some serverside tech for more PTR updates.

How come Hydrus doesn't see these two images (NSFW) as duplicates, even at search distance 8? The artist logo makes such a big difference? Or is it bugged somehow?

I seem to have gotten my account banned by accidentially spreading the "your filenames suck" tag or a sibling using the duplicate tool... I usually only bother with filenames for soundposts

>>19832 One seems darker colored than the other.

>>19834 That shouldn't matter imo, as each pixel is being compared to the average value of the entire pic. But I don't know.

>>19826 The --write-metadata option in gallery-dl will make a json file that has the related metadata for the file. Then, you can use sidecars to import this data into hydrus. https://hydrusnetwork.github.io/hydrus/advanced_sidecars.html Unfortunately, by default, gallery-dl does not have the source url in the json file. You have to manually add it with postprocessors for each site. First make a file in the same directory as galleru-dl.exe called gallery-dl.conf, that's where gallery-dl configuration settings go. Put the following in that file: { "extractor": { "danbooru": { "postprocessors": [ { "name": "metadata", "mode": "modify", "event": "init", "fields": { "source_url_for_hydrus": "https://danbooru.donmai.us/posts/{id!s}" } } ] }, "gelbooru": { "postprocessors": [ { "name": "metadata", "mode": "modify", "event": "init", "fields": { "source_url_for_hydrus": "https://gelbooru.com/index.php?page=post&s=view&id={id!s}" } } ] } } } I got these from https://github.com/mikf/gallery-dl/discussions/3014

>>19836 I should mention, there are some extractors that are used for multiple sites. For example, the "gelbooru_v02" extractor is used for multiple sites. https://github.com/mikf/gallery-dl/blob/master/docs/supportedsites.md So gelbooru and danbooru are fine, but as far as I can tell there's no simple way of making a postprocessor that works for rule34.xxx as well as safebooru.org since they use the same extractor and their urls are different. I guess you could code something outside of gallery-dl to check the possibilities and add the correct url and make gallery-dl automatically run it after each download with the "exec.command" option. I don't understand why this isn't in the metadata by default, it seems like such an obvious useful thing.

>>19826 >>19836 Oh, you specifically mentioned twitter. For twitter it would look like this: "twitter": { "postprocessors": [ { "name": "metadata", "mode": "modify", "event": "init", "fields": { "source_url_for_hydrus": "https://twitter.com/{author[name]!s}/status/{tweet_id!s}" } } ] } Add this next to the danbooru and gelbooru ones, inside the "extractor" part.

>>19813 >How are updates to tags on files handled? If a file gets uploaded to e621 then the PTR gets those tags (how?) then 3 months later someone on e621 fixes a mistake in the tags on e621, does that ever make it to the PTR? Are the PTR jannies manually doing this? The basic sync is JSON magic. Everything users upload to the PTR via the 'pending' menu (via the tag import options and manage tags dialog) gets crunched into JSON update files. All clients that sync with the PTR download these, roughly one update a day. The process of replacing/fixing existing tags is very basic (my own fault for never fleshing it out nicely), but users can 'petition' to have tags removed, and the janitors approve/deny them, which filters to all users who sync with the PTR. There's also tag siblings, which helps fix typos, renames, and duplicates. >The other question I had was specifically for our gracious developer and that was have you ever considered any kind of in-place conversion to save space? I was mostly thinking of jpeg to jpeg XL since from what I understand it has a nearly(?) lossless re-encode function and the savings are significant (around 30% afaik) but you could also do lossless png compression and stuff. I would just keep the hash of the file and all the records, tags, etc. then have a redirect for the path of the file in SQlite to the re-encodeded one. Yeah, this has come up a few times. My feelings, broadly and simply, are that most media files are pretty well compressed and the complexities of tracking many different duplicates is not worth the hard disk savings. Having many duplicates flying around is a pain for hash matching, not to mention the general UI and workflow hell of merging content appropriately across dupes. That said, JpegXL really does have great savings, as you say. But then again, on a typical client, the total size of bloaty images tends to shrink before the total size of the video content, so even if JpegXL gives us 40-60% savings, if the total is only 10%, that's 4-6% savings. So, this is on my mind, but I'm not totally sold on it yet. Hard drives are very cheap and getting much cheaper all the time. Management is very expensive in human time and getting more complicated as we add more files and filetypes. I'm planning to figure out some conversion workflows though, for users who want to give it a go. Search this and previous threads for 'exe manager' and 'executable manager', and you'll see me talking about setting up ffmpeg conversion workflows.

>>19814 >>19822 Interesting thing with the error screenshot here--8000396 = (1871*1425*3)+1871, so this clipboard is delivering an extra one byte-wide line for the 1871 height of the image (1425 should be the width, and 3 is your RGB 24 bit pixel depth). Sounds like there are some bitmap clipboard formats being sent by browsers that aren't being converted to Qt properly. Even when the shape of the data coming in is correct, maybe it is coming in with width/height reversed, or BRG channels or something crazy, and it is throwing the data generation off. Can you please send me some direct URL examples, and say which browser you are using? I'd like to do my own tests and see what this data is. Since you are all caught up on processing, your search tree is fine. This is some borked clipboard paste. >>19816 Thanks, I got this earlier and fixed it for the 532 release! I was pretty specific with the error catching, so let me know if you get any more and I'll add them to the 'oh shit, dump out' error list. >>19823 It is mostly just for convenient grouping of files. If you want to apply a tag to everything of a certain collection, or work through a comic by chapters, it is great. I've never really liked it though, and I except to add some more navigation tech like you suggest, letting a media viewer know about collection boundaries and doing stuff like virtual page numbering and stuff. I'll be revisiting this when I eventually get around to CBR/CBZ support. >>19824 >>19825 Sorry, things often get slipped in and then never properly integrated into the 'getting started' stuff. Btw I think you can edit predicates from right-click too. I'll see if I can add this to the help somewhere. Eventually, eventually, eventually, all this stuff will get integrated into the shortcut system, and the shortcut system UI will become nicer, so this stuff will be easier to stumble across.

>>19836 >>19838 Im gonna be honest, i dont know what half of this shit means, but im gonna see if i can get it working anyway. Thanks for the help though.

>>19830 If you are new to things, I recommend you first get some experience just with the normal client. Learn about archiving and setting up downloads and subscriptions and stuff, and then move to browsing with your phone. Very briefly: your home computer with hydrus on has to be on for you to interact with it. You need to do a bit of network tech to make it all work (all googlable, if you know what an IP address is you likely know enough). Hydrus Web is a nice easy testing ground, so when you feel ready, start here: https://github.com/floogulinc/hydrus-web https://hydrus.app/welcome Ah, now I look at it, it has a guide on using Tailscale. This is a good but technical program that makes old boomer shit that I'd talk about, like port forwarding, much less of a headache. Good luck, and let me know if you run into any trouble! >>19832 Thanks, that's interesting. I bet it is the watermark, yeah. They match at 10 distance. I think one way the bigger companies like google get around this is they partition the picture into different quadrants or interesting segments and search those separately, and then if one corner is all borked, it doesn't matter because the other three are perfect. In a future expansion of all this search, I'll look into this, and related solutions that handle rotation and flipping. Clipping your similar files comparison area to interesting segments can solve the cropping issue, too. For now though, most people I know are overwhelmed with dupes so my main job is to work on improving the filtering workflow and get started on the first auto-resolution systems. There are several excellent perceptual hash libraries available for python these days, so I may also just ditch my old hardcoded shit for something a bit more professional. BTW my current system works only on shape. Colour doesn't matter, but sometimes moire effects due to resizing on weirdly patterned geometric images can mess it up.

>>19841 If you run into issues just ask :) If you've never done sidecars or worked with json before it can be pretty confusing.

How can I import a tweet commentary downloaded with gallerydl to hydrus notes? on pic1 you can see the tweet commentary and how it appears inside gallerydl .json file, on pic2 is how the twitter commentary appears on hydrus notes when you use the classic twitter hydrus downloader, however when I import a downloaded Twitter image with gallerydl and with the .json file as a sidecar nothing happens, in pic3 is the options that i use but the notes don’t appear or when i try some other configurations i get the disaster that you see on pic4. Is there something that I need to do on the gallerydl .conf file? Or I’m a missing an important step when importing the file sidecar with the image to Hydrus?

Does anyone have an e621 parser that pulls in descriptions and translations into notes?

>>19844 I'll start by saying this: make sure you're doing this in an import folder, not the import dialog! The import dialog doesn't remember anything! Import folders will remember everything you've set up so you don't have to do it all over again! https://hydrusnetwork.github.io/hydrus/getting_started_importing.html#import_folders You add sidecars at the bottom where it says "sidecar importing". In the "edit metadata migration source" window, you have to set up a parsing formula. Click "edit parsing formula". This will bring up the formula editing window. First, you should put your raw data in. Copy the entire contents of the gallery-dl json file to your clipboard, and then click the little clipboard icon at the top right to paste your raw data in. Second, you can set up parsing. Select the default "get all items" entry on the left and delete it with the X button. Click "add" button and add a "dictionary entry" parse rule with match type "fixed characters" and set the "fixed text" to "content". Then click apply. Third, if you click test parse, you should see the correct text there. (That's all you need to do with the json parsing in this case because the json is pretty simple. If you have more complicated json you might have to learn more about json to figure it out.) Fourth, now that the parsing is done, you need to turn it into the right format for a note. There's important info at the bottom of the sidecars help page: https://hydrusnetwork.github.io/hydrus/advanced_sidecars.html#note_on_notes Notes need to be in the format "name: text". You've got the text, now you need a name. At the bottom left, click the button that says "no string processing". It opens a new window. Click "add", then "String Converter", then "add" again in the new window. Select the "conversion type" to be "prepend text" and set the name to whatever you want it to be. Then click apply on the string conversion windows to get back to the json one. Fifth: click test parse again and you should see the name and the text like in the pic. Click apply on all the windows and it should be good. For adding the source url, you'll do basically the same thing but of course have the destination be "a file's URLs" and in the json parsing the "fixed text" will be "source_url_for_hydrus". And you don't need to do step 4 since it doesn't have to be prepended with anything.

>>19846 And at the end I've got this. >>19845 >descriptions Sure, here you go. >translations The problem with importing translation notes into hydrus is that there's no way of knowing where the note box was in the original file. It can be pretty confusing if there's multiple note boxes on the image. I figure it's better to wait until hydev adds the required tech for this. For now, gallery-dl can get the notes as well as their size and position if you set a value in the configuration. https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst#extractore621metadata

>>19840 >Sounds like there are some bitmap clipboard formats being sent by browsers that aren't being converted to Qt properly. Actually, that's a clipboard copied straight from Hydrus using share->copy->image (bitmap). When copying from browser, it pastes just fine, but won't find it at distance 0. >Can you please send me some direct URL examples, and say which browser you are using? I'm using Waterfox. I also tried Brave and it does the same thing. I can't really provide direct links for most of the examples as I either forgot the source, the author privated or in the case of the error one, it's from a steam artbook iirc, so I'll send you the files, just drag the files into the browser and copy from there. As for the image that gives me the error, I forgot which image it was originally, but I found you another one, it's in the folder named hydrus_clipboard_error. The files are also named by the distance needed for them to be found. The only file from the bunch I could find online: https://www.pixiv.net/en/artworks/57961592 The images: https://files.catbox.moe/jrlr7x.7z Btw, it doesn't matter if you copy clipboard from the pixiv link or if you open locally, they give the same hashes.

Is there any way to remove deletion record from selected files that are still used in other file domains? I moved a lot of files from "my files" into another domain and now all the files have a "removed from my files" deletion record, which I think is completely unnecessary.

>>19847 >The problem with importing translation notes into hydrus is that there's no way of knowing where the note box was in the original file. It can be pretty confusing if there's multiple note boxes on the image. That's actually what I was hoping for. Basically just each translation box shoved into a note labeled something like translation_1, translation_2, etc. and in a perfect world dump the x,y coordinate into the note as well. I think that would be good enough just because most images are only going to have one or two text boxes anyway and for ones with more having the translation even if a bit confusing is way better than not having it. >I figure it's better to wait until hydev adds the required tech for this. I'm planning to download quite a bit right now so once this happens I'll have to figure out how to go back and do this either way. I'd like to get everything I can now if possible. I don't know anything about the parsers so idk how hard that would be to do.

>>19843 Run into issues i have, i cant get the gallery-dl config file to work. I have placed it pretty much everywhere i can think of, named both config.json and gallery-dl.conf, placed them both in a folder called config.json but it does nothing. I am 100% missing something incredibly obvious but i cant get this shit to work.

>>19851 Looks like your file is actually named config.json.txt You should at least enable file extensions in windows explorer so you can easily see and prevent such mistakes.

>>19852 That'd be the incredibly obvious thing i've missed. Im a goddamn idiot. It works fine now, thanks.

I'm getting an error whenever I right click on a particular tag in the manage tags dialog or in the tag display on the left panel. Here is the error: v532, win32, frozen DBException KeyError: 'filename:fztgktyamaaulyy' Traceback (most recent call last): File "hydrus\client\gui\ClientGUIAsync.py", line 60, in _doWork result = self._work_callable() File "hydrus\client\gui\lists\ClientGUIListBoxes.py", line 2854, in sp_work_callable num_parents = len( parents_to_service_keys ) File "hydrus\core\HydrusController.py", line 700, in Read File "hydrus\core\HydrusController.py", line 201, in _Read result = self.db.Read( action, *args, **kwargs ) File "hydrus\core\HydrusDB.py", line 891, in Read self._loop_finished = True File "hydrus\core\HydrusData.py", line 1491, in GetResult hydrus.core.HydrusExceptions.DBException: KeyError: 'filename:fztgktyamaaulyy' Database Traceback (most recent call last): File "hydrus\core\HydrusDB.py", line 574, in _ProcessJob File "hydrus\client\db\ClientDB.py", line 6343, in _Read elif action == 'tag_siblings_all_ideals': result = self.modules_tag_siblings.GetTagSiblingsIdeals( *args, **kwargs ) File "hydrus\client\db\ClientDBTagDisplay.py", line 443, in GetSiblingsAndParentsForTags tags_to_service_keys_to_siblings_and_parents[ tag ][ service_key ] = ( sibling_chain_members, ideal_tag, descendants, ancestors ) KeyError: 'filename:fztgktyamaaulyy' Database Traceback (most recent call last): File "hydrus\core\HydrusDB.py", line 574, in _ProcessJob File "hydrus\client\db\ClientDB.py", line 6343, in _Read elif action == 'tag_siblings_all_ideals': result = self.modules_tag_siblings.GetTagSiblingsIdeals( *args, **kwargs ) File "hydrus\client\db\ClientDBTagDisplay.py", line 443, in GetSiblingsAndParentsForTags tags_to_service_keys_to_siblings_and_parents[ tag ][ service_key ] = ( sibling_chain_members, ideal_tag, descendants, ancestors ) KeyError: 'filename:fztgktyamaaulyy'

>>19843 So it works great when downloading from a single post on twitter, and i've gotten the hydrus importing to work properly, but when downloading from my twitter likes or a media tab it doesnt add the url in the json file. Is a different postprocessor needed for that or is it just not possible and i have to do it one by one?

If I get multiple results from a parser is there a way to just concatenate those together?

>>19855 You're right, it looks like the postprocessor event should be changed. https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst#metadata-event I tried changing it from "event": "init" to "event": "post" and it worked for both media timelines and single posts.

>>19850 >I'm planning to download quite a bit right now so once this happens I'll have to figure out how to go back and do this either way. I'd like to get everything I can now if possible. That's why I would suggest using gallery-dl, it puts it in a nice json format that you'll probably be able to automatically import once the tech is in place. Likely easier than fiddling with previously added notes in the GUI. >I don't know anything about the parsers so idk how hard that would be to do. I tried a bit, but it looks like the best I can do is putting all the translation notes into a single hydrus note with no information about where the boxes are. And they're in an arbitrary order not related to their reading order. Having each translation note become a separate hydrus note is not possible simply because the names of notes in the parser has to be decided ahead of time, and if a parser makes multiple notes with the same name, only the last one is kept. Pic related number 2. Perhaps hydrus dev could update parsers to be able to determine the name of a note with the parsing formula.

>>19858 I really don't want to involve gallery-dl just because it makes this even more complicated. I also decided to try and make my own parser starting from your description one and seem to have ended up with basically the same thing you did. I did manage to keep all the notes by making a separate parser for each (so parser for note 1, parser for note 2, etc.). Mine supports up to 10 notes + description and I attached it just for shits. There might be a way to sort them using the regex and string selector but I didn't really try since I could probably only sort them by x or y not both. I think it'd be good if note parsers would just concatenate all their results together at the end into the single note. Not only does the page say you should only have one text result but if you have multiple they just over-write each other so why is that even possible? Regardless thank you for the effort. https://anonfiles.com/u5ZeE7y5z3/e621_notes_parser_png

>>19859 >I really don't want to involve gallery-dl just because it makes this even more complicated. If one day the hydrus tech is in place to have notes with a position, it'll be a pain in the ass to manually turn your notes into those. That's why I think it's worth it to download the notes with gallery-dl now and keep those json files around somewhere to use them later.

>>19859 >Not only does the page say you should only have one text result but if you have multiple they just over-write each other so why is that even possible? I think it's done like this so that if a website updates and a parser that previously produced a single note suddenly starts producing 100 instead by accident, the end user won't end up with a file with 100 notes. You could imagine a parser that gets description text accidentally grabbing every comment on the image as well, for example. But yeah, if there was some way sort of like a compound formula of just saying "if there are multiple results, concatenate them like this" that would be very helpful.

Also, hydrus dev, it would be useful if sidecar importers had compound formulas like parsers. Weirdly, unlike parsers, I found that sidecar importers that produce multiple notes with the same name won't overwrite the previous note but instead add a new note with a number appended. As in: the first one will be called "note", the second one will be called "note (1)", then "note (2)", etc. (I'm not sure if this is intended behaviour.) So I thought I could use sidecar importers to deal with the issue. But sidecar importers don't have compound formulas so I can't put together the x, y, width, height, and text body entries into a single note.

>>19857 That did the trick, thank you very much.

>>19858 >>19860 >If one day the hydrus tech is in place to have notes with a position, it'll be a pain in the ass to manually turn your notes into those. My plan in that case is just to re-download them using whatever new hydrus feature and then delete the notes. Related to this I've noticed in my testing that notes aren't removed if you delete the file they're associated with. Is there a setting to make that happen or at least a way to find orphaned notes?

>>19846 Thanks a lot for the guide, both the notes and the URL's in the json file were successful imported to hydrus.

>>19864 >My plan in that case is just to re-download them using whatever new hydrus feature and then delete the notes. No true hoarder believes that things stay on the internet forever! The golden rule of backups is "one is none, two is one"! Artists request takedowns of their images from e621 all the time.

>>19866 Entirely correct however because notes are basically just translations even if something gets removed before an official hydrus notes feature comes out it would just require it to be re-translated and I won't have lost anything. The extra hassle of using multiple programs just isn't worth it for translation notes. Plus I do have my parser now and even if it's a bit stupid it does capture notes.

I had an ok week. I didn't get to the larger work I wanted, but I managed to finish a mix of small stuff. There's some bug fixes, UI quality of life, and an important crash fix for macOS. The release should be as normal tomorrow.

So what's the strategy for downloading NSFW from Twitter now? I used to have a subscription in Hydrus to auto-download liked tweets. Is there a way to get something similar in hydownloader/gallery-dl? I can only seem to get subscriptions for profiles, but not likes.

>>19854 It seems like it had something to do with the siblings getting messed up somehow. Regenerating the tag storage mapping caches fixed the issue.

https://www.youtube.com/watch?v=x33quKIyVu0 windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v533/Hydrus.Network.533.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v533/Hydrus.Network.533.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v533/Hydrus.Network.533.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v533/Hydrus.Network.533.-.Linux.-.Executable.tar.gz I had an ok week. There's a mix of different work and a fix to the recent macOS crashing. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html all misc this week Subscription popup windows should update far more smoothly and frequently this week. Some of the 'should I check for new text?' logic was broken when I embedded the popup toaster into the main UI a few months ago--it is fixed. If you right-click an image that is larger than 1,024 pixels on a side, the share->copy menu now has a second 'copy bitmap' that crops the image down for easier and faster pasting in lookup services like SauceNAO. I added some info about the not-well-documented tag/predicate mouse shortcuts (e.g. shift+double-click to edit system predicates) to the help here: https://hydrusnetwork.github.io/hydrus/getting_started_searching.html#editing_predicates I added a link to the Client API help for a neat new tool that finds video duplicates and queues them up in the duplicate filter. Here: https://github.com/appleappleapplenanner/hydrus-video-deduplicator Since v529, at least some users of the macOS app have had terrible crashing. Last week's attempt didn't do it, but I think we've fixed it now! It was something I never even knew about updating in the background to an incompatible version. Thank you for your patience and help testing, and let me know if you still have any trouble! If you are an advanced downloader maker, check out the changelog regarding sub-gallery urls. Nothing huge has changed, but they'll inherit and parse tags and some other stuff from their parent parsing step now. The experimental QtMediaPlayer now initialises with and updates according to the volume and mute settings. Previously it was stuck on 100% always. The volume bar is still hidden on the actual player, but we are moving forward. next week I failed to do as I planned this week. I'd hoped to set aside a good whack of time to figure out a PTR tag filtering system, but it didn't happen. I'll see if I can get the time and energy together again this week, and if not, I'll try and kick out some ratings in the Client API instead.

>>19871 >I added a link to the Client API help for a neat new tool that finds video duplicates and queues them up in the duplicate filter. Hell yes.

So I'm trying to get Twitter notes to import with Hydownloader in the same way that the twitter parser does (tweet text:some text), but I'm kind of stumped. I have the g.notes function working but when I try to import the note I get "Title: tweet text:note text, content length: 0" and no notes get added. First of all I don't want the "Title" namespace if at all possible, but more importantly I'm really confused by the "content length: 0" message and I think that's the main problem. Does anyone have any tips for me? Pic related is the function.

>>19810 It's not a reddit aggregator, no. It gets it's content by scraping a bunch of different sites ranging from e-hentai, twitter, rule34, pahael, pixiv, deviantart, as well as basically every other booru in existence. It's still a pretty young booru, and as such it's still in the process of importing a lot of it's content from other sites. I assume that the recent reddit fiasco made them re-focus their efforts on importing everything they could from reddit before the site implodes in on itself, which is why you mostly see reddit as a source for all recent pictures. I'll provide some examples for non-reddit, and actually tagged art below. https://booru.plus/+rule6708531#c=azbesczc3ph3&o=top https://booru.plus/+rule4051426#c=azbesczc3ph3&o=top https://booru.plus/+rule1635361#c=azbesczc3ph3&o=top https://booru.plus/+sfm16459#c=azausmzf3xyb&o=top https://booru.plus/+gardevoir15416#c=azausmzakp8ut&o=top https://booru.plus/+nia5345#c=aza3unzcyqqq&o=top

>>19874 PS.: I'm just taking a guess here, but I'd assume that the difference between timestamp and newest would probably be import-time vs actual-creation-time

>>19871 Really hoping for that PTR tag filtering. 3rd time getting a 30 day ban for "sankaku url spam" for no apparent reason. I've petitioned for removal of everything tagged *sankaku* multiple times but it still keeps happening. I also have *sankaku* blacklisted in the dupe filter. I don't particularly mind the ban and I get why it happens (even if I do think a full month is excessive). I just don't get why it keeps happening. It's trivial to bypass the ban but I kinda don't care at this point. The janitors will just have to deal with my inevitable 1m+ tag dump later on.

>>19871 How easy would it be to add either a new string processing step or add an option to the existing string select to concatenate some or all of the results into one output?

It's over https://github.com/zedeus/nitter/issues/919

>>19878 F But there is hope. This fork is working: https://github.com/PrivacyDevel/nitter/

I don't suppose there'd be any interest in making it possible to store media in a blob table in the database? That requires some tuning to retain good performance (mostly related to the in-memory cache size), but it'd make backing up a lot easier and partial/incomplete files from crashes and other annoyances would be gone. SQLite (https://sqlite.org/c3ref/blob_open.html) can do incremental IO in blobs (so can others) and Python (https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.blobopen) actively supports it.

My db is seemingly empty..? Help. Context: I moved hydrus versions. I previously had the manual installation running, but I wanted to have updates without the annoyance of constantly downloading a new version, so I decided to move to the chocolatey.org package of hydrus instead. This cause some strange stuff to happen. I previously had hydrus installed on E:\Hydrus Network but chocolatey installed a completely fresh install of hydrus to C:\Program Files\Hydrus Network. Just to make things more clear: old hydrus install: E:Hydrus Network new hydrus install C:\Program Files\Hydrus Network I backed up my current db and imported the backup to the newer hydrus installation located on my C: drive To my great surprise everything worked! Problem is... I now have no idea where my db is... Cus it isn't under my newer hydrus installation... but it couldn't be under the old hydrus installation either since when I sort by the date of files changed, the latest file modification happened yesterday (when I did this migration), instead of today, when I did a bunch of work on my hydrus client mass-tagging and configuring sibling tags and parent tag relationships. Just to make it clear. Everything seems to be working 100% fine... I just don't know where my db is... ..And more importantly... I don't know how to safely migrate my db to be under my new hydrus installation without losing data Actually... I don't even care to migrate the db if I don't have to. I just want to know where the fuck it is so I can configure the periodic backups for it. PS.: I don't remember whether when asked to import a backup db, I chose my at the time live/actual db, or it's backup copy, which makes things a bit more complicated... PSS.: for extra context I have attached a screenshot of the "migrate database" window and to where it says my files are. PSSS.: the first picture is a picture of the db directly under my newer hydrus installation PSSSS.: Sorry for the many "PS.:"-es lol

>>19881 I would guess you didn't actually move the data at all. Your DB is pretty big so it should be hard to miss... does your C: drive even have 650GB of space total? Anyway first thing I would get something like windirstat then run it on all your drives and identify all the copies of your database. If there's only one then mystery solved.

>>19734 This issue still persists. >>19741 works to prevent the issue, but putting the file ID at the end looks ugly, and de-anonymizes myself as few other people do this. However, since Hydrus treats certain symbols like "-" and "_" as spaces for searching, but not for anything else, you can just slap some of those in there when you want to retain the same filename for multiple files for whatever reason.

is it okay to NOT back up the thumbnail folders? i assume hydrus will automatically generate the thumbnail folders and .thumbnail files if they don't exist. i use freefilesync to mirror drives and having to scan through 5 millions of thumbnail files every synchronization just takes too long. i just don't find backing up a bunch of thumbnails worth it

>>19848 >Actually, that's a clipboard copied straight from Hydrus using share->copy->image (bitmap). When copying from browser, it pastes just fine, but won't find it at distance 0. Damn, thanks--I read your post wrong! Ok, I've downloaded your archive, I'll see what's going on here this week. >>19849 No, no neat way. I think I'm going to remove those messages--as you say, they aren't useful and just end up being spammy. >>19850 Just as background to this, yeah, I've been talking with some guys over the years and what we'll probably settle on is storing a semi-hidden 'translation JSON note' on the file, with the JSON coordinate and text gubbins, and I'll hack together a way for hydrus to show the boxes the same way as danbooru et al do it when that data exists. Feel free to figure out whatever you like in the meantime, but anything too complicated will probably need the Client API. >>19854 >>19870 Thank you for this report, and sorry for the trouble! I'll fix it up so it fails better in future. >>19856 It depends on exactly what you are going for. If you want to concatenate different things, like 'https;//site.com' with '/post/123', then look at the COMPOUND formula type in your content parser. It does regex-style \1 \2 substitution. If you are parsing 'a' 'b' 'c' in one formula and want to join them up, I don't think there's a great way to do this yet. The 'String Splitter' in the String Processing system does the reverse, but I don't have a 'String Concatenator' yet. Good idea though! >>19862 I'm not really happy with the sidecars UI, and some elements of the implementation. It technically works, but it is a mess. The note overwriting/renaming issue is tricky. I have some complicated 'note import options' logic in importers, and for sidecar importing I have it set to some basic/simple defaults that essentially say variants of 'if the incoming note is the same as the last note but with more text on the end, then overwrite, otherwise rename with 'name (x+1)' format. I could expose this 'note import options' if you like, and let you change the behaviour, but I'll warn you that this can turn into a rabbit hole. That's interesting about the compound formula idea, to combine things back together. Yeah, unfortunately, my sidecar system can't support that, and I don't think I'll add it soon. I already overengineered the whole thing, and it is groaning under its own weight with a whack UI. I can't add more cross-sidecar combinations right now--I first need to wangle things into a nice tree structure UI or something, just anything to stop us falling into nested dialog hell yet again.

>>19869 Just as an aside, with today's news, I have to assume we are in the complete fuck-zone now. I'm going to remove our limping-along twitter downloaders from the defaults for the next release. We'll hang on to whatever nitter and gallery-dl can patch together, but for now, twitter is a no-download zone, I think. I'm crossing my fingers the japs who only post to twitter will finally consider migrating to something else. >>19874 Ah, thanks! >>19875 Makes sense, too. I've got a crazy week coming, but if I find the time, I'll have a proper look here. >>19877 Yeah not too difficult, I think. I'll have a think, and I might be able to roll out something quick. It'll only work within a single parsing object, though. It won't be able to cross multiple formulae or string processors or sidecars. The pipeline just isn't set up to stitch together things so separate. >>19880 A million years ago, hydrus did work like this. Fundamentally, I found it reduced performance, including by forcing db access to compete with file access. I know that modern SQLite articles, maybe somewhere in your links there, suggest they perform faster than NTFS etc..., so perhaps this is a thought revisiting sometime, but probably not for hydrus. Unless you are talking about clever incremental db-to-db backups (which is its own kettle of fish), I'm afraid this does not make backing up easier. If I store my 7TB of files in one client.files.db database file, and I import one 78KB jpeg, that whole file needs to be backed up. In the current file system storage, I only need to update that one new file. Now, maybe a way around this would be to split the database into 500GB pieces, in order to avoid having to back everything up on a single change, but then if we are going in that direction, why not break it up into three million pieces and store files separately? So, I'm not sure. Maybe quick thumbnail access might benefit. I'm not really sold on the benefits vs the drawbacks though. I'm fine with file access latency right now.

>>19881 >>19882 No worries. It is probably in C:\Users\You\Hydrus or similar. Hydrus falls back to that location if the first choice is read only, and I think chocolatey forces it somewhere like that too since it doesn't like writing to its own directory. Hit up file->open->database directory and/or help->about to see your install dir vs db dir. >>19883 Sorry, I missed this the first time around. I'll give it a proper look. >>19884 Yeah, it is ok. The client will moan if you boot without existing thumbnail locations, but it gives you the option to recreate the folders and just let it regen. If you do this and ever had to restore your database, then whenever it loaded files for the first time, the thumbnails would each come in 50-300ms late since they'd have to be regenerated on the fly. It isn't a big deal, and you can set up file maintenance to catch up in the background, but remember 5 million is an unfathomable number. It is less hassle to wait a couple minutes to scan than to add constant latency-inconvenience or hours and hours and hours of heavy CPU and HDD access if you ever need to recover. Good compromise is probably to make two scripts and back up your thumbs only every three months. Hydrus is fine with a thumbs directory with a few holes, so it isn't a big deal if it is a bit out of date.

>>19885 Not the guy you were responding to, but with regards to the translation note how will it interact with the duplicate filter? It would be really annoying to have to readjust them for every image with different dimensions.

>>19887 thank you that worked. responding to make sure others who search for this problem/have this problem can have confirmation. plus it might be good for SEO too.

>>19885 >>>19856 >It depends on exactly what you are going for. If you want to concatenate different things, like 'https;//site.com' with '/post/123', then look at the COMPOUND formula type in your content parser. It does regex-style \1 \2 substitution. If you are parsing 'a' 'b' 'c' in one formula and want to join them up, I don't think there's a great way to do this yet. The 'String Splitter' in the String Processing system does the reverse, but I don't have a 'String Concatenator' yet. Good idea though! >>19886 >>>19877 >Yeah not too difficult, I think. I'll have a think, and I might be able to roll out something quick. It'll only work within a single parsing object, though. It won't be able to cross multiple formulae or string processors or sidecars. The pipeline just isn't set up to stitch together things so separate. https://anonfiles.com/d4k9U6z1za/Screenshot_2023_07_02_18_38_28_jpg Yeah, I just want a way to have both/all of these results added to "e621 note 1" nothing crazy. I think the easiest thing would be to simply have both/all the results you see in the screenshot get automatically concatenated. Right now they just over-write one another so the last one gets picked which isn't clear and makes the warning about only parsing one thing confusing. I also think this makes sense because I can already filter which ones get picked easily in the string operations thing. It would give me what I'm looking for, fix the "conflict hell", AND it doesn't break any parsers as long as they weren't selecting multiple things here which they shouldn't be anyway.

quick question, is there any way to move the top bar here on the side or make it permanently on top of images? cause it's been driving me insane and i've looked everywhere to try and fix it but there just doesn't seem to be an option for it and having to move my mouse for every single image i sort to see their size is a colossal waste of my time and has resulted in me saving a shitton of low quality images

Is there an add-on for firefox that generates a cookies.txt in the same format as hydrus companion? I don't wanna keep a second browser around just for hydrus companion when all I even need from it is the cookies.

I've been fucking around with Hydrus on Fedora Siverblue with GNOME as my desktop. I made a toolbox (Silverblue doesn't like installing any non-default programs to the system, so you can make environments called toolboxes that pretend to be regular Fedora and can have programs installed to) in which I installed ffmpeg (via RPMfusion) and mpv-libs; doing this and setting up the venv lets Hydrus launch. Weird GNOME stuff: - mpv kind of works, but it spawns two windows that aren't embedded. I didn't do extensive testing but it sounds similar to the Qubes thing that was going on in one of these threads - I frequently get popups that say "python 3: main -- hydrus [version] is ready" when Hydrus opens a new window (options, page selector, etc) - Enabling dark mode (both in Hydrus and in system settings) and changing style to a darker one works, but the window border is still white - My cursor is slightly smaller and the bottom is cut off when it is over a Hydrus window. This might be related to fractional scaling in GNOME being in beta though Weird Silverblue stuff: - The user's home folder is not /home/user, but /var/home/user. Hydrus seems to assume it's always /home/user, which can cause issues when copying paths if the db is under the home directory and exporting files within the home directory That's all I've noticed so far. Thanks for making Hydrus, hopefully this post will be useful.

I had a good productive week. I fixed many bugs, including some bad bitmap pasting and the worst media viewer note layout issues, improved some quality of life, and added file rating viewing and editing to the Client API. There’s also some user work, including .sai2 file support. The release should be as normal tomorrow.

https://www.youtube.com/watch?v=wpUEdGDdaTQ windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v534/Hydrus.Network.534.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v534/Hydrus.Network.534.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v534/Hydrus.Network.534.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v534/Hydrus.Network.534.-.Linux.-.Executable.tar.gz I had a good week. There's lots of smaller fixes and improvements, and ratings are added to the Client API. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html highlights Thanks to a user submission, hydrus now recognises and supports importing .sai2 files! Basic metadata for now, but I'll see about resolution and thumbnail support in future. It turns out PNG and WEBP files can have EXIF data, and our EXIF scanner works on them! All newly importing PNGs and WEBPs will be scanned, and on update, you'll have a yes/no dialog asking if you want to scan your existing PNGs. If you have a gigantic database, or your files are on a remote cloud storage, you might want to defer it, but it isn't really a big deal. I banged my head against the media viewer 'notes hover' layout code, and I was actually really successful this time. Many of the borked layout issues are fixed, and I think I've completely fixed the overlapping hover problems in the duplicate filter. I fixed a bunch of weird bitmap paste problems in the new 'similar files' paste button. Let me know if you still have any issues! Just a small thing, but the 'loading' page in hydrus, when you refresh a search, is no longer grey. It is the same colour as the normal thumbgrid, so the loading flicker is gone. Twitter is gone from the downloader defaults for all new users. All existing users should look to migrate away, too. I hope to roll out some downloader engine improvements in the near future that'll make nitter (on multiple domains) easier to set up, but there aren't a lot of excellent solutions. I also added ratings to the Client API. The service object now talks about min/max stars and star shape, the file metadata call gives every file's ratings, and you can set and clear ratings too with a new command and permission. It is all fairly simple, but there's several possible data types depending on the rating type, so check out the Client API help for details and examples. next week I regret that I again did not find time for PTR janitor workflow improvements this week. Now that ratings on the Client API is clear, this is top priority. Depending on how things go, I may take a two-week release, since I definitely want some solid work done here.

>>19895 >I'm bumping the default distance on this search predicate up to 8 now, to catch the weirder instances So it's not fixable?

>>19893 Tried Fedora Kinoite (same thing but KDE instead of GNOME). Still have the mpv issue, but none of the other issues I mentioned in the GNOME portion. I also tested the QtMediaViewer thing, it instantly crashes when attempting to view a video (I may be missing some dependency though, it complained about gstreamer not being found). One thing I forgot to mention that affects both GNOME and KDE is that the program icon and title for Hydrus doesn't appear. It shows as the default icon and has the window name python3 when I mouse over it on the task bar, but when I open it and look at the title bar of the program it shows "main -- hydrus 533". That may be related to Wayland, I don't remember having this problem when I used XFCE a few months ago. I also wonder if mpv not behaving is a Wayland thing too.

>>19895 Is it intended behavior for "open in a new page" to show this in the search panel? previously there would be no search panel when files were opened in a new page

>>19898 Yeah it was added a few updates ago. It's so that you can perform searches on the results. Very helpful change for me.

>>19895 >when you drag and drop thumbnails out of the program while using an automatic pattern to rename them (options->gui), all the filenames are now unique, adding '(x)' after their name as needed for dedupe. previously, on duplicates, it was basically doing a weird spam-merge Thanks. I no longer have to go through making sure all filename tags are unique.

Sooo.... does anyone here know of a good way of tagging un-tagged hydrus files? I have a bunch of them already imported into hydrus and I deleted the originals from their folders: I tried to use https://github.com/nostrenz/hatate-iqdb-tagger but every time I click on the "Query Hydrus" menu it crashes the app with an error log saying something along the lines of not being able to convert a "Newtonsoft.Json.Linq.JObject" to "Newtonsoft.Json.Linq.JArray" I could manually add each and every 256 folders hydrus creates but that is just extremely wasteful as it means re-checking a bunch of already tagged files, as well as screenshots and random memes, stuff which I know iqdb will not be able to find. I'm kind of bummed as to what to do. I'd really like to get this going as soon as possible, since fetching all those tags from iqdb or any other site is going to take so long anyway, that I could be doing a lot of other stuff in the meantime. Currently I feel like I'm just banging my head against a wall. I even tried submitting an issue on github but from what I can see there hasn't been any activity on the repository for 5 months now.

>>19901 >does anyone here know of a good way of tagging un-tagged hydrus files? Manually, like a real man. If they're unarchived, just go through your inbox. If you made the mistake of archiving untagged files, then system:untagged. If you partially tagged them, then good luck to you as all I can come up with is "systems:number of tags < x" were x is arbitrary depending on how many false positives you want to deal with.

>>19888 Any system that tried to handle this stuff would have to try and wangle that sort of thing. Maybe something like converting the json boxes as they come in to store percentage coordinates instead of absolute pixels. It'd probably be a nightmare. Honestly this sounds pie in the sky, but we'll probably get very fast auto-translation AI tech in the next few years and it'd be easier to plug that into any image, generating translation boxes on the fly, than try to bork our way through converting legacy 'hard' translation boxes over. You could even train the AI on the existing booru doujin boxes to get the correct writing style. >>19890 I'll see what I can do with different smaller tools here. Please let me know how they help or not in your specific problems and I'll keep hacking away. >>19891 I think I just talked to you elsewhere, but if not, I'll try and figure out a cog button that spawns a menu for 'always on top' for it and the other hovers, and eventually hang any other options off it. >>19892 I use this: https://addons.mozilla.org/en-US/firefox/addon/cookies-txt-one-click/ https://github.com/Flexa-B/cookies.txt-one-click

>>19893 >>19897 Thanks, this is all interesting. Although there's a ton of annoying things here, it feels like, other than mpv, most of this is normal bug stuff that will, fingers crossed, improve with future updates. The mpv stuff is trickier. I'm still planning to investigate a different way of embedding mpv for various situations like yours. I'm not sure what the /var/home/user problem is about. If you are using the Arch package or whatever as your base, maybe they hardcode that in the code or some config somewhere? I do a test on boot to see if install_dir/db is writeable-to, and if not, I fall back to ~/Hydrus. If that is expanding wrong, I guess that is python or your OS giving a weird value back? But if you are getting issues on copying/exporting paths, I am really not sure what's going on there. Any chance there is a symlink here and I'm handling it wrong? I'm sorry to say I'm not enough of an expert to talk too cleverly. >>19896 I improved the situation, and I got some bad test files matching at 6 instead of 8, but I figured I'd just move it up anyway to catch other weirdness. It seems like many other programs produce slightly different bitmaps for whatever reason. I wouldn't have thought some ICC profile could produce such a radically different perceptual hash, it almost feels like what would happen with a movie grain filter over the image, so I suspect I have misunderstood something. Maybe I am still screwing up some of the bitmap conversion here, although I don't know how it would be close. When this stuff goes wrong, it tends to completely go nuts into a fully static image etc... Or you get channel swaps, BRG instead of RGB, but my phash system is colourblind so that shouldn't matter. Let me know how it works for you in future! >>19901 If you haven't checked out my and others' thoughts on tagging in the help docs, it might be useful as background reading. https://hydrusnetwork.github.io/hydrus/getting_started_tags.html https://hydrusnetwork.github.io/hydrus/getting_started_more_tags.html https://hydrusnetwork.github.io/hydrus/tagging_schema.html (particularly the 'protip: rein in your spergitude' section--this is my golden rule) https://hydrusnetwork.github.io/hydrus/PTR.html https://hydrusnetwork.github.io/hydrus/petitionPractices.html If you have a lot of booru images, a powerful computer, and 70GB of SSD free, the PTR is a great option. If you have custom images to tag, it isn't. There's also the hydrus-dd tagger, which talks to the client in a variety of ways and uses AI magic: https://gitgud.io/koto/hydrus-dd Some hydrus users are working on similar tech with all the newer AI stuff going on. I expect the coming years to add several new options. For now, if you hate tagging as much as me, and there isn't a good automatic solution available, then just focus on stuff you actually search with. I mostly just do character and series tags.

>>19904 >rein in your spergitude Never. just 7600+ files to go in addition to whatever I download in the remaining months.

>>19904 >If you have a lot of booru images, a powerful computer, and 70GB of SSD free, the PTR is a great option That would ALMOST be a solution to my problems, however I'm concerned about the fact that many of the files I'd want to tag are most likely not the highest-quality version available on most boorus, and so I doubt that their file hashes would be able to match the one in the PTR. Plus it also wouldn't allow me to download the higher quality equivalents since as far as I know the PTR contains file hashes, and tags only, not source links.(and I don't know if hydrus even has a function to mass-redownload from known sources anyway.) Thank you for the help regardless. Can't wait for the day this stuff can be done by AI for us. >>19902 >If you partially tagged them, then good luck to you as all I can come up with is "systems:number of tags < x" I actually already tried that. In fact I made a pretty good query for this exact reason that could serve my purposes, which is why I'm so disappointment that the hatate-iqdb-tagger doesn't work. There's about 50,000 files I'd need to tag if I were to want to do it manually, which is just... I have better things to do is what I'm trying to say.

>>19901 >>19906 Have you tried dragging and dropping from hydrus onto hatate? At least that worked for me back when I imported my stuff to hydrus.

to anyone who uses the e-hentai and exhentai downloader, a major change was made to the "download original image" feature. Now, for any galleries that are 1 year old or older, downloading the original images is no longer free. It always costs GP to download images from those galleries. If you use the downloaders for them now, it will silently spend all your GP, then just start failing every download from then on. This change was just implemented, so there's a small chance it could be reversed, but in a forum thread about it, admin says that it's unlikely this will change because he doesn't know of any other way to stop IP hoppers from overloading the original-image servers' bandwidth.

>>19906 >That would ALMOST be a solution to my problems, however I'm concerned about the fact that many of the files I'd want to tag are most likely not the highest-quality version available on most boorus,... You never know until you try! I've been going finding better quality copies of things and merging the tags as with the duplicate finder. And I've manually tagged 10s of thousands of files by hand as well. And I'm far from the only one. Just make a new database, add the PTR, and copy all your current stuff. Then see if PTR tags make a significant difference or not to the amount of untagged files. If it does, great! If not, either scrap it and go back to your original database, or keep moving forward with the PTR and help others like you who may have untagged files. Also do not underestimate to duplicate finder. It can be configured to merge tags between the higher and lower quality images. As you work through any duplicates you have you could be tagging some of those low quality ones you have. For details on that see https://hydrusnetwork.github.io/hydrus/duplicates.html

>>19908 >exhentai downloader, Wait, that already exists?

Hey hydev, I just had an empty tag error when uploading a chunk of stuff to the PTR. Somehow a tag that I think was either "::" or ":" snuck in. I tried running a "fix invalid tags" job and it didn't find the tag, I eventually just found it through loading a bunch of images and scrolling through the pending taglist. IDK how the invalid repair function works, but that seems like a bug or an oversight to me.

>>19908 This only happens if you download during peak hours. I always have 0 GP, leave the downloader all day and retry failures every now and then

>>19912 no. This change was just made 2 days ago. That's how it used to work. Now, the download will always cost GP if the gallery is a year older or more. There's no longer any way to get the original image download for free.

>>19904 >I'm not sure what the /var/home/user problem is about My bad, it was a flatpak issue. I was trying to open the file's path in a program installed via flatpak which has limited access to my home directory under /var from its perspective. I'm running Hydrus from source, so it is in and can access /home/user just fine, you haven't done anything wrong. I'm just new to flatpaks and didn't realize what was going on. Exporting was also me being stupid, I had a bad filename pattern that caused the files to start with a . and thus hidden.

>>19901 I guess it depends on what you keep in Hydrus. My client is exclusively porn, and I do lots of manual imports of higher quality and/or uncensored versions of booru images that I grab from kemono. If I'm lucky I can download the inferior versions with tags from a booru and transfer the tags, but even that can get tedious especially when the booru has a single image from a set of many variants. When I manually tag, I only use specific sets of tags that I know I search for. Artist, characters, sex positions, breast size, etc are much more useful for me than hair color or the background of the image for example. This makes tagging faster of course, and I usually do some manual tagging even for images with booru tags because they often don't quite fulfill all my requirements. I also make heavy use of tag siblings and parents to make using booru tags less of a pain. For example, the concept of "girl but with a pp" goes by a different tag on every single booru site it feels, so I flatten them all into one. Parents are good for things like making vaginal imply vagina and so on, less manual adding of redundant tags. I've also been considering taking lots of tags that I don't care about and making them siblings with something like "meta:useless tag" in an attempt to make hiding/deleting them easier.

>>19886 >Unless you are talking about clever incremental db-to-db backups (which is its own kettle of fish), I'm afraid this does not make backing up easier. If I store my 7TB of files in one client.files.db database file, and I import one 78KB jpeg, that whole file needs to be backed up. In the current file system storage, I only need to update that one new file. Differential backup of the db file itself would also be an option, but yeah otherwise it's more complicated synchronization (I think the differential option is much less likely to end-up with bugs, just a reflinked copy when the database is inactive & one can leisurely back the copy up afterward). >Now, maybe a way around this would be to split the database into 500GB pieces, in order to avoid having to back everything up on a single change, but then if we are going in that direction, why not break it up into three million pieces and store files separately? I'd say the consistency benefits for the downloaders would be nice to have. Splitting though would involve unpleasantness to handle multi-db queries (through sqlite schema features) or hacks that would involve even more complexity. >so perhaps this is a thought revisiting sometime, but probably not for hydrus. A bit unfortunate, but understandable.

>>19915 >Artist, characters, sex positions, breast size, etc are much more useful for me than hair color or the background of the image for example. You don't like to single out the gingers, blondes, and alternative hair colors? My autism is too great. I need hair length (short, long, very long), style, and color. I have to be able to find the twintalls, buns, ponytails, drills, et cetera. Oh ,and colors also include type of multicoloring such as streak, split, fade, et cetera. Hair can also be made of an unusual substance like tentacles, slime, snakes, fire, wires, vines, and more. There's also simulated hair made of other body parts, like with robots, or gardevoir's natural helmet, or lopunny's bunny ears that are clearly supposed to be twintails, but I just tag the style they simulate. Maybe I should have a "faux hair" tag cover all that? >Parents are good for things like making vaginal imply vagina Wait, you have a vagina tag implied by vaginal? How does that work? Do you mean a tag just to imply a character has a vagina? Wouldn't female/futa/cuntboy tags do that on their own and be necessary regardless? I hope you don't mean that it's supposed to be for a visible pussy, because I separate that into an "exposure:*" namespace because there's lots of things where penetration of a hole or holes is clearly happening, but not visible either in the frame, at an angle, or because something is in the way.

>>19917 Maybe hair color/length wasn't the best example, but still I don't care that much. I usually care much more about the sexy bits or characters than the hair. Tags like hair color are more useful for trying to find one specific file that you can't remember the character's name, so I consider them more of a tag to be used in booru searching than one I would use on my curated collection in Hydrus. >Wait, you have a vagina tag implied by vaginal? How does that work? If a file has the tag 'vaginal', then that must mean there is a vagina in it. So I made a tag parent where any file with vaginal gets the tag vagina. >Wouldn't female/futa/cuntboy tags do that on their own and be necessary regardless? Yes, but it helps find mistagged files. If I search 'vagina -female -futa -cuntboy' etc and some files come up, I need to fix some tags. >I hope you don't mean that it's supposed to be for a visible pussy I only use the 'vaginal' tag when it's extremely obvious that a vagina is getting stimulated, which almost always means that it's visible in some way or another. Maybe there's an edge case where you can only tell via x-ray and the pussy isn't directly visible but I don't care that much. I use the tag 'ambiguous penetration' when it could be vaginal or anal but you can't tell because of the angle. I have lots of implication overlap. My autism lies in describing the sex act, so I use tags like "penis in vagina" which implies sex and penis and vaginal penetration (-> vaginal -> vagina), then "male penetrating female" which implies male penetrating (-> male) and female penetrated (-> female). So by adding just those two tags I don't have to manually tag vaginal, vagina, penis, male, female, or sex. Random aside, but it would be nice if there could be numbered namespaces. Instead of having tags like '1 female' '2 females' and so on you could just have 'females:#' and run searches like "male:1 female:>=2".

>>19918 >Random aside, but it would be nice if there could be numbered namespaces. Instead of having tags like '1 female' '2 females' and so on you could just have 'females:#' and run searches like "male:1 female:>=2". this already exists. Check out "sytem:tag as number"

>>19917 >My autism is too great. Join the club! >Wait, you have a vagina tag implied by vaginal? How does that work? Basically you make a tag automatically add associated tags. Pic related. It simplifies tagging quite a bit. You can of course create your own parent/sibling tags, but the PTR has a shitton already baked in if you're lazy. Be sure to make use of the "related tags" feature. This will allow you to do just the basic tagging, and pull up a list of tags frequently used together with either a specific tag, or all the tags of an image. For Example: if you have an image of Asuka Langley. Tag your image as character:asuka Langley, then switch to the related tags tab. You'll find the most common tags associated with that tag such as red hair, blue eyes, plugsuit, bodysuit, and hair ornament (among others). Just add the relevant tags from the list. Once you have a decent amount of tagging done (or if you use the PTR) pulling tags from the related tags does about 85% of the work for you.

>>19918 >Yes, but it helps find mistagged files. If I search 'vagina -female -futa -cuntboy' etc and some files come up, I need to fix some tags. That redundancy would be too much for me. Labeling character sex and character count are near the top of the list of tags I apply to untagged files and I practically never miss them. > I use the tag 'ambiguous penetration' when it could be vaginal or anal but you can't tell because of the angle. I probably should have done that, but instead I just headcanon'd which hole was filled when ambiguous. >>19919 Fuck, I should have used this. I have everything tagged "1girl-10girls" in keeping with common booru practice, and any more than that I just tag "many girls". >Basically you make a tag automatically add associated tags. Oh, I know about parents and such, I was more asking what the meaning of his "vagina" tag is, since I tag things by explicit exposure and "vaginal" doesn't necessarily imply a visible vagina. >Be sure to make use of the "related tags" feature I need to start doing that.

>>19921 >I practically never miss them I find that booru tags often suck (tagging futas as both girls and futas just off the top of my head) and so I sometimes remove them and redo them manually. I'm considering disabling grabbing tags from boorus in the first place since so many tags are useless to me and just clutter the tag list up. >>19919 Oh shit, nice. Time to make siblings for all of my "# sex" tags. I feel like there's tons of these little features that I just missed, would be nice if there were a compilation of them. >>19920 >vaginal masturbation (50,870) Are you preparing for the collapse of the internet so you can single-handedly restore all of the porn once it's back up?

>>19922 >I find that booru tags often suck I entirely ignore booru tags, except for creator/artist. They're not necessarily consistent with each other and I prefer tagging on my own terms as that is what I'm likely to search. I also don't like siblings much. Feels messy. I want my own set of ideal tags only. >Are you preparing for the collapse of the internet He has me beat for autism by a long shot. He has nearly twice as many files as I have in total just of pussy masturbation.

one day I'll get this one day

>>19924 Not good enough. Get it yesterday.

>>19922 >Oh shit, nice. Time to make siblings for all of my "# sex" tags. like "1 sex" "2 sex" "3 sex" etc.? what do those tags mean for you?

>>19926 Sex is obviously also a placeholder for the character's sex.

>>19926 It obviously counts the sex visible in the file.

>>19922 >Are you preparing for the collapse of the internet so you can single-handedly restore all of the porn once it's back up? Kind of! It's partly obsessive compulsive data hoarding, partly because things tend to disappear, get taken down, domains disappear, or things start getting bit rot. It's not ALL porn (okay it's still *a lot* porn), a good chunk comes from the late 90s and early 2000s from image boards, forums, and webrings (remember those?). Lots of low quality and jpeg artifacts. Old siterips, back when such a thing was done with a simple crawler. Chat logs from IRC and ICQ, emails, old school projects, scans of generations of family photographs all tagged. Hydrus is basically a database of my entire life at this point. >>19923 >He has me beat for autism by a long shot. Be careful! Hydrus has a bit of a learning curve, but once you start seriously using it you may find that it makes collecting things *too* easy and you too may begin to walk the data hoarder path.

>>19929 I'm glad people like you exist. Don't let that internet history die! I used to be a data hoarder but it was starting to fuck with my general wellbeing. Now I handpick what images I consider the best of the best, things I personally don't want to disappear, and manually tag them instead of scraping sites for any image with tag I felt interested in at the time or importing thousands of personal photos I didn't care about and letting them sit untagged. t. 10.5 GB archive, 1.24 TB deleted

I'm in the middle of a big janitorial petition-processing update for the PTR and other repositories. There's a lot of work still to do, and most of what I have done so far doesn't apply to normal users, so I won't go for a release out tomorrow. I'll just work more instead. v535 should be out on the 19th of July. Thanks everyone!

The following error occurred while reading the pair after starting the duplication processing work, and the reading of the pair was not completed and the duplication processing work became impossible. What should I do? v534, win32, frozen ZeroDivisionError division by zero Traceback (most recent call last): File "hydrus\client\gui\QtPorting.py", line 1302, in eventFilter File "hydrus\client\gui\QtPorting.py", line 1275, in Execute File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 3413, in qt_continue File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2947, in _ShowCurrentPair File "hydrus\client\ClientDuplicates.py", line 28, in GetDuplicateComparisonScore File "hydrus\client\ClientDuplicates.py", line 123, in GetDuplicateComparisonStatements ZeroDivisionError: division by zero

Is there a reliable way to get the hash or information about the better image when importing the worse image that has been deleted? In the log under the 'hashes' context menu, only the hash of the deleted file that I just imported is listed. I have been just removing the deletion records and re-importing, and then running the duplicate filter again, but this isn't ideal.

It would be nice if there were a way to bind the advanced deletion menu to some keybind without having to have it enabled in the options. Most of the time I'm fine with just deleting, but sometimes I want to specify a reason or delete without keeping a record. Having the advanced dialog enabled when I don't need it most of the time can get annoying.

>>19933 Why don't you trust the duplicate filter decisions you've made in the past?

Having problems with submitting to the PTR. I think the error below means I somehow have a null tag that's trying to be submitted but I could be way off. Found what appears to be a similar issue on GitHub at https://github.com/hydrusnetwork/hydrus/issues/821 I have done Database --> Check and Repair --> Fix invalid tags which was the solution of the above GitHub issue. It fixed 3 tags out of 38 million. Unfortunately I still have the same error submitting to the PTR. I imagine that I may be able to solve this by forgetting the pending tags, but I'd really rather not have 5,600 tags vanish into the ether if possible. Any help would be appreciated. Error Log: ---------- v534, 2023/07/13 10:51:44: The repository encountered an error it could not handle! Here is a dump of what happened, which will also be written to your client.log file. If it persists, please forward it to hydrus.admin@gmail.com: Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.10/site-packages/twisted/_threads/_threadworker.py", line 47, in work task() File "/usr/lib/python3.10/site-packages/twisted/_threads/_team.py", line 182, in doWork task() --- <exception caught here> --- File "/usr/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda> inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext return func(*args, **kw) File "/opt/hydrus/hydrus/server/networking/ServerServerResources.py", line 1273, in _threadDoPOSTJob HG.server_controller.WriteSynchronous( 'update', self._service_key, request.hydrus_account, client_to_server_update, timestamp ) File "/opt/hydrus/hydrus/core/HydrusController.py", line 934, in WriteSynchronous return self._Write( action, True, *args, **kwargs ) File "/opt/hydrus/hydrus/core/HydrusController.py", line 241, in _Write result = self.db.Write( action, synchronous, *args, **kwargs ) File "/opt/hydrus/hydrus/core/HydrusDB.py", line 945, in Write if synchronous: return job.GetResult() File "/opt/hydrus/hydrus/core/HydrusData.py", line 2069, in GetResult raise e hydrus.core.HydrusExceptions.TagSizeException: Traceback (most recent call last): File "/opt/hydrus/hydrus/core/HydrusDB.py", line 606, in _ProcessJob result = self._Write( action, *args, **kwargs ) File "/opt/hydrus/hydrus/server/ServerDB.py", line 4904, in _Write return self._write_commands_to_methods[ action ]( *args, **kwargs ) File "/opt/hydrus/hydrus/server/ServerDB.py", line 4114, in _RepositoryProcessClientToServerUpdate master_tag_id = self._GetMasterTagId( tag ) File "/opt/hydrus/hydrus/server/ServerDB.py", line 1070, in _GetMasterTagId

[Expand Post]

HydrusTags.CheckTagNotEmpty( tag ) File "/opt/hydrus/hydrus/core/HydrusTags.py", line 186, in CheckTagNotEmpty raise HydrusExceptions.TagSizeException( 'Received a zero-length tag!' ) hydrus.core.HydrusExceptions.TagSizeException: Received a zero-length tag! v534, 2023/07/13 10:51:44: uploading pending to public tag repository public tag repository error v534, 2023/07/13 10:51:44: 45869 GET /add_urls/get_url_files 200 in 2.00 milliseconds v534, 2023/07/13 10:51:44: Traceback (most recent call last): File "hydrus\core\HydrusThreading.py", line 427, in run callable( *args, **kwargs ) File "hydrus\client\gui\ClientGUI.py", line 331, in THREADUploadPending content_updates = client_to_server_update.GetClientsideContentUpdates() File "hydrus\client\ClientServices.py", line 1360, in Request File "hydrus\client\networking\ClientNetworkingJobs.py", line 1977, in WaitUntilDone hydrus.core.HydrusExceptions.ServerException: 500: The server's error text was too long to display. The first part follows, while a larger chunk has been written to the log. The repository encountered an error it could not handle! Here is a dump of what happened, which will also be written to your client.log file. If it persists, please forward it to hydrus.admin@gmail.com: Traceback (most recent call last): File "/usr/lib/ v534, 2023/07/13 10:51:44: Exception: v534, 2023/07/13 10:51:44: ServerException 500: The server's error text was too long to display. The first part follows, while a larger chunk has been written to the log. Traceback (most recent call last): File "hydrus\core\HydrusThreading.py", line 427, in run callable( *args, **kwargs ) File "hydrus\client\gui\ClientGUI.py", line 331, in THREADUploadPending content_updates = client_to_server_update.GetClientsideContentUpdates() File "hydrus\client\ClientServices.py", line 1360, in Request File "hydrus\client\networking\ClientNetworkingJobs.py", line 1977, in WaitUntilDone hydrus.core.HydrusExceptions.ServerException: 500: The server's error text was too long to display. The first part follows, while a larger chunk has been written to the log. The repository encountered an error it could not handle! Here is a dump of what happened, which will also be written to your client.log file. If it persists, please forward it to hydrus.admin@gmail.com: Traceback (most recent call last): File "/usr/lib/ The repository encountered an error it could not handle! Here is a dump of what happened, which will also be written to your client.log file. If it persists, please forward it to hydrus.admin@gmail.com: Traceback (most recent call last): File "/usr/lib/ ----------

>>19936 D'oh! Forgot to mention, this is on Windows 10 x64 that I'm getting this error on.

>>19935 I do, this is for tagging purposes. Typically this is when I find a sequence of images and import them, and see that there is one or more of them that have already been deleted. I would like to have these better quality images together with the ones that have been imported.

4chan /v/'s booru, vidya.pics, uses a new booru software called shimmie. I couldn't find any already existing parsers for shimmie boorus, so I made this one for vidya.pics. Perhaps it can work for other shimmie boorus with minor changes.

>>19911 EDIT: This is probably the same issue as >>19936, in which case the problem is slightly different to what I first thought here--please try again next week and let me know how it goes. Thank you for this report. I'm sorry this didn't work--I thought I had this thing fairly well nailed down by now. You might like to run database->regenerate->local tag cache. Maybe the problem wasn't in the master store, but the cache? If that fixes things and you can't find the bad tag any more, let me know. If it doesn't fix it, if you go to 'install_dir/db/client - date.log', can you find the actual error text and post that? It may have some extra info that will help me out. You are right that the bad tags tend to be some version of '::' or often '[][][][]' spam of the 00 byte, which, when my cleaner removes all the bad characters becomes a variant of the empty tag. In that case, the client is supposed to rename it to like 'invalid tag (7)', but it sounds like your situation is more complicated with the explicit scan unable to find it. >>19914 No worries, thanks for letting me know. >>19932 Damn, sorry for the trouble! That's my mistake, I will fix it for the next version. >>19933 Not a quick way, but copy that hash and make a 'system:hash=blah' search page (or maybe that 'open selected in a new page' will work for deleted files?), then make sure the search page is set to 'all files ever imported/deleted' on the file domain, then right-click the file and manage->file relationships->view xxx, and you should be able to 'view' everything you've set, albeit all the deleted files will be without thumbnails. >>19934 Great thought, thanks. >>19936 Thank you, sorry for the trouble! Don't clear any of your pending stuff--I'll fix whatever's going on here, and the PTR will be updated this week (I note it is still on v521, so I think it may have missed a rule change I made in the intervening weeks), so please give it another try next Thursday or later. At the very least, the server shouldn't throw an error like this, even if it receives gubbins.

>>19895 I have "transparency as checkerboard" unchecked under Media options and it's started showing checkerboards on the preview window and media viewer regardless of the setting again since updating to this version. Though I did something like 6 updates at once.

>>19941 >>19895 Actually, no. This is only happening for this one file. The file clearly shows no background in the thumbnail, but the preview and media viewer add a checkerboard. Other files with transparency work fine.

Probably a dumb question, but how do I get Hydrus to forget unused tags? I have a lot of tags that are not on any files but still show up when entering tags. I've tried a few things under database > regenerate, but I haven't found a fix for it yet.

>>19942 After further testing it seems this only happens with files that have both transparency and duration.

>>19943 Try tags > migrate tags (or select your files and pick migrate tags in the tag editor under the cog icon), then set source as deleted and filter as clear deletion record. I think at least, back up your shit just in case.

>>19945 That seems to have worked, thanks!

>>19940 >>19911 here, I actually found the bad tag through brute force like I said, it was a ":" or a "::" and I already uploaded my tags to the PTR, thankfully it was just the one colon tag blocking me. Here's the log, from the first failure to upload to the invalid tag scan. There were some more failures to upload afterwards but I assume they're just all the same message. v533, 2023/07/09 16:50:05: The repository encountered an error it could not handle! Here is a dump of what happened, which will also be written to your client.log file. If it persists, please forward it to hydrus.admin@gmail.com: Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.10/site-packages/twisted/_threads/_threadworker.py", line 47, in work task() File "/usr/lib/python3.10/site-packages/twisted/_threads/_team.py", line 182, in doWork task() --- <exception caught here> --- File "/usr/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda> inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext return func(*args, **kw) File "/opt/hydrus/hydrus/server/networking/ServerServerResources.py", line 1273, in _threadDoPOSTJob HG.server_controller.WriteSynchronous( 'update', self._service_key, request.hydrus_account, client_to_server_update, timestamp ) File "/opt/hydrus/hydrus/core/HydrusController.py", line 934, in WriteSynchronous return self._Write( action, True, *args, **kwargs ) File "/opt/hydrus/hydrus/core/HydrusController.py", line 241, in _Write result = self.db.Write( action, synchronous, *args, **kwargs ) File "/opt/hydrus/hydrus/core/HydrusDB.py", line 945, in Write if synchronous: return job.GetResult() File "/opt/hydrus/hydrus/core/HydrusData.py", line 2069, in GetResult raise e hydrus.core.HydrusExceptions.TagSizeException: Traceback (most recent call last): File "/opt/hydrus/hydrus/core/HydrusDB.py", line 606, in _ProcessJob result = self._Write( action, *args, **kwargs ) File "/opt/hydrus/hydrus/server/ServerDB.py", line 4904, in _Write return self._write_commands_to_methods[ action ]( *args, **kwargs ) File "/opt/hydrus/hydrus/server/ServerDB.py", line 4114, in _RepositoryProcessClientToServerUpdate master_tag_id = self._GetMasterTagId( tag ) File "/opt/hydrus/hydrus/server/ServerDB.py", line 1070, in _GetMasterTagId HydrusTags.CheckTagNotEmpty( tag ) File "/opt/hydrus/hydrus/core/HydrusTags.py", line 186, in CheckTagNotEmpty

[Expand Post]

raise HydrusExceptions.TagSizeException( 'Received a zero-length tag!' ) hydrus.core.HydrusExceptions.TagSizeException: Received a zero-length tag! v533, 2023/07/09 16:50:05: uploading pending to public tag repository public tag repository error v533, 2023/07/09 16:50:05: Traceback (most recent call last): File "/opt/hydrus/hydrus/core/HydrusThreading.py", line 427, in run callable( *args, **kwargs ) File "/opt/hydrus/hydrus/client/gui/ClientGUI.py", line 331, in THREADUploadPending service.Request( HC.POST, 'update', { 'client_to_server_update' : client_to_server_update } ) File "/opt/hydrus/hydrus/client/ClientServices.py", line 1345, in Request network_job.WaitUntilDone() File "/opt/hydrus/hydrus/client/networking/ClientNetworkingJobs.py", line 1977, in WaitUntilDone raise self._error_exception hydrus.core.HydrusExceptions.ServerException: 500: The server's error text was too long to display. The first part follows, while a larger chunk has been written to the log. The repository encountered an error it could not handle! Here is a dump of what happened, which will also be written to your client.log file. If it persists, please forward it to hydrus.admin@gmail.com: Traceback (most recent call last): File "/usr/lib/ v533, 2023/07/09 16:50:05: Exception: v533, 2023/07/09 16:50:05: ServerException 500: The server's error text was too long to display. The first part follows, while a larger chunk has been written to the log. Traceback (most recent call last): File "/opt/hydrus/hydrus/core/HydrusThreading.py", line 427, in run callable( *args, **kwargs ) File "/opt/hydrus/hydrus/client/gui/ClientGUI.py", line 331, in THREADUploadPending service.Request( HC.POST, 'update', { 'client_to_server_update' : client_to_server_update } ) File "/opt/hydrus/hydrus/client/ClientServices.py", line 1345, in Request network_job.WaitUntilDone() File "/opt/hydrus/hydrus/client/networking/ClientNetworkingJobs.py", line 1977, in WaitUntilDone raise self._error_exception hydrus.core.HydrusExceptions.ServerException: 500: The server's error text was too long to display. The first part follows, while a larger chunk has been written to the log. The repository encountered an error it could not handle! Here is a dump of what happened, which will also be written to your client.log file. If it persists, please forward it to hydrus.admin@gmail.com: Traceback (most recent call last): File "/usr/lib/ The repository encountered an error it could not handle! Here is a dump of what happened, which will also be written to your client.log file. If it persists, please forward it to hydrus.admin@gmail.com: Traceback (most recent call last): File "/usr/lib/ v533, 2023/07/09 17:50:16: Invalid tag scanning: No bad tags found!

>>19947 Also, one of the problems I had was trying to actually figure out what was wrong. The zero-length tag message is kinda buried in all these backtraces. I actually thought it was a PTR server issue for about a day due to the HTTP 500 errors.

>>19919 A request related to this: it would be nice if a namespace itself could be used for parent tags. So in that example females: could imply female for easier searching, since throwing "female" into the search box is easier than system:tag as number > type in "females" > select "greater than 0". You can currently assign individual members of that namespace to be a parent/child but you have to do so for every number you use.

I had a good couple of weeks working on the server janitor petition workflow, making it able to work on more than one petition at a time. Unfortunately, this is UI that only a handful of users ever see, and I didn't have time for all that much else, so there aren't a lot of changes for regular people. One neat thing, however, is the introduction of early support for Krita and SVG files. The release should be as normal tomorrow.

https://www.youtube.com/watch?v=qv6yDjtIl8k windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v535/Hydrus.Network.535.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v535/Hydrus.Network.535.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v535/Hydrus.Network.535.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v535/Hydrus.Network.535.-.Linux.-.Executable.tar.gz I had a good couple weeks working on better petition-processing workflow for repository janitors. Most users won't see this, so you can ignore most of this release! Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html krita and SVG Thanks to two different users, hydrus gets Krita (.kra, .krz) and SVG (.svg) support today. Both formats also get thumbnails! These formats can get quite complicated, so if you have one that won't import right (for Krita, it'll look like a zip; for SVG, probably HTML), please send it in and I'll have a look. Same for if the thumbnails are broken, although there's a limit to how clever we can go there. Adding SVG also exposed that I don't have a good category/pipeline for handling vector vs raster graphics. Several things like thumbnail size calculations are different with an SVG (which has no true fixed resolution), so I will be doing several waves of background updates in future to handle this and other vectors better. Who knows, we might have beautifully zooming SVG media viewer support with one day. petitions Ok, this was almost all of my work this past two weeks. It kind of kicked my ass, but I'm happy it is done. It changes UI that only a handful of users see, so most users can skip over it completely. For server admins, this new tech needs a server update, and it is not compatible with older clients (nor can newer clients talk to older servers), so please plan to update your server and janitors' clients at roughly the same time. It isn't going to break anything, but the client won't be able to do janitor work and will produce some harmless errors either way. For janitors, I've made the petition processing page handle multiple petitions at once. There's a new panel, a multi-column list, and when you click the 'fetch petitions' button, it populates with as many petitions as you like. You can see and sort by the action (ADD, DELETE, REPLACE), account id, reason, and total size (e.g. '60 mappings in 12 petitions'), and then double-click any to highlight it in the normal petition UI you are used to. This is about speed of processing. When you process a petition, it starts uploading in the background and the page moves immediately on to the next one. No more waiting for the server to summon another petition. Also, it gives you a better overview of your queue and even lets you mass-approve/deny a set of petitions when one account or reason keeps making good or bad stuff. You can fire off twenty petitions in one go and quickly move on to the next. There's also some quality of life improvements. There's a little layout improvement. The current petition type you are working on is now highlighted with a '*' on the respective button. The page also remembers what type of petitions you were looking at, and what you set the 'show this many files' widget to, and the new 'how many petitions to fetch' value. Now, what works in my small testing environment may struggle at scale, especially the PTR. I dump a good amount of new CPU into selecting this new petition list, so maybe fetching 40 petitions takes too long and you have to turn it down to 10, or maybe this lags out when you get a 100,000 mapping petition, or maybe the queue order is borked somehow. This is just a first version of this new system, so let me know how it works and I'll tune it. And there is more I already want to do. The new list could do with a right-click menu for copying account id and retrying failures, and some of the widgets (sort-by, num files) could do with some show/hide tech on petition types they don't apply to, and it would be nice to have some shortcuts working. I also think the management panel is too tall now, so maybe I'll fold it somehow into tabs, or allow a double-wide panel, or just have more aggressive show/hide or resizable layout? In any case, I'll keep banging away at this. next week I've got a ton of normal small work piled up, so I'm back to that for a while. Thanks for your patience.

Any particular reason why hydrus hangs when I apply certain PTR tags? I've noticed it pretty consistently with species:human and christmas for example.

So I made a downloader to download all files linked on a page (https://files.worlio.com/files/~media/retro-renders/) but I'm unfamiliar with Hydrus parsers so it gets a bunch of junk that gets ignored. Does anyone have any tips to make this more broadly applicable/cleaner? I don't know if it would work on any other site.

>>19953 This seems more like a job for something like Link Gopher from https://sites.google.com/site/linkgopher/ This extension will grab all urls on a page. Just highlight the ones you want, copy, and paste directly into hydrus. Attached is a sample of what Link Gopher does to the page you mentioned.

Is there a way to have a tag activate/set a rating or a rating set a tag? As an example I was thinking of using a "content rating" field which is only goes up to 3, essentially safe, questionable, and explicit. It would be pretty useful to have it so a tag could set a rating or vice versa. It would also be a cheap method of having mutually exclusive tags/ratcheting(?) tags, e.g. a tag that precludes another tag, explicit overrides questionable which overrides safe.

>>19955 Personally I just have safe, lewd, explicit tags, and I make explicit a parent tag for nipples, penis, vagina, anus, balls, realistic dildo, and realistic fleshlight. Every once in a while I run a search for safe+lewd, safe+explicit, lewd+explicit to make sure there's nothing with more than one of those tags, because as you mentioned, tags cannot be mutually exclusive on their own.

Any ideas as to why my download speed from Gelbooru is less than 100kbs? I haven't touched downloader options in years. I used to get like 3 images per second, but now it's closer to 3 minutes per image.. It's not a big deal, just something I noticed.

>>19941 >>19942 >>19944 I'm afraid I think this is due to mpv. mpv always does this checkerboard, and my checkerboard is really just to line up with that. If you really want, you can show gifs in my native viewer under options->media->media viewer filetype handling, but mpv really is the superior viewer, especially when complicated transparency starts happening. >>19943 >>19945 >>19946 Thanks, this is interesting. I just worked with a user on this today and reproduced the bug, so I hope to fix it properly for next week. database->regenerate->tag text search cache also seems to fix it. >>19947 >>19948 Thanks. This should be fixed now, I think! The PTR is updated and should A: accept this tag, and B: stop moaning so loudly about borked tags. Let me know if you still get anything. >>19949 Great idea. A bit like a sibling namespace ('convert all "artist:" tags to "creator:"'). Unfortunately namespace tech is CPU expensive and complex, and so integrating it into siblings and parents has been long long long delayed, but I like the idea. >>19952 If the tag has parents, it can take a little longer to calculate all that. The e621 tags like 'species:' tend to be pretty complicated. If you are adding hundreds at once, I could see it adding, say, 500ms at most. Are you getting worse than that? If so, please check out help->debug->profiling mode. Do some runs of these tag adds with that on and pastebin or email or whatever me the profile, and I'll check it out.

>>19953 Yeah, in your formula, there's a bit under where you say what tags to look through, 'content to fetch'. Instead of fetching the html and then chopping it up with regex, try setting it to 'attribute' and then 'href' as the attribute to fetch. It'll pull the links right out of the 'a' tags for you, and if you really need to filter out '../', then maybe a String Match could do that for you. >>19955 Not yet, but I think it makes sense. I use some like/dislike ratings to act as 'these are good post images' or 'favourite' or 'move this to the slideshow folder', and while I search for them using ratings, it would be nice to have the option of getting the tag too. Having a three-part that was synced to rating tags is a great idea. I'll keep this in mind, but I can't promise when it will move forward. >>19957 I don't think I did it. In general, while there are plenty of limits on when to start downloads, once they are going hydrus tries to pull them down fast as possible, and I haven't ever added a xxxKB/s limit or anything to any of the defaults, I don't think, so if you haven't added any either, I'm guessing it is gelbooru being busy or your router straining or your ISP getting pissy somehow. If you used to get 3 images per second, did you remove the '1 request per domain per second' default global rule from hydrus? It _could_ be that there's some Cloudflare-like gateway based slowdown rule being applied to you because of long-term spamming, but that stuff is often hard to diagnose. I'm getting multiple MB/s here, fast enough that I don't see the progress bar, on my test machine. So it isn't a site-wide permanent rule for everyone.

>>19955 I usually manually do those because of the problem of rating exclusivity. I'll check for unrated files every once in a while and do those in case I missed any, and also check for files with multiple ratings and fix them. There's also how rating is a bit subjective. Sure, if it's got full blown sex then it's obviously explicit, but there are tons of time that the lines between ratings are blurred. What if it's an innocent image but there's a tiny bit of cameltoe, safe or questionable? What if it's of a girl wearing a thin bikini in a non-provocative way but if you squint you can see some slight pink near the asshole? Does a girl wearing only a bra and panties count as questionable, and if so, do bikinis also automatically count? Does nudity imply explicit, even if it's from an angle where no lewd bits are visible? In the end I just go with whatever I feel fits best personally.

>>19959 >try setting it to 'attribute' and then 'href'. Thanks! Worked perfectly. Here's the final version if anyone's interested.

I think this is a bug? I'm not really sure, capital letters show up before lowercase in the "gallery search" selection box and nowhere else. It's easy enough to change them in the GUG panel but since it's inconsistent I think it's more of an oversight.

>>19962 Where did you get aibooru downloader?

Can someone give me some advice on something? Is there a way to redo a gallery import so that I can get tags from a parser without redownloading everything? I made a fapello parser and scraped a ton of images, but just now added a tag parser to grab usernames. I want to "refresh" each of my imports and automatically grab the usernames with the parser now.

I had a mixed week, a bit short on productivity. Despite a light changelog, there's a neat fix to the 'this tag is still in my autocomplete even though I deleted it from everything' problem, and, thanks to user help, support for importing XCF and gzip files. The release should be as normal tomorrow. >>19964 Run the download again, but set a custom 'tag import options' (which is inside the 'import options' button) before you start it. Hit the 'force page fetch even if url/hash recognised and...' checkboxes, and the downloader will parse everything again. It won't redownload the files, so while it'll take longer than a normal one, which will skip any single-file pages it has seen before, it won't eat too much bandwidth.

>>19965 Ah yeah, you're right that worked. Weird, I tried that exact thing before this, but I guess that I just did it on a page that it didn't work correctly on, because it's working for the other ones? I'll look into the parser more I guess, thanks for the help. Excited for the new release.

>>19963 I got it from this repo https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts

https://www.youtube.com/watch?v=EtVjGWpbE7k windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v536/Hydrus.Network.536.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v536/Hydrus.Network.536.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v536/Hydrus.Network.536.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v536/Hydrus.Network.536.-.Linux.-.Executable.tar.gz I had a mixed week, but I fixed some things, and there is more filetype support. There are a couple of database fixes this week, so anyone with a large database, including anyone who syncs or has synced with the PTR, can expect the update to take several minutes. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html highlights Thanks to a user, the program now supports gzip (a type of archive) and xcf (GIMP savefiles)! SVG thumbs also look correct now if you have 'fill' thumbnails. Since the 'filetype selector' widget is getting pretty stacked, I'm going to rewrite it very soon, and I think we are going to have one or more groups, maybe spitting raster/vector and grouping 'image' application types like psd, krita, and xcf, together. I fixed an issue where even if you deleted a tag from all the files somewhere (i.e. reduced its autocomplete count to 0), it wasn't disappearing from autocomplete results. It shouldn't happen again, and one of the updates this week will be scanning through your autocomplete data to remove all existing orphans. Otherwise it is mostly background work this week! One thing is more responsive, another is more efficient, and some UI protects against mistakes, but none of it is too interesting. There's a clever delete-lock update to the Client API if you handle a lot of advanced file deletions. future build There's another test build out today, here: https://github.com/hydrusnetwork/hydrus/releases/tag/v536-future-2 It is the same 536 but uses newer libraries. If you have trouble with the current version of Qt (e.g. borked high UI scale text sizes) or just want to test out newer tech, feel free to have a go. We've had some real problems with the recent crop of Qt updates, but this one looks tentatively good, so I'm very hopeful this jump will put the issues to bed. Let me know if there are any problems! next week I'm behind on my github issues, so I'll give them a good look, and otherwise continue pushing at my immediate backlog of smaller jobs.

If I have two images in the duplicate filter and the better one has a rating of 4/5 stars and the worse one has 5/5 stars, if I select the better one to keep, does it stay at 4 stars or go to 5? What does copy both ways do?

>>19968 I'd like to humbly shill this >>19877 for smaller project work. I want to import a bunch of stuff from e621 with translations but am waiting for that so I don't end up with a billion notes I have to delete.

Hydrus is great for searching, but is it good for browsing? Not really! How about a new special page, a tag cloud. Doesn't even need to be that advanced, just a simple list of the most used tags, and a way to filter it by namespaces, like only show creators, series, or characters etc. Not that different from the current search suggestions popup, but a little more browsable and convenient. Then you just click a tag and a new tab searching for that tag pops up.

>>19967 I already checked that and it's not there.

>>19971 >browsing This.

>>19895 >>19968 Just updated after about a month and an error occurred with the EXIF parse. You coded your interest in seeing this v536, 2023/07/27 22:09:16: hydrus client started v536, 2023/07/27 22:09:18: booting controller… v536, 2023/07/27 22:09:18: booting db… v536, 2023/07/27 22:09:19: checking database v536, 2023/07/27 22:09:21: updating db to v534 v536, 2023/07/27 22:10:05: v536, 2023/07/27 22:10:05: Exception: v536, 2023/07/27 22:10:05: AttributeError: 'Controller' object has no attribute 'files_maintenance_manager' Traceback (most recent call last): File "/opt/hydrus/hydrus/client/db/ClientDB.py", line 9508, in _UpdateDB self.modules_files_maintenance_queue.AddJobs( hash_ids, ClientFiles.REGENERATE_FILE_DATA_JOB_FILE_HAS_EXIF ) File "/opt/hydrus/hydrus/client/db/ClientDBFilesMaintenanceQueue.py", line 45, in AddJobs HG.client_controller.files_maintenance_manager.Wake() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'Controller' object has no attribute 'files_maintenance_manager' File "/usr/lib/python3.11/threading.py", line 995, in _bootstrap self._bootstrap_inner() File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner self.run() File "/opt/hydrus/hydrus/core/HydrusThreading.py", line 427, in run callable( *args, **kwargs ) File "/opt/hydrus/hydrus/client/ClientController.py", line 2184, in THREADBootEverything self.InitModel() File "/opt/hydrus/hydrus/client/ClientController.py", line 1016, in InitModel HydrusController.HydrusController.InitModel( self ) File "/opt/hydrus/hydrus/core/HydrusController.py", line 564, in InitModel self.db = self._InitDB() File "/opt/hydrus/hydrus/client/ClientController.py", line 228, in _InitDB return ClientDB.DB( self, self.db_dir, 'client' ) File "/opt/hydrus/hydrus/client/db/ClientDB.py", line 245, in __init__ HydrusDB.HydrusDB.__init__( self, controller, db_dir, db_name ) File "/opt/hydrus/hydrus/core/HydrusDB.py", line 267, in __init__ self._UpdateDB( version ) File "/opt/hydrus/hydrus/client/db/ClientDB.py", line 9513, in _UpdateDB HydrusData.PrintException( e ) File "/opt/hydrus/hydrus/core/HydrusData.py", line 872, in PrintException PrintExceptionTuple( etype, value, tb, do_wait = do_wait ) File "/opt/hydrus/hydrus/core/HydrusData.py", line 901, in PrintExceptionTuple stack_list = traceback.format_stack() v536, 2023/07/27 22:10:06: updated db to v534 v536, 2023/07/27 22:10:08: updating db to v535 v536, 2023/07/27 22:10:08: updated db to v535 v536, 2023/07/27 22:10:10: updating db to v536

[Expand Post]

v536, 2023/07/27 22:17:55: updated db to v536 v536, 2023/07/27 22:19:33: preparing db caches v536, 2023/07/27 22:19:34: initialising managers v536, 2023/07/27 22:19:37: booting gui… v536, 2023/07/27 22:19:39: The client has updated to version 536! v536, 2023/07/27 22:19:39: Some exif-scanning failed to schedule! This is not super important, but hydev would be interested in seeing the error that was printed to the log.

>>19963 There's one from a couple threads ago. https://8chan.moe/hydrus/res/18264.html#18454

>>19975 Thanks, but >I know some posts there have the prompt info in the description, but as far as I know there isn't a way to make Hydrus add the post description as a note. and >I didn't bother with login stuff because they don't make any content exclusive to logged-in users as far as I know. seems to be outdated now sadly.

>>19976 I'm the original anon you responded to, sorry, I guess I misremebered where I got the parser from. It's probably from the post the other anon linked to you. >login With hydrus companion I can send the aibooru cookies to hydrus and access login restricted posts so that's not an issue. >prompt This does need improvement, maybe I (a parser novice) can take a look at it in the future, no promises though.

My log files are getting bloated with millions of lines in the vein of 'Wanted to physically delete the "aaaaaaaaaaaaaaaa175..." file, with expected mime "101", but it was not found!' The monthly log files have been about 2 GBs in size for a couple months.

>>19959 >If you used to get 3 images per second, did you remove the '1 request per domain per second' default global rule from hydrus? I just misremembered; it was one per second. I think this has been going on a for a long time, but I only now consciously noticed it. Occasionally I do get speeds over 1MB/s, but it's sporadic at best, at least every time I look at it.

>>19968 >I fixed an issue where even if you deleted a tag from all the files somewhere (i.e. reduced its autocomplete count to 0), it wasn't disappearing from autocomplete results. It shouldn't happen again, and one of the updates this week will be scanning through your autocomplete data to remove all existing orphans. I just had one tag pop up in the tag editor that's only present on a file that has been completely deleted for a while (has the hydrus icon as thumb). Maybe these tags should not pop up too? Or is that intended?

>>19968 >one of the updates this week will be scanning through your autocomplete data to remove all existing orphans. This might be more difficult to implement, but could you add some button somewhere to do the same kind of thing, but for tag relationships instead of the tags themselves. I recently stopped syncing with the ptr, and I'd rather only keep the relationships for tags if at least one of the tags involved shows up in my files. I don't want a bunch of irrelevant relationships filling up the parents and siblings pages and slowing it down.

>>19969 Thanks, I'll improve the description labels around this stuff. In 'copy both ways', then a rating change can propagate in either direction. When there is a conflict, like in your situation, the better rating wins. So you should end up with both files 5/5. >>19970 Thanks, this slipped through. I'll mark it up again. Can't promise anything too clever, but I can do a simple 'join' thing on the current list of strings you are working with. >>19971 >>19973 Yeah I completely agree. A tag cloud control, or just general tracking of popular tags, has long been in my 'big jobs' queue. It keeps getting put back, so I can't promise when it will happen, but I hope to keep chipping away at some surrounding tech. I do have really nice domain tracking of tags and tag counts now though, so it will be easier these days. I don't want to do a simple '20 most populous tags', though, since it'll be buried in '1girl' and similar. A user sent me a really cool algorithm the other week that ranks your tags using the Xbox competitive ranking system, and it does a great job of separating the wheat from the chaff. I think we could do something like that and really start to sift through all the data in your database and say, definitively, 'ok bro we crunched the numbers on your like/dislike ratios and it turns out you really like feet' and so on. The file history chart and Mr Bones under the help menu are actually related to this, so some of this tech and UI may end up there first. >>19974 Thank you--I just got this from another user too. No worries, it is some unlucky bitrot. Nothing is broken. The job here actually completed all correct, but right at the end a reporting step I added last week threw a stupid error. I've fixed the code so it won't do this again.

>>19978 Thank you very much. That mime is 'unknown' filetype, so it sounds like the trash system has got itself confused on a borked file record and is unable to recover. I will investigate this and make sure it handles this better. Sorry for the trouble. If you weren't aware, btw, you can delete your log files any time the client is not running. They aren't required for anything. >>19980 Damn. When you say it popped up in the tag editor, do you mean you typed some letters in and it appeared as a suggested result for what you typed? The 'correct' behaviour of that is only to deliver tags with a positive (17) count after them unless that tag is a sibling or parent, in which case it can appear without a count (it won't say (0), just 'tag'). If it turns up with a positive count, or if when you right-click it it has a sibling or parent/child relationship, I think it should be there. If it has no count or relationships, then it looks like we still have a problem. Does running database->regenerate->tag text search cache make it disappear and keep it gone? >>19981 Sure, you can do this right now if you are careful: I just noticed the migrate tags dialog has fucked UI--for some reason the mappings/parents/siblings dropdown is in the wrong cell, top right header--it should be center-left. Other buttons are wrong too. I must have rewritten the underlying grid layout it uses and it all got messed up. Don't do this this week, wait for me to fix it in 537. I'll also try to get my 'deleting the PTR isn't a multi-hour pain in the ass' tech done. - MAKE A BACKUP. This shit is powerful and can go wrong, and you want a fallback - Hit services->manage services and set up a new 'local tag service'. Call it 'local PTR' or whatever. - Hit tags->migrate tags. - Take the CURRENT MAPPINGS for ALL MY FILES from PTR and ADD to LOCAL PTR, click go. - It'll take a while. - Make sure it seemed to work well, otherwise revert to backup and try again. - Delete the PTR from services->manage services.

Is there any way to quickly import files from an imported zip file or is opening the zip file externally, extracting it and then importing the files into hydrus again the quickest way?

>>19983 >Damn. When you say it popped up in the tag editor, do you mean you typed some letters in and it appeared as a suggested result for what you typed? The 'correct' behaviour of that is only to deliver tags with a positive (17) Yeah, it appears in the tag editor as an autocomplete suggestion. It appears as "tag (1)" and the only file that has that tag has long been deleted. No siblings or parents.

>>19984 You have to extract zip stuff yourself for now. In future I'll have archive inspection tech, but not yet. >>19985 Ok, good, this looks like a different miscounting problem then! Seems like your client didn't subtract the 1 to be 0, so the (now fixed) tag-removal logic didn't trigger. You are now looking at database->regenerate->tag storage mappings cache (all...), maybe with the 'tag text search cache' regeneration to follow up afterwards once the counts are corrected. HOWEVER, note that the tag storage mappings cache regen is an expensive job. You will get the choice over which tag services to run it on. Do not run this operation on the PTR, it will take too long. If you sync with the PTR, do not select 'all services'. If this problem tag is on the PTR, we'll have to figure out a different solution. If it is on your 'my tags', which will almost certainly have fewer than 100 million tags, you can fire it off and worst case it'll take a few minutes. Should fix you, but let me know.

>>19986 >Ok, good, this looks like a different miscounting problem then! Seems like your client didn't subtract the 1 to be 0, so the (now fixed) tag-removal logic didn't trigger. Are you sure we are not just misunderstanding each other? This isn't related to the issue of a tag not being assigned to any file, but a tag being assigned only to files that are in the database as deletion records. When I select the "all files ever imported/deleted" file domain, I can search for this tag and it will return a deleted file with a hydrus icon thumbnail, and this file indeed has the tag. So what I was asking is if it's an intended behavior (because it makes sense that it does this, since the tag is really assigned to something and the "(1)" is technically correct) or if it would be better to hide these tags from autocomplete too. Anyway, I tried running both of the cache regens (with restarting hydrus for each) and it didn't change anything.

>>19986 >>19987 To add, this isn't really a big deal, since I can just manually delete the tag from that file to prevent it from appearing, but I thought it's an interesting issue that's somewhat related to the fix you just did.

>>19987 >>19988 Sorry, you are correct, I was either misreading this or thinking about three other things when I was trying to understand it. The fix I made last week was that when a file goes from 1 count to 0 count, it now disappears as intended. Your file will appear when the autocomplete is set to 'all known files with tags'/'your tag service', since there does exist a file with that tag (or "all files ever imported/deleted", as you saw, since the tag does indeed exist there). The tag count of (1) there is valid. If you were to change the file domain to 'all my files', it should disappear since the actual count there should be correctly at 0. If this is super annoying, you can either remove the tag from the file, reducing its count to 0 for all known files, or you can change the default manage tags file location for each service under tags->manage tag display and search. 'all my files' is a good one, if you want to talk about 'files you have' (it excludes the trash).

>>19989 >or you can change the default manage tags file location for each service under tags->manage tag display and search. 'all my files' is a good one, if you want to talk about 'files you have' (it excludes the trash). That works, thanks.

>>19982 Different anon here, I just git pull'd a few minutes ago and got the same error as >>19974 (only difference is python3.10 instead of 3.11 so line numbers in threading.py are different)

>import file >tag it >physically delete file >re-import same file some time later >file still has all of its old tags from before deletion Is this intended? I've done a huge refactoring of what tags I keep and every once in a while I'll end up re-importing a file that I changed my mind on deleting and have to sift through the old tags. Is there a way to wipe all tags for files that have been deleted?

>>19978 I looked at this today and could not figure it out. As far as I can tell, this situation is still supposed to clear the bad record, so I don't immediately see how it could be looping so crazily. Can you check something for me? Hit up services->review services and then local->files->all local files tab. It should say 'No files are awaiting physical deletion from file storage' and/or an actual number. Do you have any files queued here, or does it seem to be caught up? Secondly, in your log, is the hash the same every time? Is it the same bad file being cycled over and over, or is it changing? If it is changing, is it random hex over and over, or is there a clear pattern to it? Third, have you recently done any large file service related work recently, anything like deleting a large IPFS service or cancelled/completed a huge file upload to a file repository? Some of that stuff queues up a bunch of physical delete file checks, although I still can't explain how a file here would have unknown mime (this usually happens for files your client has never seen, so why would it be queued up for physical delete?).

I had an ok week. There's a new selector for filetypes (in system:filetype and File Import Options), a new database-deletion routine that makes deleting large services like the PTR instant, and a variety of small bug fixes. The release should be as normal tomorrow.

Is there a way to exclude everything inside a namespace except one tag? For example I want to search for an image with a specific character, but exclude images that contain other characters.

>>19995 I feel like this should be possible with the advanced search (button that reads OR*) but I can't into the operators enough to give an exact answer.

https://www.youtube.com/watch?v=pg-yeXEvVwA windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v537/Hydrus.Network.537.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v537/Hydrus.Network.537.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v537/Hydrus.Network.537.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v537/Hydrus.Network.537.-.Linux.-.Executable.tar.gz I had an ok week. There's a mix of bug fixes and a couple of neat improvements. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html highlights First off, the filetype selector, which you'll see in 'system:filetype' and any File Import Options, is completely rewritten to a checkbox tree. It is easier to work with than the old expanding panels system and scrolls smoothly when it gets internally tall. I've been worried about adding more filetype support for a while because of how busy this widget was getting, but this will relieve things. Also, the 'applications' group is now split up. All the archives like zip and rar are now in 'archives', and all the xcf, psd, and krita formats are in 'image project files'. It doesn't matter much yet, but these groups also exist as separate rows in the viewing settings under _options->media_. Secondly, I have figured out a 'fast delete' method for very large database objects. Deleting the PTR is now an instant operation! Unfortunately, under the hood this is really just a deferred delete, so the object still uses its hard drive space until it is deleted later in the background in pieces. Unfortunately x2 is that I haven't written the maintenance job that does the delete yet, lol, so if you need to delete some service or do a large database regenerate job, please bear with the bloat for a week. If you updated last week, jumping up several versions at once, and got a popup that some EXIF scanning update did not work, don't worry. I fixed the problem, and nothing important was actually broken anyway. Thanks for the reports! I fixed the grid layout on tags->migrate tags, which exploded under my nose. Sorry if you were recently confused by this! next week I'll finish this background deletion maintenance system and keep on trucking on smaller fixes. I've got a lot of immediate work like this to do and github issues to catch up on. I'd also love to get some more system predicate parsing, for ratings and more, working.

>>19995 >>19996 "solo" should be a tag you use for any picture containing only one character, would serve your purposes if you search both "solo" and "character:name".

>>19998 Well yeah, I'm missing some solo tags, that's why I'm trying to filter characters, so I could add it. I think I could just select all the character tags in the display column and do a negative OR on them except the one I want to filter with shift + double click.

>>19983 >Sure, you can do this right now if you are careful It looks like what you told me to try would take the mappings that are in my files, and move them to a new local service. I'm not talking about the tag mappings, I want the tag relationships, as in parents and siblings. but instead of all relationships in the ptr, I only want to keep the relationships of tags that are used in my files, so I don't get a whole much of obscure relationships that I'm never gonna need because I'll never use those tags. I was asking if there was some kind of delete button or other way to just keep those and get rid of everything else.

>>19977 >>19976 I didn't forget, I've just been busy. Anon delivers, no promises for it not breaking danbooru. The aibooru parser uses the same formula as danbooru so I just added an ai metadata note. I also exported just the note, so if you don't want to change the entire danbooru parser you can just import the ai note parser in network>downloader components>parsers>danbooru file page parser>content parsers

>>19995 Use system:number of tags.

>>19995 character:soryu asuka langley system:number of character tags = 1

>>20002 >>20003 This is a good one, thanks.

Problem with dark mode Windows 10 with dark mode enabled

>>20005 That's just for content. If you want a full darkmode, you also need to turn it on under help at the top.

>>20005 >>20006 Actually no, that's the same setting. The thing you need to change is in options under style, where you pick dark_blue.qss in the qt stylesheet dropdown.

>>20007 Thank you, dear. It's way better now.

Found out that for some reason the "my tags" tag service is filled with tons of deleted tags from every other tag service, even if I'm pretty sure those tags were never part of "my tags". I tried clearing those using pic related, but it doesn't change anything, they are still there.

>>20009 A little bit more info about this: It appears that the deleted tags in "my tags" are actually all the tags that are being used (not deleted) in other services. Can't confirm if it's 100% accurate though. The tags are also scattered randomly across all files it seems. I did a test on one file, where I would re-add the deleted tags, apply, then delete them again, apply, then it would let me clear the deletion records after.

>>20009 >>20010 Update: I force added the deleted tags to everything (8500 files), which took like 5 hours of the program not responding until it got interrupted by a power outage (but I think that happened after everything got added and the program was just trying to load all the files with all the tags). Then I deleted all the tags and cleared the deletion records using tag migration, and then I loaded an exported backup of the original tags I wanted to keep. Well, it did remove the deletion records from "my tags", but now they moved to another tag service. So I assume this is just a display bug and the deletion records aren't actually there? Anyway, the whole undeletion/deletion made my client.caches.db 5gb from 60mb, which then shrunk back by doing db vacuuming, which I found in maintenance.

>>19991 Thanks to you as well. All fixed now, and thankfully it was nothing important to worry about. >>19992 Yeah. In hydrus, I partition all the different types of metadata into the different services under the services->review/manage services dialogs. Broadly speaking, the services don't change each other. Basically, if you move a file into a new file location, that won't affect its tags or ratings or whatever. This is essentially how the PTR works, too--you get stocked up on a couple billion tags for files you don't have yet, and then when you do import the file, it suddenly appears with a bunch of tags. If you want to do an operation like 'delete all the tags for files I don't have'', then there's a couple clever things you can try: Before you do anything big, make sure you make a new backup, just in case it goes wrong and you need to roll back. Hit help->advanced mode and then load up a new search page. Set the file domain from 'my files' to 'all deleted files'. Set the tag domain to the tag service you want to nuke stuff from here. Then try searching system:everything or with normal tags, and all the search should still work--again, hydrus doesn't care if you actually have a file or not, only whether it knows about x, y, or z metadata about it. If we are only talking a few thousand files, then I'd say just do system:everything and then ctrl+a->f3 and delete all the tags through the manage tags dialog. If we are talking tens or hundreds of thousands of files, I think we want to try tags->migrate tags: Note, update the client to 537 before trying migrate tags, I just fixed some borked UI on it! For migrate tags, I think you'd want to set up: (assuming you set up a new tag service, B, to accept the tags you actually want) Get all CURRENT MAPPINGS for ALL MY FILES on (tag service A) and ADD to (tag service B). -or- (wait for me to add 'all deleted files' next week, since I just noticed it is missing here) Get all CURRENT MAPPINGS for ALL DELETED FILES on (tag service A) and DELETE from (tag service A).

>>20012 >Set the file domain from 'my files' to 'all deleted files'. Wouldn't he accidentally nuke tags for existing files, since that domain also returns files that were part of multiple file domains at some point and were only deleted from one?

>>20013 Shit you are 100% correct. >>19992 Ok instead of 'all deleted files', use the 'multiple/deleted locations' choice on the advanced file domain selector and in the next multiple choice dialog select 'deleted from all local files'. That'll only be files you don't have.

>>20000 Damn, sorry, I am on a roll of not reading people's messages properly recently. I don't mean to. Yeah, migrate tags can also do siblings and parents. Same basic deal, just select the different type in the dropdown and choose source and destination. I recommend you create a new service to hold the siblings and parents so they aren't mixed with anything you created just by yourself. Do the migrate, and then hit tags->manage where tag siblings and parents apply to make sure the siblings and parents are working on whichever other local tag services you want them to. >>20009 >>20010 >>20011 Thank you for this report. This is obviously very strange, and I am not sure what is going on here. I will investigate this issue. Ideally, as you say, this is actually a display or front-end calculation issue, and the actual deleted mappings storage is all good. Deleted tags have always been annoying to inspect, so this is a good chance to think about exposing them better.

Something random I stumbled upon right now is that "sort by tags: number of tags" doesn't respect selected tag service and sorts based on the number of all tags across all services. This can be a problem, for example, when I try to quickly see which files have too many tags for ai training (I use a separate tag service for training tags).

With 'Must not be pixel dupes' selected in the drop down, the duplicate filter is still showing me pixel-perfect duplicates. I'm not sure if this is something intentional and the option is poorly worded, or some sort of limitation, or what. I'm on v537 from the AUR, this also occurred in v536.

>>20017 Sort by hash is also not working as expected. When combined with system:limit is 64, it seems to get a random selection of images, e.g. starting with 09543c...62d96 and ending with f8a494...0ab3, when I think it should start with 000a8c...57443 and end with 00ff76...76e67.

Was there ever talk of adding a hash of pixel data alongside the traditional hashes (in for example share > copy > hash)? I think it came up in a thread once but am not sure. I know that Imagemagick can do something like this.

I've seen someone ask this before but it was several threads ago and i cant find the answer but my resolution settings got fucked yesterday because SWTOR decided to launch in some stupidly high resolution. Now some parts of Hydrus are scaled weirdly though, is there any way to reset it back to the defaults?

Is there a reason why the services object is the way it is? Having keys with random names causes problems with at least some json parsing libraries. It'd be better if it was an array of objects with a service_key field, like whats returned with the rest of /get_services, instead.

Need some help. A while back I had setup Hydrus, and I had set it up I forget exactly how but I know I had like 3 different paths. One for the database and stuff, one for thumbnails, and one for files. Well, my computer shit the bed. But I had set it up so the image files were on my iSCSI drive associated to my NAS. So those files are good. - Thumbnails, that is lost. But I assume it will rebuild them? What concerns me is I can't seem to find the database folder. I think it was on my ssd that died. However, I found a folder on my NAS that I labeled "Hydrus Backup" however it looks like the last date modified for the files were 11/10/2021. So I don't think it's updated. But I found a folder in lz4 called block that was last modified on 12/14/2021. - Is there a file somewhere in this folder here that I can see when it was last modified to see the database status? It looks like it's a full db backuo. It has all the .dll files, exe files, folders ,etc. I'm wondering if I can just use this. What really would I even lose? I was syncing with the PTR, so at most I lose my LOCAL tags correct? But everything else should rebuild from the PTR correct? Also this would restore all my settings correct? - Can someone remind me how to go in and add these paths to Hydrus> I know there is this page, but it's a little confusing for rebuilding: https://hydrusnetwork.github.io/hydrus/database_migration.html

I had a great week. The 'delete big database stuff in the background' system is finished, there's support for HEIC, HEIF, and AVIF, I fixed a bunch of bugs, and there's updates to some core libraries that should improve UI quality of life. The release should be as normal tomorrow.

https://www.youtube.com/watch?v=8yXk3_o1Hoo windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v538/Hydrus.Network.538.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v538/Hydrus.Network.538.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v538/Hydrus.Network.538.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v538/Hydrus.Network.538.-.Linux.-.Executable.tar.gz I had a great week. There's some new libraries and new file support. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html notes If you get a popup on update about missing indices, don't worry. Something I forgot last week is being fixed. Just let it work. If you run from source, I recommend you rebuild your venv this week. Just run the 'setup_venv' script again with the same settings you chose before. You'll get new Qt, OpenCV, and the HEIF support. If you are on an older OS and run into trouble with Qt, the setup now has a 'Qt (o)lder' option that gives you something we know is very stable and compatible. =highlights The main UI and image libraries that hydrus uses are updated today. We've been in an awkward place with Qt for a while, stuck on a version I regret updating to, so I hope this jump relieves this. It should speed some image load, reduce some flicker, and fix some UI scaling issues. There are no special install instructions, and we've been testing for months on different computers and haven't seen any specific update problems, so I'm not expecting any significant issues. If you do run into anything, then please roll back to your backup and let me know. Thanks to a user, we now have full support for HEIF, HEIC, and AVIF files! They import and work like any other image. We also support the 'sequence' version of these files, which are essentially like (currently super rare) high quality animated gifs. I'm very new to all these formats, and they are obviously less common, so if you have a weird version or your colours are all messed up, I'll be interested in examples. I don't even have a 'HEIC sequence' to test with! My deferred delete system is complete. Most users don't have to ever think about it, but it is under database->database maintenance. There's an UI panel to track what still needs to be deleted and some basic settings to control when it does its work. If you delete the PTR or do a large regeneration job, the (very slow) actual delete of the existing data now happens in pieces in the background in the future. I'm really pleased with the performance, and even deleting 30-million-row tables seems to work very well, but if you delete the PTR (2 billiow row table!) and get unacceptable lag every few seconds, check that menu and disable the work during idle/normal time and let me know, and we'll figure it out. tags->migrate tags has cleverer file filtering. You can now select the tags for files 'deleted from x' or 'currently in y or z' real easy. next week I'm generally happy with my current productivity, but I'm falling behind on many things at once. I would like to focus on system predicate parsing and Client API for a week, because it keeps slipping.

>>20024 Thanks devanon.

>>20013 >>20014 Anyway, can you even clear deletion records for files that are still in some domains? I only get the option for files that have been physically deleted. I think it would be nice, since I move files between domains a lot and I like to keep things tidy.

>>20024 Just out of curiosity I went looking for example avif/heic/heif images, to try to find something to test with. I found https://github.com/alexcorvi/heic2any/tree/master/demo, image 15 in the demo folder is a sequence of a car driving past, it correctly plays in my mpv. I also found https://github.com/link-u/avif-sample-images which has like one of every tweakable setting, the rotating star also plays correctly in mpv.

I don't know really where to share this but im both slightly embarrassed by how bodged it is but also pretty proud that I managed to get it working. I've managed to hack together an automatic image tagger using a couple AI models and a useful github tool. I'm gonna approach this in a problem:solution sort of format, since I think that effectively gives the timeline of events as well as what exactly I've managed to do. PROBLEM: I have 250k images, some are personal screenshots, photos, etc., but most of it is my autistic downloading of various art, animations, and other related stuff online. Most of it I rarely look at, and I don't have a whole lot of free time to sort it all and tag it all by hand. The PTR helps, but it doesn't get everything, and often misses a lot of the more content specific tags. SOLUTION: AI tagging has slowly been improving over the past few years. My understanding is that it's mostly done to help with image generation, but it works just as well for tagging already existing images. In the past, the tool of choice (from my understanding) was hydrus_dd, but... PROBLEM: DeepDanbooru isn't the most accurate AI tagging model, having some major flaws. Chiefly among them being it's lack of an ability to tag loli/shota. I don't know if this came from a place of moral disagreement over the contents of those images, or a lack of ability to download images with those tags from danbooru in the first place, but it's problematic for several reasons. It makes it hard to find, hard to filter, and effectively helps neither those who hate or love loli/shota. SOLUTION: In the past year, a new tool has been created called https://github.com/abtalerico/wd-hydrus-tagger. It uses a much more accurate model, and notably has the ability to tag loli/shota images. PROBLEM: the tool dumps all generated tags to the "My Tags" service by default, diluting the service with possibly inaccurate tags. It's better than dumping the tags into the PTR, but not ideal. SOLUTION: it's not documented, but looking at the code inside main.py reveals that the tool actually does support identifying a specific tag service to dump tags into. running the tool with the --tag-service argument lets you identify a service to put tags into. Note that for some reason this is only possible using a hashes.txt to read hashes from, but truthfully I dont think anyone is gonna be running this for individual images. you can always just copy the hash of a single image into the file anyways. PROBLEM: the tool is automatically tagging now, but it's hard to keep track of which files have already made it through the tool and which haven't. SOLUTION: I managed to figure out how to append another tag onto the dictionary that stores the generated tags before they're dumped to hydrus. I'm not educated on either python or AI tagging models, but I was able to figure out that the tags are stored using key:value pairs, with the key identifying the tag and the value identifying the confidence the model has in the tag being accurate. I simply added another pair, with the key = ' wd-hydrus-tagger_ai_generated_tags' and the value being abnormally high to ensure it always get's added. PROBLEM: Now I know which images have been tagged and what hasn't. Things are looking good, except that I'm noticing a specific problem with furry art. Anything with sharp horns is being tagged as "demon-girl", amongst other inaccuracies. the model simply isn't made to identify furry art that well. SOLUTION: Thankfully someone has made a specific furry AI tagging model with help from the developer who originally made the model that wd-hydrus-tagger uses. This means that it's entirely compatible with the tool. I don't know if this was just really good luck or if this is some standard way of designing these AI models, but I'm acting a little more careful around dangerous machinery nowadays, just in case all my good karma has run out from this. However... PROBLEM: The tool has no clear way to run models locally. There's nothing in the documentation and looking at the code reveals that the model is downloaded and stored in cache before running. I'm not entirely sure, but it looks like the code is able to download models from huggingface.co and run them, but the specific model for furry art is hosted in a discord server, which puts my archivist heart in pain. SOLUTION: the solution here is actually pretty simple, but this stumped me for quite awhile. I have very little experience in python coding, and so far I've made it through good guesses and some overlapping if underutilized C++ knowledge. At first, I figured out that I could just replace the wd model in the cache folder with the furry model, rename it to match, and it'd work alright. But this lead to the problem of me having to navigate back to this folder and swap the models anytime I wanted to tag anything that was/wasn't furry. This quickly became tiring and confusing, and I started investigating the various python files for how the tool was specifically identifying the models. turns out that by simply pointing the tool to look in a specific location for the model instead of running the downloading function and getting the cache location back, it bypasses that entire process. This meant that I was able to copy the folder the tool was in, change it's name to e621-hydrus-tagger, and point it specifically at the furry model. Then it's just a matter of running the code from the e621-hydrus-tagger folder instead of the wd-hydrus-tagger folder. The only flaw with this so far, is that if I move the repos from the current folder I have them in, the tool wouldn't be able to identify the files. I doubt this is an impossible problem to fix though, I probably just need to read the documentation on the path() function more. PROBLEM: I now have two different models, files are getting tagged, furry art is getting more accurate tags, all is good. Although it turns out that the furry model doesn't tag content ratings, meaning theres no quick way to filter out pornography. SOLUTION: I copied the folder again, pointed it to the WD model to get a ratings tag, discarded all the other tags and only returned the ratings. Then I named the new folder "ratings-hydrus-tagger", and ran it just like the others. PROBLEM: Now I have 3 different folders, each with custom versions of the tool. I don't want to get confused and accidentally run the wrong model, and storing all the relevant commands in a notepad isn't exactly ideal. SOLUTION: I created several .bat files to handle all the execution. Plus, this makes sure I don't accidentally erase the hydrus API key or the tag service from the commands when i run them. I created a start.bat that initialized the venv environment, and i have wd.bat, e621.bat, and ratings.bat, to run their specific code. I also have e621.bat automatically execute ratings.bat, since that is almost always ran directly after the former. Running the tool is as easy as copying all the hashes into hashes.txt, opening start.bat, and typing wd.bat or e621.bat and pressing enter. That about wraps up my experience. Potential improvements include a way to run the various versions of the tool through a menu system instead of having to type out the specific bat files, and fixing the path()s to identify the models within the local folder so I can move the entire folder if needed without breaking anything. I'm considering figuring out a way to release this, but it's so hacked together I'm not sure how useful it would be on other people's systems. I'm also running windows, so I'm not sure how easy it'd be to make it work on other OSes. I don't like the idea of hoarding this to myself, since I can imagine it'd help a lot of people, but I'm just not sure how to go about releasing this. Any advice, criticisms, ideas, etc. are all welcome.

>>20012 >>20014 >>20013 Thanks, I started running it last night. Is there a way to estimate how long it will take? I'm averaging about 50 rows/s but don't know how many rows I have (or exactly what a "row" means) and there's no progress bar.

>>20028 This sounds amazing, but it's also the kind of thing that's WAY over my head without detailed step by step instructions. I'm too smooth brained to even get hydrus_dd working on it's own so I gave up on this sort of thing ages ago. I keep hoping something like this will eventually get packaged into a single installer (ala Easy Diffusion) or just be integrated directly into Hydrus at some point. Until then, I'm just gonna manually tag my 400k images by hand. It's way more productive to do it by hand than to keep spend days trying to cobble together something like this with nothing to show for it at the end. Still, this looks super cool and if you are willing to take the time to write up a super detailed guide on how to make all this work I'll definitely give it a try.

>>20028 I'm also interested in using this. I'd suggest putting it on Github with any of the modified files (without important stuff like API keys) and detailed instructions. Assuming the .bat files don't do anything really complex they can probably easily be converted to .sh files. If you want to put the .bats in a pastebin I could take a look at them, I'm no shell expert but I could probably convert a basic shell script, and even if I can't do anything with it, I'm sure some other anon here could.

>>20028 >>20030 >>20031 Personally I've been using https://github.com/toriato/stable-diffusion-webui-wd14-tagger which generates text files with tags you can use as sidecars. The downside is that you need the whole sd-webui, because it's an addon, but I'm already using that so not a big deal for me. But since you mentioned ratings, I noticed it doesn't actually include ratings in the text files (probably because they're useless for training) and there doesn't seem to be an option to enable that, which sucks.

>>20032 Actually, just checked again and there's an option to save the tags as json, which has the ratings, but it also includes all the tags no matter the threshold setting along with the actual values, so importing it to hydrus would be a little harder.

I'm having some issues with some "gifs" (read:mp4s) downloaded off Twitter. Of the three that I've found, they don't have audio but they actually blank out all other audio streams on my system when they play, even in the previewer which I keep muted and have vol set to 0. All I hear is popping, like when you plug a pair of headphones in or something, it sounds like it's going to break my headphones. With hydrus's mpv muted in pavucontrol the popping doesn't happen, and it doesn't happen when playing the file with standalone mpv, it also doesn't happen if the file is paused. I see some '[MPV error] main: too many events queued' messages in the logs, but it's not consistent with the popping or selecting one of the offending files. I've attached the mp4s in case anyone is willing to take a look, the files are from retweets from twitter accounts I follow, not stuff I'm interested in myself.

I'm confused by the weights in the duplicates options menu. I like to clear out pixel dupes first, and I want to prioritize filesize over everything else, how do I make it prioritize showing the smallest file first so I can brainlessly do this? These are the weights I was testing but it doesn't seem to work, it seems to show larger pixel dupes 50% of the time. Also, what are the default weights? I think it would be helpful to have them written down somewhere in the options or have a 'reset to default' button.

>>20036 Thank you

>>20029 It just finished. The vacuum dialog says that filesize-wise over half of my external_caches and just under a fourth of main can be reclaimed after purging the tags. Is it possible to do something similar with URLs? I have a couple hundred thousand files deleted so I don't think the select all > edit URLs would be a good idea.

Would it be possible to make it so that drag-and-drop export takes a note instead of tag ex.:(notes:filename)?

>>20016 Thanks. Yeah, the stupid reason for this atm is that the thumbgrid isn't aware of the current tag context. I need to do some backend work to make sure that when you change the current tag domain, the thumbgrid listens to that signal and then applies it when it sorts itself. Atm it is just fixed to an 'all known tags' for num tags or namespace sorting. >>20017 We had a few rounds of this some months ago, and I thought I had fixed everything. Damn. iirc, there were some complicated group situations where even though the pair that look similar are/are not pixel dupes, the actual selected kings of the merge groups have the opposite relationship. I thought I had forced the filter to work around this, but either I have some gaps in my logic, or I didn't fully do what I thought I did, or there's another issue like your cache of what each file's pixel hash is is wrong. I'll have a look at the code. How common would you say this false positive situation is? One in ten pairs? One in fifty? When it happens, do you think it is with files that are already in a group, or can it happen with files that have no other file relationships yet? >>20018 Damn, I get the same. Thanks, I'll check it out. EDIT: I fixed it, it was a stupid small mistake I made, thanks for the report! >>20019 I'm going to make this available on the Client API at some point as part of a duplicate files expansion. It may end up in the normal file metadata call, or it may be a separate thing. There may also be 'your POST a file to the Client API, it gives you perceptual hash and pixel hash back'. Note that my similar files hashes are not very useful outside of hydrus contexts. You can compare hydrus phashes or pixel hashes with each other, but other programs will use their own algorithms and they won't match up with mine. I don't use a common phash library, but my own bullshit I hacked together. This may change in future. If you really want to generate one of these hashes now, copy the image bitmap to your clipboard and then paste it into the new 'system:similar files' panel. >>20020 Is your situation better with v538? I'm really hoping this new Qt is going to fix the mismatched UI scaling we've seen. Alternately, does your OS have any trouble running v538?

>>20021 I thought about and talked with some people about doing a flat list, but in the end we had enough objects elsewhere already using the 'service_key is key' system, like in the newer tags structure in file_metadata, that I decided to keep things the same everywhere, even if that was slightly awkward for automatic parsers to handle, than invent yet another format for conveying this. The original reason for service_keys being the key is my own code. Hydrus python uses service_key as the pivot for a thousand different dicts, so that eventually crept out into the Client API and is now a legacy issue. Sorry for any difficulty here--if you can, write up some methods that'll parse my JSON into a data structure more convenient for you, and I recommend, whatever you are doing, making a temporary cache of the services to hold clientside at the beginning of your session. >>20022 Yeah, you are ok if thumbs are missing. Hydrus will moan on boot and there's a button to say 'no worries, just make a new thumb location here and regenerate everything'. To help find your database folder(s), look for 'client.master.db' or 'client.mappings.db'. Those are pretty unique names. To see last modified status, check your 'client.db' file in the main db folder. That's the 'core' db and is updated most often. It is generally updated in the last moments of program exit, too. The most recent 'client - (date).log' would be good too, since that too should have lines written at the moment of program exit. If you restore an older database to a newer file store, that's generally fine. You'll get a couple of popup errors for missing files, but we can fix them. As you say, you'll lose the intervening stuff like your recent local tags, inbox/archive, ratings, import timestamps, that sort of thing. PTR will rebuild, maybe a couple of missing file errors, but it should fix them itself. If you need to go this route, you'll want to run two database routines: database->file maintenance->manage scheduled jobs->add new work - Do a search for 'system:everything' and then add the 'if file is missing, remove record (leave a delete record)' job. On the other tab on that panel, you can hurry that along. This will delete everything that was deleted since the database backup was made. database->file maintenance->clear orphan files - This exports stuff you imported since the database backup was made. Send them to a new folder on your desktop and then re-import them when you are done. For fixing any borked paths in database migration, go in that dialog and: If the location you want is not there, or it has no 'weight': Add the location. If the location you don't want is there and has weight: Remove the location. Then, all the places you want should have 'ideal >0%' in them, and the places you don't should have 'ideal 0%'. Then you can move files, if needed. Let me know how you get on, and if anything is worrying or breaks, I'm happy to help or clarify.

>>20026 You can't yet. I broadly agree. I was never happy with how the multiple local file domain delete status worked out. Just moving from one thing to another shouldn't really be a 'delete' tbh. Maybe I should scrub the whole idea, or rename them to 'moved from x', or as you say just add some 'clear deleted records from this domain' kind of thing. I don't think they are really used for much internally. >>20028 Based, well done. As this tech continues to mature, I'm looking forward to this chain getting simpler and smaller and easier to work with over time. If there's stuff I can do on the hydrus end to make your life easier, let me know. And as you figure out more pleasant ways to handle the various issues, or if you decide to write up a full how-to guide, let me know and I'll point to it or host the document itself in the hydrus help. My dream is to eliminate the bulk of what the PTR does through nice and reliable auto-taggers. I know users who are training models on PTR data, and with this tech we'll finally be able to stop spamming 'blue eyes' at each other so much, but instead share a blob that describes what 'blue eyes' fundamentally look like. >>20029 >>20038 Btw a 'row' in this context is an atom of metadata, which in this case is one tag/file 'mapping' pair. Unfortunately for these mega jobs I can't provide great progress bars. I'm glad it worked out well in the end. There's no similar system for URLs unfortunately, although I've long wanted to add one. I want to rename and reconfigure URLs en masse when websites change their formats and things. It'll have to happen when I'm ready to schedule all the time for it, and I don't know when that will be. >>20034 Damn, I know another guy to whom that happened (on Linux, which I assume you are too). Silent movies, same deal. We figured out a special muting mode for mpv loading silent movies too, so hydrus should be explicitly setting mute on the mpv player when these load, but still it did this. He scratched it up to an audio driver that wasn't too happy combined with the absolute shitshow of how hydrus gets video working through mpv. I presume the dll/so way of loading mpv is just rude in some way and it messes with the audio driver for your particular situation. You might like to check out the 'mpv.conf' in your install_dir/db folder. There's a little text file in there that says this, but I know some Linux users have overall stability success adding these lines: ao=wasapi audio-fallback-to-null=yes Maybe there's something similar that can help simply disable your audio driver, or maybe taking out the normalisation fixes you. You can look up more commands, but unfortunately it is a hellish gigantic document to walk through: https://mpv.io/manual/master/ >>20035 I hate this system and UI mess. I really want to replace it with a dynamic system where you set your own rules. A reset to defaults button for all the options panels would also be great. For now, I believe you can set negative values to those scores. So if you set -50 for non-trivially and significantly higher filesize, then it is like smaller files get that score, and they'll end up as the A of your pair more often. >>20039 Maybe. I keep meaning to overhaul this system, since it never proved as flexible as I wanted. I have all these powerful now string-handling objects, and this phrase system doesn't use any of them. I'll say that notes, however, is a complicated metadata, and obviously most notes are going to be way too long for a filename, so I'm probably 'clipping-the-first-64-characters-and-adding-hyphens' or something similar, or adding options for where and how much it clips and how it might do replacement characters, which again would benefit from an upgrade to the whole system. I'll keep this in mind, but it is probably happening after other updates here.

>>19641 my first time popping on 8ch in years - very very cool project OP I had a similar idea, but it would focus around using AI to generate tags and search images. I thought it could be made into a phone app with ads to generate revenue user-added tags are definitely better - but there are soooo many memes, and so many tags haha

>>20043 This project has been going on since about when 8chan started, pre-4chan exoduses. I wish I had known about it a decade ago, but a part of me is happy I only found it recently, because in even that short time I've witnessed usability improvements that would have been deal breakers for how much more inefficient my tagging would be.

>>20040 >pixel dupe issue >How common would you say this false positive situation is? One in ten pairs? One in fifty? About one in two to two in three, but its likely more common for me than others, as I had imported a set of files from hydrus I ran png optimizations on. >When it happens, do you think it is with files that are already in a group, or can it happen with files that have no other file relationships yet? Mostly the former, I think, but I can't say 100% for sure. I'll run some tests on a clean db and see if I can replicate and maybe determine some more specific conditions.

>>20045 I'm not sure what I was on about here, but a recent test on my main db (I have yet to replicate this in smaller scale tests on clean dbs), shows 4/32 image pairs exhibiting the erroneous behavior, with none having existing duplicate groups, near as I can tell. Further all of the images with this problem have metadata that is removed in the optimization process, and they all have a "Software: Celsys Studio Tool" field, but I'm not sure if that is a red herring or not. I've attached two of the pairs that exhibit this behavior, just in case, though they don't seem to exhibit this behavior on a clean db.

Newest version has some hard to read tooltips on Fusion + Dark_Blue.

So I'm not sure if it's the new Qt version, but there was this bug maybe the past few weeks that's been affecting me, where when you'd open a new page, it would actually freeze Hydrus for about 2 or 3 seconds, and then the new page would open. Updating to this new version and redoing the venv seemed to finally fix the issue though. If anyone else was noticing that issue, this new version might fix it.

>>20044 Oh? Then I'm sure I've seen it and I just forgot. The name and logo seemed familiar. I was pretty active in the early days - I think it was called /tech/, right?

>>20030 >>20031 >>20042 I appreciate the kind words. I've done my best to upload everything to Github here: https://github.com/Garbevoir/wd-e621-hydrus-tagger Please let me know if there is something clearly wrong with the repository or the files. I tried to make the set-up guide as detailed as possible. I also wanted to include the models with the repository, but I hit the Github file size limit, so I wrote quick guides on where to get them in their respective folders. If anyone wants to take this and improve upon it absolutely be my guest. And if anyone wants to avoid using discord or joining the goo/latex fetish server for the furry tagging model, here's the download link for it: https://pixeldrain.com/u/iNMyyi2w

>>20041 Ok I did find an up to date Hydrus db. I guess I stored it on an external hdd instead of the ssd that died. Awesome. So, what exactly is the proper way to actually load up that db? Should I just overwrite the entire new Hydrus directory my copying it over? Or is there an import db folder option? Also I did find a thumbnails backup too. Now, both of these may be slightly out of date. By a month or so maybe. Also should I still run those database routines either way? Any chance you can give a screenshot of what the whole database import screen should look like with the weights and everything set, and the 3 different paths just so I can get a solid idea? And I won't break shit if I do it wrong correct? Like worst case, I just remove the paths, and reimport them and do it again correct?

>>20050 I got curious and tried to compare your files against the original and wouldn't it be better to add a check for model selection instead of having 3 different code folders with files that have minimal differences in them? The original already had a model selection argument, so you could use that to select the correct model folder and enable/disable the ratings. You wouldn't even need 3 different bat files. I'm a codelet myself, but if anyone else with some python skills could look at it, I'm sure it would be a really easy change.

>>19993 >services->review services and then local->files->all local files It says 39 files and thumbnails are awaiting physical deletion >is the hash the same every time Most of the time it logs one hash for several days before changing. Every hash is prefixed with a suspicious amount of "a"s "aaaaaaaaaaaaaaaa7a2b5734efc42c1f94b82b3d5d495767" "aaaaaaaaaaaaaaaafa0344770f9e9f2cb74fc8bf70c547bf" "aaaaaaaaaaaaaaaae1871648f6396a5de0f0fa00de52289c". >have you recently done any large file service related work recently No. The only thing that might be related was that I had to recover my database from a corrupted disk a little over a year ago, and I occasionally get "DataMissing: Did not find all entries for those hash ids!" errors if I do a very broad search and don't include "system:import time" tag.

>>20050 Was trying some stuff and found out that the single hash version evaluate-api works fine, but you have to add support for tag services by doing these simple edits in main.py (using wd-hydrus-tagger folder as an example): >copy line 113 and replace line 71 with it >change line 48 to: def evaluate_api(hash, token, cpu, model, threshold, host, tag_service): >copy line 82 and place it between lines 47 and 48 Then just add your edits like you did for evaluate-api-batch. Also not really related to your edits, but one thing I also noticed is that the hydrus version of the tagger replaces all _ characters for every tag, while the sd-webui version in >>20032 has an exclude list that leaves these tags 0_0, (o)_(o), +_+, +_-, ._., <o>_<o>, <|>_<|>, =_=, >_<, 3_3, 6_9, >_o, @_@, ^_^, o_o, u_u, x_x, |_|, ||_|| as they are. Just a thing I thought is worth mentioning in case someone feels like improving upon the scripts.

Since this morning the twitter parser seems to be broken, I tried upgrading to the newest version but it didn't work. They must have changed something again.

>>20050 Very nice, the tagging (for {dan/gel}booru at least) seemed solid on the 30 or so images I tested. I can't test this out on my main DB atm but I was able to convert the .bat scripts to .sh, since it's mainly python it was very easy. There's a preamble bc I was going to put them in a pastebin but the scripts aren't very long so I didn't think it would be necessary, I'll just paste the whole thing. If any shell god wants to improve them feel free, the hydrus setup_venv.sh could be a baseline for improvement, from the lines '$py_command -m venv venv' to 'python-m pip install -r requirements.txt' subtracting the "if [$install_type" ... " line, but I don't have the time for it right now. These are my best attempts at sh implementations of the bat scripts. The start.sh is suboptimal but I couldn't find a satisfactory way to activate a venv in the current shell with a script, AFAICT most scripts run in a subshell and the env gets removed when they're done, and it's pretty rude to edit someone's bashrc/zshrc without their permission. >wd.sh #!/bin/sh python -m wd-hydrus-tagger evaluate-api-batch --token REPLACE_WITH_API_KEY --tag-service "A.I. Tags" --cpu 1 hashes.txt >ratings.sh #!/bin/sh python -m ratings-hydrus-tagger evaluate-api-batch --token REPLACE_WITH_API_KEY --tag-service "A.I. Tags" --cpu 1 hashes.txt >e621.sh #!/bin/sh python -m e621-hydrus-tagger evaluate-api-batch --token REPLACE_WITH_API_KEY --tag-service "A.I. Tags" --cpu 1 hashes.txt >start.sh #!/bin/sh echo 'please type `source venv/bin/activate` without quotes in your terminal, then run `pip -r requirements.txt` also without quotes. You will need to run `source venv/bin/activate` every time you open a new shell and want to run these scripts'

>>20050 Well, GPU doesn't work, it's missing some dependency in the requirements.

hey dev I'm gonna email you one of those debug profiles things, I had sent one earlier but I'm not sure if you saw it. I'm >>19952 and I'm still having the issue. Seems to be something with autocomplete predicates, probably something dumb I setup and forgot. Also, is it possible to see a total of the inc/dec rating?

I had an ok week. You can type in system:rating predicates, PSD files show fully in the media viewer, and there are some additional quality of life improvements. The release should be as normal tomorrow. >>20058 Thanks, sorry if I missed it before, I'll check it out. No way to see total of inc/dec, but that's a neat idea. I'll figure out something for the review services panel.

>>20054 I've gone ahead and done this for wd and e621. I've also added some simple if/else statements to handle whether or not to only tag ratings, meaning ratings-hydrus-tagger is no longer necessary. All the changes should be on github already. >>20052 If I'm capable, I want to try and merge e621 and wd together next. I agree that it isn't exactly efficient to have 3 (now 2) separate copies of essentially the same code, but this looks a lot more complicated than basic if/else statements. Or maybe it is that simple, and I'm just not familiar enough to recognize it. Either way, I'll chip at it in the mean time.

>>20060 as I see it at least, you only need to add a new command line argument (say, "wd" or "e621" as the parameter, default to "wd" ig) then based on that decide if you should run the wd specific stuff or the e621 specific stuff. you can look at the other arguments defined by the original files as an example. this also would let adding more model types be straightforward too. good luck!

>>20046 A further test, 13/64 pairs exhibit erroneous behavior. None have existing duplicate groups. Not all of them have metadata, so that was a red herring. I did learn that going manage->regenerate->duplicate pixel data on both the newly imported file and the existing file seems to correct the error. I have yet to replicate this on a clean db.

https://www.youtube.com/watch?v=htZEYN1TkOE windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v539a/Hydrus.Network.539a.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v539a/Hydrus.Network.539a.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v539a/Hydrus.Network.539a.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v539a/Hydrus.Network.539a.-.Linux.-.Executable.tar.gz I had an ok week. There's full PSD support and a variety of quality of life improvements. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html PSD files Just like last week, if you run from source, you might like to rebuild your venv again this week, and you'll get the PSD support. Just run 'setup_venv' again. Thanks to a user, PSD files should now show in the program just like any other image! Since it takes a little extra CPU to render them, we're starting with conservative view settings--PSD files will show in full in the normal media viewer, but you'll get the cheaper 'open externally' button in the preview panel. If you want to try showing them everywhere, you can change the view settings under options->media. PSD thumbs are also much nicer now, and your existing PSD thumbnails should regenerate very soon in the background. As always, if you run across a PSD that has crazy colours or bad transparency--or if it straight up won't load at all, which we think some will--please send them in and I'll see what I can do. other highlights I'm splitting 'gif' files into 'static gif' (under images) and 'animated gif' (under animations) this week. It isn't a huge thing, and there aren't all that many static (i.e. non-animated) gifs going around anyway, but it simplifies some view settings as well as some code, and it aligns with how the other formats like png/apng work. Rating system predicates are now parseable. Type 'system:has rating (service_name)' or 'system:rating for (service name) > 4/5' into the tag autocomplete, and it should prefill the result. It should work for all like/disklike, numerical, and inc/dec rating services, and like the other parseable system predicates, this extends to the Client API. Try it out! I fixed the duplicate filter's right-hand hover window back in place. It was a mistake to have it 'avoid' the notes hover, since the buttons bounced around too much for easy clicking, so now they can just overlap, with the duplicates hover taking precedence. next week I'm due some larger work, and I think I'm going to tackle some file storage improvements. The migrate database dialog sucks, and the file storage system in general groans when it has millions of files, so I'd like to have the first steps towards user friendly storage settings and cleverer storage overall. We'll see!

>>20063 >I'm splitting 'gif' files into 'static gif' (under images) and 'animated gif' (under animations) this week. >and it aligns with how the other formats like png/apng work. >static gif I've got quite a few of them and the split comes handy. Thanks OP.

>>20063 pls >>19877 >>19970 >>19982

>>20050 I used to copy the images on Hydrus one by one to a separate directory, generate AI tags with sd-webui, and manually import them into Hydrus, but thanks to this, I no longer use such primitive methods. When I looked up how to make it work with GPU, I installed CUDA and cuDNN on my machine and it worked. However, although the GPU operation is fast, it is not as fast as I thought because it sends tag information to Hydrus one by one, but it is much more efficient than the above primitive method, so I am very happy. thank you. Auto-tagging 10,000 files took about 1 hour and 40 minutes with Hydrus installed on my HDD, but I'm sure it would go much faster if installed on fast storage.

>>20066 >When I looked up how to make it work with GPU, I installed CUDA and cuDNN on my machine and it worked. Weird that the webui version works without it. >I used to copy the images on Hydrus one by one to a separate directory Also what do you mean by this? Can't you just drag and drop all your files at once?

>>20055 Same, stopping working for me out of nowhere on v538 and upgrading to v539a changes nothing. Trying Nitter instead gives a 404

I noticed that the "edit default duplicate metadata merge options" for same quality and alternates are missing an edit button for rating services, while "this is better" has it.

>>20055 I confirm. Changing the parser to Nitter makes no difference. F

>>20068 >>20070 https://github.com/zedeus/nitter/issues/983 nitter back soon

https://rentry.org/ckmlai#ensemblefederated-wd-taggers using 3 taggers at once seems to get better results

>>20056 >typo Damn it >start.sh #!/bin/sh echo 'please type `source venv/bin/activate` without quotes in your terminal, then run `pip install -r requirements.txt` also without quotes. You will need to run `source venv/bin/activate` every time you open a new shell and want to run these scripts'

>>20043 >>20044 >>20049 When I first was starting with betas and stuff, this would have been 11 or 12 years ago, I posted a few times to 4chan /a/, /s/, /d/, and /g/ looking for users to help test things. I think I made full on release threads on 4chan /g/ or old 8chan /tech/ for a bit, but I'm forgetting. Sometime later I made a /hydrus/ board on the old 8chan (which was migrated to >>>/hydrus/ here). If you were anything to do with those, you may have seen me or the hydrus symbol appear here or there. I agree on the AI tags part. Human tags are rich, but they are expensive, and AI opens the door to making our tagging much more productive and less dreary. If you search these threads, you'll see me going on about how 'maybe neural networks will be able to tag one day', and then recently 'oh maybe we can use the PTR corpus to train MLs and think about auto-taggers', and now we are seeing Anons actually figuring it out. There's still lots to figure out and better workflows and tools to get working, but I'm really happy with it all and looking forward to the future. I like the PTR, but it has grown hundreds of times larger than I ever expected it to, and I'd love if generally simple and non-controversial stuff like 'skirt' and 'character:samus aran' could be done automatically with your own CPU. >>20045 >>20046 >>20062 Thank you, this is helpful. If files without existing groups are doing it, then it is definitely my code messing up at one stage in the process here, most likely in the actual dupe filter search where it is supposed to do the final pixel-match filter. I'll have a look and see what I can figure out. >>20047 Thanks, yeah that is awful. We haven't changed the style, so that must be the new Qt changing what colours it uses for tooltips, or generally how it decides to render them. I guess I'll edit the style just a little. >>20048 Yes, this was one of the annoying bugs with Qt 6.4.1. It seems there was additional overhead on certain widget creation on Win 10, and since a new page has so much stuff, it added up to about 3 seconds per page. A giant 100+ page session was taking minutes to load! This was one of the main reasons why I wanted to push up to the this new version so hard, since most/all Win 10 users on the build release were getting this lag. Since it seems we'll never escape from Qt bugs, we now, on 6.5.2, have this new tooltip colouring issue, incompatibility with older macOS and some versions of Linux, and certain versions of Win 10 with multiple monitors are getting schizo wrong-monitor menu and tooltip placement. However, I'm broadly happy with how this has gone, and now the 'running from source' setup script has the '(o)lder Qt6' option, which puts you on I think Qt 3.4.1, which we know was pretty stable and reliable, despite being slower and having older font and UI scaling tech. Please let me know how you continue to get on.

>>20050 Well done, this looks great! I will add a link to it in the Client API help so more people can find it. Again, let me know if there's any new handles or parameters I can add to the API to make your life simpler or if the link changes or anything. >>20051 >So, what exactly is the proper way to actually load up that db? You have to do it all manually. We are in advanced and generally dangerous territory here, so you just have to watch your step. I generally encourage people to never overwrite anything when they are restoring a backup, since it is too easy to accidentally screw up and overwrite the wrong thing. Either move the broken db folder out and switch the good one in, or extract a fresh install folder in a new place and put the good db in there. You should still run those routines, even on just a month difference. For the 'migrate database' dialog, it is tricky since I don't know your specific paths and stuff, but check pic related. Let's say when you boot, it gives you the 'I can't find your client_files' dialog. You tell that thing the correct location, but then when you are fully booted, you open up migrate database and its records are messed up, since it isn't sure what is a legit path and which is just some recovery USB drive you pointed it all. All or some of the files are currently in and/or ideally set to be in the wrong location. In my pic, I've put some of my files in a folder 'wrong'. See how it says 'ideal usage' is 100%, which we don't want. In this case, I would click 'D:\code\db\client_files' and click 'increase location weight', and then click 'wrong' and click 'remove location', and then you get Pic 2. Now the correct location is set to ideally have 100%, and the wrong is ideally set to have 'nothing'. Note we haven't moved any files yet, we've just set our preferences for where we ideally want stuff to go. Now we click the 'move files now' button and it all figures itself out. Pic 3, current and ideal usage are 100% where we want.

I have a file that has a modified time from i.pximg.net but for some reason, it doesn't show up in the bottom bar, or the top box when viewing the file. There are other files in the set from a different website, and those modified times do show up. Why are the pximg modified times not being displayed?

>>20051 Oh yeah and the 'repair missing locations' dialog looks like pic related. It isn't anything to be super scared about, and as you say you'll be hard pressed to set something up that actually damages something, but you do have to concentrate. >>20053 >No. The only thing that might be related was that I had to recover my database from a corrupted disk a little over a year ago, and I occasionally get "DataMissing: Did not find all entries for those hash ids!" errors if I do a very broad search and don't include "system:import time" tag. Ah, shit, that's our issue. Now I look, when you lose a master hash definition from client.master.db, I fill in any gaps with 'aaaaaaaaaaaaaaaa(random hex)'. Although it is good that I highlighted these damaged entries for later discovery, these aren't actually valid hashes, and I bet it is screwing with whatever recovery system is trying to work here. I now know what the problem is and I will see if I can fix it this week. Let me know how v540 works for you. >>20055 Yeah, I think it is over. yt-dlp broke for vids too. Maybe nitter can figure something out for a while, but I don't think this train is headed in our direction. I am praying that japanese twitter-only artists wake up and start posting to a real gallery site. >>20058 I did not receive this follow-up, if you sent it. Not in spam or nothing. I'm hydrus.admin@gmail.com . If you like, you can just pastebin it here--double-check so you are happy, but it doesn't have any personal info in it. Or hit me up on discord and we can talk live. https://discord.gg/wPHPCUZ

>>20076 btw, the pximg modified times are earlier than import time, so it's relevant that they should be displayed like the ones from the other website. And there aren't any other modified times on the file except for the local one. I see this happening with some other files too, so it's not just one file acting weird

>>20065 Thanks for the reminder. I'll bump it up my list again. >>20069 Thanks for this report. This is bad UI, and I'll do something. There's only one duplicate action for 'same quality' and 'alternates', since there is no concept of 'worse/better' here. I'll show the edit button again but throw up an info note or something if you click it. >>20076 >>20078 If you open manage->times on the file, what do you see? If I add a fake date for i.pximg.net on a file, it all seems to work correct, and by the sounds of your situation, it should be working. There are a couple of caveats though, so maybe there's a misunderstanding: The modified date you see is the aggregate of all the modified dates a file has. In general, outside of absurd times, it is the earliest of all the website modified times and the hard disk modified time (called 'file modified time' in the dialog). It should be reported in addition to the import time--import and modified shouldn't be merged. I wondered if I had an option somewhere that excluded modified timestamps from web domains that don't have a certain kind of matching URL Class definition, but I couldn't see one. I think all the domain modified timestamps are treated equally always, and that aggregate is just the earliest of them. EDIT: Ah, wait a sec, if the modified time is very close to the import time (I think like within 5% of the distance from now), I hide it. Could this be happening? I should probably have options around this behaviour.