regarding import folders, what's the difference between disabling the "check regularly" checkbox and enabling the "currently paused" checkbox. they both stop hydrus from autoimporting from the folders, right? so what's the difference? I didn't see a tooltip or info on the help pages about this.

I had an ok week with a variety of work. I fixed some bugs, tweaked some UI (including a neat change to shift+select of thumbnails), improved the new file history chart, and added to the Client API. The release should be as normal tomorrow.

https://www.youtube.com/watch?v=P7MsTw9s03o windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v479a/Hydrus.Network.479a.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v479a/Hydrus.Network.479a.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v479a/Hydrus.Network.479a.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v479a/Hydrus.Network.479a.-.Linux.-.Executable.tar.gz I had an ok week doing a mix of work. highlights I made it so when you shift-select some thumbnails, you can now move 'back' to deselect what you just selected. This also remembers what was previously selected before the shift-select started, so it works basically like an undo. I like how this works, but as part of it I had to make every thumbnail 'hit' focus in the preview viewer, which is not how all selects worked before. I already find this annoying, so I think I am going to make the system more clever and add some options around this behaviour. I think I improved the duplicate filter's zoom locking, particularly when one of a pair is portrait and the other is landscape. It should generally be more 'stable' now, but let me know if you still have any trouble. WebPs should now show transparency correctly! The new file history chart has another round of better math, an 'archived files' line, and you can hide the deleted files line if it is too big. I reworked a little help to make some 'ok, I know the basics, what next?' things clearer to find. If you missed learning about the autocomplete dropdown, tag wildcards, or OR searching, please check here: https://hydrusnetwork.github.io/hydrus/getting_started_searching.html full list - misc: - when shift-selecting some thumbnails, you can now reverse the direction of the select and what you just selected will be deselected, basically a full undo (issue #1105) - when ctrl-selecting thumbnails, if you add to the selection, the file you click is now focused and always previewed (previously this only happened if there was no focused file already). this is related to the shift-select logic above, but it may be annoying when making a big ctrl-selection of videos etc.. so let me know and I can make this more clever if needed - added file sort 'file->hash', which sorts pseudorandomly but repeatably. it sounds not super clever, but it will be useful for certain comparison operations across clients - when you hit 'copy->hash' on a file right-click, it now shows the sha256 hash for quick review - in the duplicate filter, the zoom locking tech now works better™ when one of the pair is portrait and the other landscape. it now tries to select either width or height to lock both when going AB and BA. it also chooses the 'better' of width or height by choosing the zoom that'll change the size less radically. previously, it could do width on AB and height on BA, which lead to a variety of odd situations. there are probably still some issues here, most likely when one of the files almost exactly fills the whole canvas, so let me know how you get on - webps with transparency should now load correct! previously they were going crazy in the transparent area. all webps are scheduled a thumbnail regen this week - when import folders run, the count on their progress bar now ignores previous failed and ignored entries. it should always start 0, like 0/100, rather than 20/120 etc... - when import folders run, any imports where the status type is set to 'leave the file alone' is now still scanned at the end of a job. if the path does not exist any more, it is removed from the import list - fixed a typo bug in the recent delete code cleanup that meant 'delete files after export' after a manual export was only working on the last file in the selection. sorry for the trouble! - the delete files dialog now starts with keyboard focus on the action radiobox (it was defaulting to ok button since I added the recent panel disable tech) - if a network job has a connection error or serverside bandwidth block and then waits before retrying, it now checks if all network jobs have just been paused and will not reattempt the connection if so (issue #1095) - fixed a bug in thumbnail fallback rendering - fixed another problem with cloudscraper's new method names. it should work for users still on an old version - wrote a little 'extract version' sql and bat file for the db folder that simply pull the version from the client.db file in the same directory. I removed the extract options/subscriptions sql scripts since they are super old and out of date, but this general system may return in future - . - file history chart: - added 'archive' line to the file history chart. this isn't exactly (current_count - inbox_count), but it pretty much is - added a 'show deleted' checkbox to the file history chart. it will recalculate the y axis range on click, so if you have loads of deleted files, you can now hide them to see current better - improved the way data is aggregated in the file history chart. diagonal lines should be reduced during any periods of client import-inactivity, and spikes should show better - also bumped the number of steps up to 8,000, so it should look nice maximised on a 4k - the file history chart now remembers its last size and position--it has an entry under options->gui - .

[Expand Post]

- client api: - thanks to a user, the Client API now accepts any file_id, file_ids, hash, or hashes as arguments in any place where you need to specify a file or files - like 'return_hashes', the 'search_files' command in the Client API now takes an optional 'return_file_ids' parameter, default true, to turn off the file ids if you only want hashes - added 'only_return_basic_information' parameter, default false, to 'get_metadata' call, which is fast for first-time requests (it is slim but not well cached) and just delivers the basics like resolution and file size - added unit tests and updated the help to reflect the above - client api version is now 29 - . - help: - split up the 'more files' help section into 'powerful searching' and 'exporting files', both still under the 'next steps' section - moved the semi-advanced 'OR' section from 'tags' to 'searching' - brushed up misc help - a couple of users added some misc help updates too, thank you! - . - misc boring cleanup: - cleaned up an old wx label patch - cleaned up an old wx system colour patch - cleaned up some misc initialisation code next week Next week is a medium sized job week. I would like move the 'notes' system forward. Top priority is to get some preview of notes on the media viewer, next to think about is duplicate file note merging and parsing notes from sites.

>>17567 >I would like move the 'notes' system forward. Top priority is to get some preview of notes on the media viewer This.

What's the easiest way to share a hydrus db/files instance between two different users on a shared network drive without causing concurrency / data integrity issues?

>>17567 >I had to make every thumbnail 'hit' focus in the preview viewer This is exactly how I want it to work, but since you're saying you find it annoying I'm guessing it's not gonna stay this way. I hope it becomes a configuration setting before you revert this behavior, because it's already making some of my file tagging jobs easier.

should i stop it or will it corrupt the database?

>>17559 Thanks!

>>17567 >Next week is a medium sized job week. I would like move the 'notes' system forward. Top priority is to get some preview of notes on the media viewer, next to think about is duplicate file note merging and parsing notes from sites. Would you consider adding a function parsing notes from a text file something like hydrus parsing tags from text files now

>>17571 You're fine I've done it hundreds of times. Your worrying about it suggests that you aren't backing things up regularly though. if that's the case then that's your most important issue atm

is there a way to manually add tags to a group of files, but not if the tag was previously deleted. Like performing a search then selecting all files to give them tags, but acting like downloaders do where they won't add the tag if the tag was already deleted from the file. Also is there a way to search for deleted tags specifically as part of a bigger search query? like say files that have the "blue hair" tag and also have the "female" tag deleted?

Accidentally quit hydrus while it was starting and upon starting again all my pages had been closed. A problem since I had a page with dozens of creator tags filtered out and I couldn't get it back from undo > closed pages or anything. Will pressing pages > sessions > append session backup > exit sessions > (some session) restore my pages or will it do something completely else? I couldn't find any documentation about sessions.

>>17576 Ya it's there to prevent exactly those kinds of situations. Hydrus's documentation isn't very good, but that's mostly because it's both a huge application with a lot of feature and also a constantly moving target.

>>17577 That saves me a ton of work, thanks!

>>17574 just downloaded the ptr before importing my shit to test if it will tag existing entries automatically, dont really want to do it again but ill have backups set up on my nas later

>>17555 >Is this a good backup practice? Yeah, I think you are good, maybe overkill to go to all three at once, unless you mean it is spread out over the drives. You have probably seen how well the database compresses, too. Should be able to get about -75% at least with a decent 7zip run. I personally do something more like >>17563 . For my different storages, I have a USB drive or network PC that I back up to once a week, and then once a month I back that up to a second 'cold' backup that sits in a safer place. Anything can break at any time and I'll still have at least two copies and worst case I lose a week of work. I migrate my drives from main storage to hot backup to cold as they get older. Even though it sounds excessive to have three copies, when something does break and you are currently stressed, you feel a lot different restoring to a new drive over fourteen hours when you have two of the thing remaining rather than just one. But for most people, especially those starting out, I think one backup once a week is completely fine. Like exercise, the details are far less important than keeping the schedule. >>17557 >>17560 Great example of this is when you are in a thread and see a 'Clipboard.png' filename. That happens when you 'paste' into an upload form, it is the filename your browser gives the file it pulled from your clipboard. This is fine when it is a screenshot of UI or something, something fresh that compresses great to png, but many times it is someone who went 'copy image' on a normal nice jpeg of an anime girl from a program like discord and pasted into the upload form. It pulls the raw bitmap data and makes a new lossless file in a super inefficient format. Just a pet peeve of mine, so I'll attack them first.

>>17561 >Would allowing multiple actions for a shortcut be something you're interested in adding? Yes, definitely. I still have a lot of shortcut work to do before I am ready, but I do want it. Anyway I can let people lego-brick their way to custom filters is fine by me. Since you are into this, make sure you are help->advanced mode and then hit file->shortcuts. You'll see 'custom shortcut sets', which are sets you can turn on and off in the media viewer with the little keyboard icon on the top hover. That whole system originally spawned from a first attempt at custom filters (with custom shortcut sets while they were alive). You might like to play around with some custom sets that assign 1,2,3,4,5 keys to tags or ratings or something and try turning them on and off for a bit. I'll expand this in future as shortcuts gets better integrated, so if you like it, let me know what you'd like for workflow improvements. >>17562 That's Jpeg XL, right? I'm super interested in that format, but my current block is I can only support what either PIL or OpenCV (or, at a stretch, FFMPEG) can handle. Can you post/point me to some example JXL files so I can test them my end? I think PIL has provisional support, but I haven't tested it yet. It might just be able to read height and width and things but not render. Similar problems with HEIF or AVIF or whatever they are called. Once PIL or OpenCV can do them, I basically just flick a switch, but being so new and generally complicated, it'll take time on their end. Sometimes there's patent bullshit too, but I don't know enough about that end. >>17564 My main hope here is to just improve the filter's queue. I want a whole bunch of options so you can just process pixel dupes or files that are super similar or have a more random set, or focus on finishing one group at a time, etc... The code in the filter is complete hell, and one of these 'medium' weeks I am going to cut it all out and rewrite it, and then I'll be able to work on the db side getting nicer queues. If I get that tech going, maybe I can copy it to the quick and dirty buttons (although really I want to ditch them, they were always a hack patching the filter). >>17565 Sorry, that's a stupid thing on my end where I accidentally exposed some technical stuff to the user. They are both basically the same. But if you say 'doesn't check regularly', you can still fire them manually with the 'check import folder now' menu. The 'pause' state should stop any run and is something I apply after an error. I should clean it up.

>>17569 Database on a shared drive is tricky. If the db is small, I would do this: - Store master copy of database on network - When you want to use it on a client computer, sync the db folder to local - Use the client db on your local machine, it points to files on network - When you are done, sync the local db folder back to network I know a user who does this and it apparently works ok. But if you have a 50GB db then it is probably too clunky. --database on network drive-- This is dangerous because a network interruption can cause database corruption. I know some people who got it to work ok though (with backups to be careful). I was just talking to a person today who was actually running their db on a SMB network drive and figured out settings that seem safe. If you have technical experience with this, you might want to try it out, but as always, make a backup beforehand. The comment they had on the underlying CIFS instance was: "need to disable cifs cache with cache=none and byte range locks with nobrl" Which is greek to me, but you may know more. Main thing in this situation, obviously, is to ensure that only one user connects with the shared db at a time. In the longer term future, I hope the Client API will evolve to allow clients to dial in to each other for seamless file service sharing over API. But I can't promise when that will be ready. >>17571 >>17574 Yeah, most of the time you can kill the process, even when the database is busy, and there will be no problems. It might take a minute or more to boot the next time as SQLite cleans up some remaining gubbins from the unclean exit. There's a small chance that killing the process right at the busiest time of a repository processing run will cause a weird id problem, but I have code to detect this and recover from it. If the current hard drive usage is less than 20MB/s, no worries™. The main issue for drive corruption is a power loss or other hardware failure. If the drive is writing a line of 0s to the disk platter because power is dropping, it doesn't matter how safe SQLite's journalling code is.

>>17573 That is definitely a long term goal. Main thing I want for this week is note preview on the media viewer, then I am going to work on two bits of tech: - Note merging object - Note parsing capability The merging object will basically be 'note import options' and will govern things like how to update an existing note if a new note with different content comes in. If the newly parsed note is just the old note but with appended text, we can probably overwrite no problem. Once I have that tech ready, I'll be able to think seriously about duplicate merging and then parsing notes from sites or external import files. Please keep reminding me about all this. Import and export of data to xml, json, txt, or whatever is something I want to do more of. >>17575 Unfortunately I don't have great support for searching or filtering by 'deleted status'. The one place where I do what you are talking about is when I pull tags from a website. The downloader system won't overwrite a deleted tag, but I assume if you are adding them manually, you are ok with overwriting a deleted status. I'd like to have more search options here. I have to do some other things first, but ideally I will be moving my tag search predicate from the current system, where it is basically text, to a full thing that can have service attached to it (e.g. 'samus aran on my tags') and some other things like storage/display tags and including weird characters like [ or not. So, I can't promise it any time soon, but this is something I would like to eventually support. >>17576 Sorry for the trouble! I didn't know it could do that. I'll write a hook to stop it saving exit/last session if it hasn't loaded the initial session yet.

>>17473 >running a search with a large file pool and multiple negated tags, negated namespaces, and/or negated wildcards should be significantly faster. an optimisation that was previously repeated for each negated tag search is now performed for all of them as a group with a little inter-job overhead added. should make '(big) system:inbox -character x, -character y, -character z' like lightning compared to before Thanks, much appreciated. t. >>17393 I have a bug to report. >v479 on Linux >Open new page->files->all local files >Select "Sort by file:hash" in top left >Put in a "system:limit is 20" search >Expected 20 images to show up >Instead, no results shown and exception is logged in the GUI Files related.

Is there a way to automatically add a file's filename to the "notes" of a Hyrdrus file when importing? Some of the files have date info or window information if they are screenshots and I'd like to store that information somehow. If not, is there some other way to store the filenames so that they can be easily accessible after importing?

>>17585 >notes I think notes are for when there's a region of an image that gets a label (think gelbooru translations), it's not the best thing for your usecase. The best way would be to have them under a "filename" namespace. In the file->import files dialog, after you've selected your files click on the "add tags before the import" button to open another dialog. This dialog has many more options including importing the filename as a tag under a namespace. By the way Hydrus dev, a standard UI idiom in menus is that when an item in the menu needs further user configuration in a dialog before the item can be considered 'done', the item's name ends with a "...". So the import files item would be called "import files..." and many other items in the dropdown and right click context menus should also end with a "...". Items don't end with a "..." if they don't need any configuration to finish their action after the user clicks on them (confirmation dialogs not counted as configuration). For example, items that open an external link or directory, or toggle a setting on or off, or whose purpose is explicitly to open some dialog and no more (as in the dialog is the end, not a means to an end).

Is it safe to let the downloaders rip and grab entire booru's? If not how do we get around that. We've got several large booru.org domains with 80K+ images that we need to back up. In the past I remember sites were very pissy about scrappers and I'd rather not eat bans or IP blocks for domains I'm still using.

>>17581 >Can you post/point me to some example JXL files so I can test them my end? I can't post any because 8chan itself doesn't yet support jxl, but the canonical example of a jxl file is the one on the community website: https://jpegxl.info/logo.jxl

>>17499 >>17501 Tag autocomplete on the Client API is working splendidly so far in my little Flask app. A simple (literally copy-paste from the website) HTMX implementation on the front end was all I needed for it to work perfectly. I added a new function to Cryzed's hydrus-api Python module to cover the /add_tags/search_tags API request. I think I got the tag_service_name and tag_service_key parameters correct, but I'm not using them at this point, so I can't be certain. I can share it if that would be helpful, but it's pretty straightforward anyway. A useful addition to /search_tags would be a limit parameter so that only X number of tags with the highest values are returned. Probably easier and faster to do that on the API side than to receive the full list on the front end app and truncate the results.

Could you add namespace relationships like you have tag relationships? It would be helpful to be able to set namespaces as parents of others, so that any tag in a child namespace also turns up in searches for the equivalent tag of the parent namespace. Having siblings for namespace could also help me to remove a lot of redundant namespaces. As it is right now, I just have to do everything on a per tag level, which is tedious and brittle.

Is there any way to get this thing to automatically pull tags from the galleryinfo.txt files in my multiTB hoard of shit I've downloaded off exhentai in the last decade or so?

>>17587 I do HUGE rips off of sites. Just tags I'm interested in, but I go as far back as the site will let me. Some let you go back years, all the way to the beginning. I'm do this on about 6 boorus, and when I began, I let Hydrus go 24/7 for about 3 months. None of the sites seemed to have a problem with it, none of them ever left me messages or banned me. Just don't try to hack past their rules, and you should be ok.

>>17591 You could write a parser for it, and point the parser at that file. I've made a parser for Deviant Art that got tags, but I don't know how you would point it at a file. Also interested how you could get it to read a file.

>>17502 It's been a while, but it seems that the "If file is missing, remove record" didn't work. I ran it and forgot about it, then noticed today they were still there. I ran it again and it said 0 files were missing despite no thumbnail and giving error 101 when I try to open them.

I had a good week. There's a simple first version of showing notes in the media viewer and several quality of life UI improvements. The release should be as normal tomorrow. >>17584 Thank you for this report. I have fixed it for tomorrow!

Is there something I can do with this error? Also thank you for your hard works!

>>17595 What version of glib2 does the linux build of hydrus use? I recently reinstalled my system and it doesn't want to function because of an error: >>8048 ./client: symbol lookup error: /usr/lib64/libgio-2.0.so.0: undefined symbol: g_module_open_full

https://www.youtube.com/watch?v=R1t6iNG28zI windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v480/Hydrus.Network.480.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v480/Hydrus.Network.480.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v480/Hydrus.Network.480.-.macOS.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v480/Hydrus.Network.480.-.Linux.-.Executable.tar.gz I had a good week. Notes now display on the media viewer. notes Notes have always been a slightly hidden system, a bit like ratings were. Today is a step forward to exposing them. Any file that has notes (you can start adding notes to a file by hitting manage->notes on their right-click menu) will now show them in the media viewer, just below the top-right hover window. They get their own hover window too, if you mouse over them. If you click on a particular note, the 'edit notes' dialog opens on it. This is a first version, and a little ugly, but I'm happy we now have something I can iterate on in future. If you are a big notes person, please let me know how it works best and worst for you. If you have unusual font style, size, or colour, let me know if it goes crazy or sizes too short or tall. While working on this, I rewrote the media viewer's hover windows to be more sensible, something I have been planning for a long time. They are now 'embedded' into the parent canvas, which should reduce a variety of jank behaviour--particularly, if you now click a hover, the main media viewer window no longer loses focus. There is still some hackery in the system to clean up, but I hope it'll work better overall for you. Unfortunately, I just did not get to note merge in the duplicates system or note parsing. That'll have to be for the future. the rest A user is working on a neat 'gallery share' system that plugs into the Client API, here: https://github.com/floogulinc/hyshare . It looks like a great replacement for my old 'local booru', so if you are interested in sharing groups of files straight from your client with friends over an attractive booru-like interface, check it out! I copied the 'file log' and 'search log' button menus, where you can do en masse actions like 'retry all failed' and 'export all to clipboard', to both the log review panels and the downloader/watcher list right-click menus. It is now possible to big actions on logs without highlighting anything. Just a small thing, but when you select a gallery in the gallery downloader page, the focus moves straight to the query text input, so you can start typing immediately. full list - file notes and media viewer hover windows: - file notes are now shown on the media viewer! this is a first version, pretty ugly, and may have font layout bugs for some systems, but it works. they hang just below the top-right hover, both in the canvas background and with their own hover if you mouseover. clicking on any note will open 'edit notes' on that note - the duplicate filter's always-on hover _should_ slide out of the way when there are many notes - furthermore, I rewrote the backend of hover windows. they are now embedded into the media viewer rather than being separate frameless toolbar windows. this should relieve several problems different users had--for instance, if you click a hover, you now no longer lose focus on the main media viewer window. I hacked some of this to get it to work, but along the way I undid three other hacks, so overall it should be better. please let me know how this works for you! - fixed a long time hover window positioning bug where the top-right window would sometimes pop in for a frame the first time you moved the mouse to the top middle before repositioning and hiding itself again - removed the 'notes' icon from the top right hover window - refactored a bunch of canvas background code - . - client api: - search_files/get_thumbnail now returns image/jpeg or image/png Content-Type. it _should_ be super fast, but let me know if it lags after 3k thumbs or something - you can now ask for CBOR or JSON specifically by using the 'Accept' request header, regardless of your own request Content-Type (issue #1110) - if you send or ask for CBOR but it is not available for that client, you now get a new 'Not Acceptable' 406 response (previously it would 500 or 200 but in JSON) - updated the help regarding the above and wrote some unit tests to check CBOR/JSON requests and responses - client api version is now 30 - . - misc: - added a link to 'Hyshare', at https://github.com/floogulinc/hyshare, to the Client API help. it is a neat way to share galleries with friends, just like the the old 'local booru'

[Expand Post]

- building on last week's shift-select improvement, I tweaked it and shift-select and ctrl-select are back to not setting the preview focus. you can ctrl-click a bunch of vids in quick silence again - the menu on the 'file log' button is now attached to the downloader page lists and the menu when you right-click on the file log panel. you can now access these actions without having to highlight a big query - the same is also true of the search/check log! - when you select a new downloader in the gallery download page, the keyboard focus now moves immediately to the query text input box - tweaked the zoom locking code in the duplicate filter again. the 'don't lock that way if there is spillover' test, which is meant to stop garbage site banners from being hidden just offscreen, is much more strict. it now only cares about 10% or so spillover, assuming that with a large 'B' the spillover will be obvious. this should improve some odd zoom locking situations where the first pair change was ok and the rest were weird - if you exit the client before the first session loads (either it is really huge or a problem breaks/delays your boot) the client will not save any 'last/exit session' (previously, it was saving empty here, requiring inconvenient load from a backup) - if you have a really really huge session, the client is now more careful about not booting delayed background tasks like subscriptions until the session is in place - on 'migrate database', the thumbnail size estimate now has a min-max range and a tooltip to clarify that it is an estimate - fixed a bug in the new 'sort by file hash' pre-sort when applying system:limit next week I would like to push multiple local file services some more. Probably some more infrastructure work in delete and import UI.

>>17598 Marvelous!!! It is a nice touch that when more than one note is present and clicking on that note, the Note dialog will open showing that precise tab on foreground. Something that at least for me is annoying is that Notes will open with the scroll at the end and not at the beginning as should be expected. See pic 3.

Hey, unfortunately 8chan has had some posting trouble and we have lost a week or so of posts. Since this thread is bumplocked, I was going to make a new thread for 481 anyway, so Hydrus General #4 is here: >>>/t/8151 This thread will be moved to the /hydrus/ archive soon. Thanks everyone!