Hydrus is the best, thanks hydev!

Sankaku is the worst, fuck that bulgarian cunt!

I have been manually tagging my files for over a year.

https://www.youtube.com/watch?v=xNKXyr6YN8Y windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v557/Hydrus.Network.557.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v557/Hydrus.Network.557.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v557/Hydrus.Network.557.-.macOS.-.App.dmg linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v557/Hydrus.Network.557.-.Linux.-.Executable.tar.zst I had a good week back after the holiday. There are some bug fixes and improvements to system:hash parsing. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html highlights Tag filters (which operate the various tag whitelists/blacklists across the program) now edit much much faster when they are full up with stuff. You can paste 5,000 items into an empty tag filter in less than a second now, and removing one or a handful of items from a filter with thousands of items already in is now instant. Previously, these things could take many seconds or even minutes due to inefficient overhead. CBZs that have four or fewer pages should now be recognised correctly. When you copy 'system:hash' and the 'system:similar to' predicates to clipboard, you now get a much longer string that includes all the hashes in the predicate. These copied strings are all parsable, meaning you can paste them into the same client elsewhere or a different client entirely and it should all just work in a complete loop. The human labels are updated to give you more information, too. There may be other predicates that are currently parsable if you type carefully but do not themselves copy a parsable string, but I am not sure which they are, so if you run into one, let me know and I'll see what I can do. If you run from source on Windows--or you'd like to--but you haven't installed the extremely convenient but bulky-to-install 'Git for Windows', I had to go through the installer again to set up my new dev machine, and I wrote out a guide for the full 12-page wizard here: https://hydrusnetwork.github.io/hydrus/running_from_source.html#core It is actually easy to do, and almost everything can be left as default and things will be fine. In any case, Git is great and lets you update in about three seconds, so if you run from source on Windows, give it a go! next week My old dev machine died just before Christmas, but I bought a new one and am back to normal work. For next week, I'd like to get some sort of system:num_urls figured out.

>>14276 Thanks man! You're program is Awesome!

Also, a shoutout to the people making the Ai taggers! You're making it even better!

>>14276 Glad you're back anon. /)

>>14270 Not a new issue, but README.md and some image files in docs/images/ and static/ are marked executable (755 rwxr-xr-x permissions), while .sh files are not marked executable. > For next week, I'd like to get some sort of system:num_urls figured out. \o/

is there any way to mass change my subscriptions from sankako to gelbooru? I have 987

>>14282 You can export and import subscriptions in the manage subscriptions dialog. Select a subscription and click "export" then "export as json". Open it in your text editor of choice. At the top you'll see the tag search the subscription uses. The tag search is identified by hash and by name in the data structure. See pic related. You'll have to do a find & replace to change the highlighted section. Export a subscription with a gelbooru tag search to see what it should look like. Find & replace the sankaku one with the gelbooru one. Import it again to verify it worked and is now searching on gelbooru instead. Once you know it works, select a bunch of your subs and modify them all at once. Also: Don't import/export all 987 subscriptions at once, that would probably crash hydrus or something. Some artist names may be different on gelbooru versus on sankaku, so you may have to change the query.

I accidentally double clicked the resize bar when resizing the preview and the whole preview vanished. How can I make the preview reappear?

Are there any issues stopping someone from making a generic *.booru.org downloader? There are many unique and specific boorus on there all using the same software, so it feels like a massive easy win. (I don't know the details of how the downloaders work, if it's a straight-forward task then I'm happy to try prototyping it, or even just a template for installing a [x].booru.org downloader)

>>14278 Seriously, I am often skeptical of ML hype but image recognition and tagging is one of the real legitimate uses of it. And with how atrocious most boorus are at tagging (with no intention of improving or even forcing people to add the most basic tags) this kind of thing will be a lifesaver.

>>14290 I'm using an older Ai tagger that got taken down, but if you look back at the old thread near the end, Gardivor? took it up and has it running. I'll probably switch to him later. If you set the threshold at .10, it will label loli, which is probably the most untagged type of pic there is. I hate that stuff, so I just delete it. It's really good at recognizing it. The only problem so far is that you get a lot of false loli positives on very realistic or pics of real people. It sees something in them that makes it think loli. Maybe it's a hint that the Ai thinks we're all children still. :p

>>14289 If the booru is indeed the same software, all you have to do is a make a new url class for that booru (and a gallery downloader) and link it to the already existing parser. Simply duplicate an existing url and change the domain. If you want an universal url, then I think that's not possible.

>>14291 Heh, it's trying to tell us something! I don't know much about loli crap either but I have gotten the impression that they made relatively more photo-realistic art than other communities because obviously real photos are illegal (let alone horrible for other reasons). If that is the case, an AI training model could end up linking the two too much and leaning into false positives.

>>14286 I will need to tweak some tags and delete thousands of dupes but this is a godsend is dupeguru the best way to find dupes this days?

>>14294 I've thought about something like this myself. Using something like NoClone to just go through the image files, then repairing Hydrus database by vacuuming. Just make sure you backup your database and all your image files before trying it. It would be interesting just to see how good Hydrus has been at avoiding dupes.

Does anyone know how to backup all the settings? Is there one config file or many and where to find it/them? I mean primarily the settings that one can find under 'file -> options', the so called 'manage options' window. Just in case i want to have those settings on other clients that dont have the same files and databases. I don't think the hydrus website explained this.

>>14294 dupeguru is good. but hydrus finds dupes already so i don't understand?

I posted this in the previous thread right at the end but I'm reposting it since I don't want it to get lost since I think it's an actually useful feature. >>14258 Would it be possible to expose information about subscriptions through the API? My thought was for 1-click exporting/importing subscriptions to hydownloader but I bet there could be other uses for it. I briefly looked at the API docs, I can't grok them very well, but I don't think this feature currently exists.

>>14295 > just to see how good Hydrus has been at avoiding dupes. I'll let you know in some hours Downloaded 42gb off gelbooru so far

>>14297 I just use a backup program named Macrium Reflect to back up all the Hydrus Database, thumbnails, and files. It works well.

>>14301 As I mentioned, I only want to backup the files that are responsible for the settings in file -> options ('manage options' window) and maybe other settings that can be setup elsewhere, to use in a client wiht other files and databases. As a general backup program, i already use FreeFileSync.

>>14293 You are stupid.

>>14304 No, he's not. It's good we have Ai tagging programs to tag what uploaders have uploaded untagged. Images and video without tags gets past our Blacklists. A few hours of the Ai looking at your latest mass download shows you what you have really downloaded. And then it's just a quick mass delete on all tags you didn't want.

>>14297 >>14302 The settings is part of the database (the 4 .db files), so you can't really backup only the settings, you'll have to backup the whole db. Though if you know sql, you could edit the db to only have the settings and then merge that into other clients, but I don't know which tables have the settings or if they are in multiple tables and such.

>>14287 Heh, that's a good one. I don't know, but if you open a new tab, it shows up again.

>>14308 Thanks for the help! Well, seems i have to record a clip or make screenshots then to get the settings over. Not a big deal. >>14287 Lol, first i thought "c'mon, it can't be that hard", then i tried it myself and got fucked too x) But restarting Hydrus did solve it actually. Restarting does reset the size of the preview panel to its default for all tabs/pages, so if a preview panel on one tab/page for example is big and the panel in another is completely gone, restarting Hydrus makes them the same size/visible again. >>14309 That helps too, but if you have a tab/page with complicated search terms and dont want to setup a search from new, you can just right-click the tab/page -> duplicate page. It gets the default preview panel size too. Hydev could maybe put in a setting (checkbox) if there isnt one already, which would save the state/size of the preview panel per tab, for those who want to keep it different per tab. Personally i don't think it's necessary though. Also the double clicking on the resize bar makes it vanish without a way to get it back. If you resize by mouse left-click drag and make it so small that it vanishes, it is possible to grab the resize bar and resize it to make it appear again. So maybe the double-click thing is a bug?

>>14307 I imagine he's referring to the claim that lolicons are making more realistic art than others.

>>14300 This shit took forever I feel like hydrus didn't detect a single dupe after switching from sankaku to gelbooru and just redownloaded fuckingeverything, got 76k dupes

>>14313 Hydrus doesn't do anything with dupes unless they are identical files, that's on the user to decide.

>>14314 Yeah, but I wonder if it's doing it on exact identical files as well. Some files in my collection sure look like identical dupes. I would have to run NoClone or something like it on the files to be sure though.

>>14315 Don't confuse visual duplicateswith mathematical duplicates. Hydrus only looks at the file's hash to determine duplicates during importing. If there is a single byte different between 2 files, even if they are otherwise exact pixel-for-pixel matches, they are NOT identical files. There's loads of reasons that can happen, most usually to do with metadata or from people "optimizing" images. You will just have to run thru the duplicate filtering process. 76k dupes will probably take less time than you think if you're smart with your filtering criteria.

>>14315 Even if it's a pixel for pixel dupe, if the hash is different, Hydrus will still wait for you to do manual duplicate processing. This is because some files will have metadata. It may be possible to set it up so it autodeletes pixel dupes and always chooses the one with or without metadata, but it still needs to be run manually after you download the files.

>i cant wait to be a useless piece of shit all day and tag all these images >FUCK IM EDGING FOR HOURS ....................................

>>14281 Thank you, I will see what I can do! >>14282 Just to add my thought here, I would either: A) create a new subscription, and then use 'copy queries' and 'paste queries' to copy the query texts from the old to the new B) just change the 'sankaku' to 'gelbooru' at the top of the subscription edit dialog, just under 'site and queries'. Also hit 'check now' on everything. A is more work, but it would be generally safer/cleaner and will lead to nicer timings. B should 'just work', I think, but it might take a couple weeks or months to figure out how often it should be checking everything. If you have lots of separate sankaku subscriptions, not one sub with many queries inside it, I would make use of the 'merge' button in the larger 'manage subscriptions' dialog. >>14287 >>14309 >>14310 Try pages->sidebar and preview panels->show/hide a couple times. I think that'll reset for all pages. >Hydev could maybe put in a setting (checkbox) if there isnt one already, which would save the state/size of the preview panel per tab, for those who want to keep it different per tab. Personally i don't think it's necessary though. Yeah, I'd like per-page options, I'm still toying with how to do it and not make a clusterfuck of certain other data stuff. This is related to the options object I'm talking about later, too, actually. >>14289 >>14292 BTW it is a near-term goal, fingers crossed, to finally figure out multi-domain URL Classes, which will make this job one step easier. We'll probably be able to spam a thousand booru.org domains into one. >>14297 >>14302 Sorry, there's no excellent solution to this right now. The settings are stored in a couple of different locations in the database, and parts of them are unique to each db, so are not migratable. I hope to have better options export/import in future! When I figure this out, I'll also sort out 'reset to default' tech too, which is still sorely lacking. If you are comfortable with SQLite, then head on in to client.db and rip out the dump_type=22 item in json_dumps. That's the main options object, but as I said, injecting it into a new database without any modifications might cause an error, so don't do it on anything now brand new or backed up. >>14275 >>14318 based >>14272 >>14277 >>14280 Thanks lads, keep on pushing.

>>14317 >It may be possible to set it up so it autodeletes pixel dupes and always chooses the one with or without metadata, but it still needs to be run manually after you download the files. Yeah, this is what I want. Delete all other dupes, and then merge all the tags into the one image left.

>>14319 >We'll probably be able to spam a thousand booru.org domains into one.

>>>13914 My nigger-rigged solution for tag definitions is a simple text file with one tag and its definition per line. Hopefully something like a booru wiki gets added some day, I quite like how existing booru sites can show example images and link to other similar tags.

Found a file that doesn't load. https://gelbooru.com/index.php?id=9439666&page=post&s=view

>>14320 ...you do know that hydrus has the entire duplicates processing system, right?

>>14327 That's not automated. You have to check each and every comparison, even if they're pixel dupes.

Very glad for the new Ai taggers. So much loli not being tagged now. I downloaded about 10000 new images from Pixiv ( a major culprit of no or few labels ), and the Ai tagger tagged over 900 of them as loli. Probably a few false positives, but 99% correct on them. So many uploaders not tagging anymore. And if you really hate some fetish or style, you don't want to be ambushed by it. Especially over 900 times! Thank you again people who made the Ai taggers! It is Appreciated!!

>>14324 here's another https://gelbooru.com/index.php?id=9454653&page=post&s=view

Can we get a "delete both" action in the duplicate processor? I hate it when I get about 100 files from an artist at once and I try to do duplicate processing to weed out what I already have or what I just got a better quality version of, but it's making me make decisions on a bunch of dupes of entirely new images within the download that I don't want.

One reason I don't like relying on the, Booru tags, AI tagging which is trained on booru tags, or the PTR which scrapes tags from boorus (I think), is that some big sites like danbooru and gelbooru strictly use the xgirls tag for futanari.

>>14333 Yeah, AI tagging sounds nice but until I can train something on my own collection I'm not too interested. There are several tags I use that boorus don't and vice versa, beyond what tag siblings can fix.

>>14332 Press the delete key on your keyboard.

>>14334 I find that the ML (machine learning is a much better term than AI, thought it's obviously the more popular expression) tagging at least gives me generic tags for things that had no tags, or very limited tags. I'm sure that eventually there'll be more / better models to choose from. I'd love an easy way to train your own models, but you would need a pretty large data set to do that.

>>14335 Damn, I'm retarded. Thank you. >>14336 How big must the dataset be? I have nearly 20,000 image files I've manually tagged so far.

I had an ok week. I fixed some bugs, improved some quality of life, and figured out a basic first version of 'system:number of urls'. The release should be as normal tomorrow.

What are good ways to find tags without parents (including parent tags without parents) with their quantities?

>>14154 Here is your 4chan comment parser and my parsers for subject, post number and flag. This is for 4chan thread parser -> subsidiary page parsers -> posts -> content parsers. [26, 3, [[2, [30, 7, ["comment", 18, [31, 3, [[[0, [51, 1, [0, "com", null, null, "com"]]]], 0, [84, 1, [26, 3, [[2, [55, 1, [[[9, ["<br>", "\\n"]], [9, ["</?[^>]+>", ""]], [9, ["&lt;", "<"]], [9, ["&gt;", ">"]], [9, ["&quot;", "\""]], [9, ["&amp;", "&"]], [9, ["&#039;", "'"]]], ""]]]]]]]], "comment"]]], [2, [30, 7, ["flag_name to \"meta:posted with flag\" tag", 0, [31, 3, [[[0, [51, 1, [0, "flag_name", null, null, "flag_name"]]]], 0, [84, 1, [26, 3, [[2, [55, 1, [[[2, "meta:posted with flag:"]], "Discord"]]]]]]]], null]]], [2, [30, 7, ["post number to \"meta:post:\" tag", 0, [31, 3, [[[0, [51, 1, [0, "no", null, null, "no"]]]], 0, [84, 1, [26, 3, [[2, [55, 1, [[[2, "meta:post:"]], "40643692"]]]]]]]], null]]], [2, [30, 7, ["post number to \"post #\" note", 18, [31, 3, [[[0, [51, 1, [0, "no", null, null, "no"]]]], 0, [84, 1, [26, 3, []]]]], "post #"]]], [2, [30, 7, ["subject", 18, [31, 3, [[[0, [51, 1, [0, "sub", null, null, "sub"]]]], 0, [84, 1, [26, 3, [[2, [55, 1, [[[9, ["<br>", "\\n"]], [9, ["</?[^>]+>", ""]], [9, ["&lt;", "<"]], [9, ["&gt;", ">"]], [9, ["&quot;", "\""]], [9, ["&amp;", "&"]], [9, ["&#039;", "'"]]], ""]]]]]]]], "subject"]]]]]

>>14339 and tags with 1, 2 or 3 parents.

https://www.youtube.com/watch?v=crJst7Yfzj4 windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v558/Hydrus.Network.558.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v558/Hydrus.Network.558.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v558/Hydrus.Network.558.-.macOS.-.App.dmg linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v558/Hydrus.Network.558.-.Linux.-.Executable.tar.zst I had an ok week. I figured out 'system:number of urls', and you can now import rtf files! Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html highlights So, you can now search for 'num URLs'. You will find it under a new stub, 'system:urls', at the bottom of the normal system predicate list (where 'system:known urls' has also moved). This first version is simple--it counts all URLs, regardless of how important, but I can see a future version having the ability to scan by 'post URLs' or specific URL classes. Give it a go and let me know if you run into any trouble! Also, thanks to a user, you can now import rtf files! There's no word count yet, but I think it should be doable in future. If you select multiple tags, you can now hide or favourite them all at once. There's a bit of workflow and presentation improvement here, too. The old 'tag suggestions' box called 'favourites' is now called 'most used'. It was stupid to have two tag systems called 'favourites', so this is now fixed. You still edit them under options->tag suggestions; they just have a more precise and less confusing name now. If you are an advanced macOS user (i.e. you know how to build something with xcode), you might like to check out Hydra Vista, a new user-made macOS App that presents your client (via the Client API, e.g., perhaps on another computer) in a booru-like wrapper. https://github.com/konkrotte/hydravista next week I would like to add timestamp editing to the Client API.

Hi everyone. I'm hoping someone can help me with making a downloader. It's an odd use case for Hydrus but I want to see if I can use it for managing a my shopping lists at Temu.com. What I'd like to do is this: For a product page, I want Hydrus to download all media (images/video) on a product page, associate the product page url with the images, and parse the page for keywords that I can use for adding tags. For example, many sellers sell the exact same items and use the same images so I can use Hydrus's image hash checking and duplicate filtersing to group the product page urls for the same image, which I can then use to make price comparisons.. The issue I'm having is that the website is heavily Javascript dependent, which I'm inexperienced in. The initial page request doesn't load/show any information initially and uses the scripts to make subsequent data fetches. I'm wondering if there is enough information in the html response I can use to do what I want or if there's too much Javascript that I couldn't use Hydrus realistically for the job. Any advise or help you can give would be awesome. Can anyone help?

New user trying to work out how to get around Hydrus, sorry for the dumb question. I'm still trying to work out the ergonomics of processing my initial import of ~20k files (and ongoing). A significant chunk of my files are booru hash names and Pixiv IDs. Is there no way to have these processed (pull tags from a booru/Pixiv) when importing them from the filesystem, only when downloading them through Hydrus? Is the expectation for users like me to rely on the PTR to save on manual tagging (or script it using the API)? I have found the file lookup script tab, but that's too involved to run on every file.

>>14344 I'm not an expert, but I mainly downloaded from two boorus which have a way for you to search images by their hash. So I made a basic script to go through each of the files in my formerly-horrendous Downloads folder, search for them on each site (it's just an API link), and if found, then use the Hydrus client API to make Hydrus automatically download that file with its tags. That second part is just: (shell script) curl --header "Content-Type: application/json" \ --request POST \ --data "{\"url\":\"$1\"}" \ -H "Hydrus-Client-API-Session-Key: ###########################################################" \ -H "Hydrus-Client-API-Access-Key: ##########################################################" \ http://127.0.0.1:45869/add_urls/add_url So there are certainly options for automation, and with 20k files you'll definitely want to lean on that (importing and manual basic tagging on 1k was enough work for me) but best to see if anyone here has made something more robust.

>>14345 That seems feasible enough with some regex to find files that look like IDs/hashes. Was a little afraid it would be more detailed based on some of the projects I found searching for Hydrus. I'll start throwing hashes at multiple boorus, pull tags into individual services, and use the main UI to consolidate them. Cheers.

When Hydrus subscriptions sync, and you're keeping the tags from the booru you got the files from, does Hydrus also check old urls for new tags and add them to your already downloaded files? How is this sync different from the file sync? It has to be, right?

>>14344 You do have tags downloading in your settings, don't you?

>>14348 This will scrape tags from the boorus. You can also do like I and a few others do, and use an Ai tagger to send tags to "my tags". I've found the Ai to be pretty precise, at least for the general tags. It even picks up things like "hair bow". :p

>>14348 Correct me if I'm wrong, but that doesn't do anything if you import files from disk ("initial import").

>>14345 >search for them on each site (it's just an API link), and if found, there's no need for your script to check if the booru has your file, hydrus can do that. just have a script that can go through a folder of files and for each one go from "081b69c578de52d33a3954d86d3b442a.jpg" to "https://gelbooru.com/index.php?page=post&s=list&tags=md5:081b69c578de52d33a3954d86d3b442a" and put the results in a text file. then simply copy paste all those urls into a url downloader page. >>14346 boorus typically use md5 which is 32 characters. something like "[a-f\d]{32}" is probably good enough.

How do I get Hydrus from source to use my qt6 theme on Linux? I have qt6ct set for dark mode but Hydrus stays in bright mode. I have been using one of the qss' but I prefer when everything follows the same theme.

>>14342 >So, you can now search for 'num URLs'. You will find it under a new stub, 'system:urls', at the bottom of the normal system predicate list (where 'system:known urls' has also moved). I don't see it, and don't recall ever seeing system:known urls. You don't have it hidden behind some setting that you always have on but is off for normal users, do you?

>>14324 >>14331 Thank you! >>14334 >>14336 >>14337 My private dream here is that as it becomes easier to train your own models for various things, we'll be able to pipe our non-tag data, like archive/delete decisions, into new models, too. I'd love to have a thing that knows what I like and dislike, even if it is only somewhat accurate, so I can spend my human time on only the most uncertain cases. There's also some fun 'probably in ten years' ideas like, if you train a model to eat an image and output 'you would 92% like this image', you can then invert it and say 'assuming 100% like, what pixels fit best?' and have it either produce novel images or modify existing images to be more to your liking. We've seen '(re)draw this image in the style of artist x' work super well, so while we aren't there yet in infrastructure terms, this whole idea generally seems feasible to me. I'll say, though, that while I understand the basic concepts here and I've dabbled in making pictures of princesses in front of castles like everyone else, I don't have the raw technical experience on the guts of making models, and as much as I have plans and notes I just don't have the time to do it, so I'll probably focus on making the Client API as open and convenient as possible, and we'll pray that an up-and-comer who is neck deep in this tech can provide the other side. >>14343 I do not work on downloaders directly much any more, but a general good trick when you have a javascript site is to load the site with your browser's developer mode on. Watch the network tab, and you'll see all the requests the js is making. Fingers crossed, there will be one request that is 'json', and it'll be something clean like site.com/v0/api/products/gubbins/search?page=0&num=50. That's the site's internal API. Download that file yourself and hopefully your browser will render the json beautifully and you'll be able to rip everything you need on the hydrus side by simply pointing at that URL instead and using a JSON parser. The 'API redirect' stuff in the URL Class UI helps out here, where you can say 'every time hydrus sees this html page, actually download and parse this json instead'. The imageboard parsers work this way, so check them out for an example. Unfortunately, a lot of sites don't use such nice JSON any more. The cloud-isation of sites, now a thousand different containers in some virtualised fog, means things are more machine-friendly and less simple for humans to go through. Often instead of 'search.php' or something, you'll get dynamically named '0d45gf.js' shit that hydrus just isn't clever enough to walk through.

>>14347 No, it does not re-check old URLs for new tags. It only gets new stuff, and furthermore, in general, any normal URL that hydrus has seen before is never visited again (that's the quick 'already in db' results you see in a typical repeat gallery download). There are a bunch of ways to force it to recheck, although it is ideally only ever on a one-time basis. Unfortunately, this is the technical issue at the heart of why I made the PTR; there just isn't a way to do this (on our scale) without wasting a ton of bandwidth and annoying server owners. I am planning a maintenance system/workflow that will allow us to do retroactive checks, where you'll hit up known urls for new metadata, but I will be careful how I do it. If you think about a medium collection of 300,000 files, and then checking them only once a year, that's still ~1,000 page hits (and multiples more if the files have several known urls) on top of your existing download work, every day, from every hydrus user, and only ever growing, in order to capture what is usually a handful of tags here and there. The PTR has a bunch of drawbacks, but it solves the syncing problem very efficiently. Some other guy runs a new subscription on the content you have already have, and his uploads to the PTR are then synced to your machine, and you don't have to do anything. >>14352 I'm sorry to say I don't think it is (easily) possible. Your system-level Qt6 themes are actually some clever C++ objects, non-portable, while hydrus's Qt6 is inside a private python environment and, afaik, can't always access them. I know some Linux users do see their system theme on boot, but if they alter anything under options->style, they lose the ability to switch back until a restart, so I think this is all based on some environment variable tech where the local python Qt is instructed to consult system-level .so files or theme directories, but I am just not expert enough to talk about it. >>14353 Sorry, try scrolling down, if it cuts off around 'tag as number' or similar. I think, since we added a couple, the default height setting for the autocomplete dropdown is like two short of the current system list length--I'll check this and make sure to expand it if it is.

>>14270 Can you make it possible to copy a list of tags from the 'Manage tag parents' dialog? Maybe highlight several and right click -> copy child tags or copy parent tags. As of now it seems to only copy the first child tag of the selection.

>>14355 >If you think about a medium collection of 300,000 files, and then checking them only once a year, that's still ~1,000 page hits (and multiples more if the files have several known urls) on top of your existing download work, every day, from every hydrus user, and only ever growing, in order to capture what is usually a handful of tags here and there. Can't you limit your scope so you're not looking for files missing "only a handful of tags", but instead files that are obviously under-tagged which are likely to get more tags later? Make sure it only checks files with "number of tags < x" once a year? I think your solution with the PTR is better though. >if it cuts off around 'tag as number' or similar Yep. If I change the monitor my Hydrus window is on to the smaller one, I can see the last two lines. May be an issue specific to me because having multiple different size monitors doesn't mesh well with that big qt update. Now that I know they're there though, I can just navigate to them with the arrow keys.

Trying to pull from a tumblr user using the gallery downloader, every download attempt results in an error status. "Looks like HTML -- maybe the client needs to be taught how to parse this?" here is one of the source urls: https://www.tumblr.com/blog/view/dovewingkinnie/738626839585701888 I'm trying to scrape posts from dovewingkinnie

>>14362 Yep. That looks like HTML. Reveal the code within your browser. i.e. on Firefox, its "inspect" when you right click on the page. You'll need to write a downloader / parser for it.

>>14362 >>14363 You can find how to do so in the "help and getting started guide" under the help tab in Hydrus. I remember it's buried in their somewhere. Good Luck. It's not easy.

>>14364 EDIT: I was just looking at Cuddlebear92's page. Under downloaders, there is a "Tumbler - liked - page" downloader. It might be what you're looking for. It's in .png format, so you can just import it into Hydrus.

>>14365 And, of course, I forgot the link. Here you go - https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Downloaders

>>14362 Don't use the hydrus tumblr downloader. It doesn't get every image for some reason. Example: https://putrydraws.tumblr.com/post/158563259427 It will only get the first 10 images even though there are 13. Use gallery-dl instead. https://github.com/mikf/gallery-dl But you may need to set up oauth for tumblr which requires an account.

Hydev, one of the sites I've been scraping periodically has been having SSL cert issues. Is there a way to bypass the error and continue the scrape, or do I just have to wait it out? >>14362 Use TumblThree, has a lot more options than both Hydrus and gallery-dl for downloading and supports logged-in only blogs.

>view file history >shows graph from db creation to today >add "system: import time: since 1 year ago" to search >still shows graph from db creation to today, but only shows stats for files that have been imported since 1 year ago Is there a way to limit the graph's X-axis itself? Once you have several years with one it gets a bit cramped. It would be nice if you could specify a start and end date to zoom in on that defaulted to 'creation' and 'now'. Also, minor bug with that graph: unchecking 'show deleted' and then modifying the search will revert to showing deleted (box still unchecked), but checking the box to show deleted will hide them again.

Something happened to gelbooru now? I got a new subscription to some artist that have 146 files configured it to get 200 on first check and only got 62 Just to be sure I did the same with gallerydl and got all the 146 files

>>14370 I did a reset just to be sure and still is only downloading 62 files

>>14371 >>14370 I see the issue now and why the lack of files, it is not downloading loli stuff Apparently now you need to actually login or something?

>>14372 Login didn't fix it Hydrus can't see stuff you need to login to see now on gelbooru, for some god damn reason

>>14370 >>14371 >>14372 >>14373 You don't need to login to see loli. You just have to go to Account (even if you don't have one) -> Options -> Check "Display all side content". It's been this way for very a long time, so maybe something in the html changed recently that prevents hydrus from automatically turning on the fringe content toggle?

>>14374 Yes because even with a login from some acc that have the "show all content" checked still won't download dem lolis I guess I'm not the only one that gave up on shitkaku and started raping gelbooru servers and they are upset

>>14375 The problem is entirely on your end. I just set up a new downloader to test it and it still gets loli files. Perhaps you set the downloader to grab files from the wrong booru and are instead getting files from some site that bans loli? Is your gelbooru tag downloader up to date with the one available on the github list?

>>14376 >Is your gelbooru tag downloader up to date with the one available on the github list? I'm using hydrus 557, am I supposed to update downloaders too? Where is it and how do I update it? Because now that I'm checking I have been missing stuff for around 2 weeks, when I updated, mebe something broke then

>>14377 Like, the gelbooru downloader from cuddlebear github is 4 years old and this is a 2 weeks issue

Did a test with a clean instance of 557 Still getting only 62 And of course its gelbooru

>>14379 Have you made a test downloader that just grabs the first few loli files on the site yet to see if it really is loli subscriptions that're the issue for you?

>>14380 I just added some mostly loli artist and got 17/413 files

>>14344 >>14345 >>14351 There's a way to do it without scripts: https://wiki.hydrus.network/books/hydrus-manual/page/file-look-up You might want to decrease the wait time between gallery searches to something like 1 second down from 15. The manual method using links from >>14351 may be faster though, because there's no wait time. >>14370 Just add this cookie for gelbooru in network > data > review session cookies.

The derpibooru parser doesn't fetch tags with data-tag-category="content-fanmade" (e.g., fanfic:anthropology), "spoiler" and "error". I am very disappointed, because I couldn't find what I downloaded using the tag. The following are content parsers for those tags. I don't know what "spoiler" and "error" are for, so they are namespaced. [26, 3, [[2, [30, 7, ["tags content-fanmade (unnamespaced)", 0, [27, 7, [[26, 3, [[2, [62, 3, [0, "span", {"data-tag-category": "content-fanmade"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]]]]], 0, "data-tag-name", [84, 1, [26, 3, []]]]], null]]], [2, [30, 7, ["tags error", 0, [27, 7, [[26, 3, [[2, [62, 3, [0, "span", {"data-tag-category": "error"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]]]]], 0, "data-tag-name", [84, 1, [26, 3, []]]]], "error"]]], [2, [30, 7, ["tags spoiler", 0, [27, 7, [[26, 3, [[2, [62, 3, [0, "span", {"data-tag-category": "spoiler"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]]]]], 0, "data-tag-name", [84, 1, [26, 3, []]]]], "spoiler"]]]]]

I had a great week. I completely overhauled how file timestamps are stored across the program, adding millisecond resolution so 'sort by import time' is more reliable, and I then added timestamp editing to the Client API. The release should be as normal tomorrow.

>>14391 RADICAL!

>>14392 Yes, my dick is radical.

So do I understand correctly that right now sankaku, hentai foundry and pixiv (explicit) are all borked and there's no way to fix any of them? sankaku with its yet another url type change that makes hydrus see links as galleries, the age old issue of PHPSESSID with HF (even when cookies are copied) and whatever's wrong with pixiv login.

>>14394 >sankaku with its yet another url type change that makes hydrus see links as galleries Just add a new url class. That's what I did. >pixiv (explicit) Just tested it right now. works for me.

https://www.youtube.com/watch?v=zBuJD-ugT_g windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v559/Hydrus.Network.559.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v559/Hydrus.Network.559.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v559/Hydrus.Network.559.-.macOS.-.App.dmg linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v559/Hydrus.Network.559.-.Linux.-.Executable.tar.zst I had a great week working mostly on one thing: converting the timestamps in the program from seconds to milliseconds. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html milliseconds tl;dr: time numbers better, you don't have to do anything Since the program started, it has tracked the various file 'times', like import and archive time, using a system that stores time with second precision. It knows that file x was imported at 2023-12-02 14:04:51, but nothing finer. This is common in many computing applications, but sometimes you want more. Particularly, in hydrus, whenever you have a fast importer that brings in two files within the same second, the program does not know, later, which was actually imported first (e.g. when you sort by 'imported time'). This can be annoying when you import a whole chapter of a comic and don't have good page tags set up yet--random pairs of pages will be flipped whenever you next try to sort those files by 'import time'. So, this week I converted the database to use a time system that has millisecond resolution. Any files imported from now on, or deleted, archived, modified, or 'last viewed', will now get a millisecond timestamp. You don't have to do anything, and nothing outside of the 'edit file timestamps' dialog will appear any different, but it may take a minute to update. This work was quite simple, but there was a ton of it, years and years of system build-up to go through. Rather than make an already messy system more complicated with a rough injection, I decided to invest the time and clean everything as I moved to the new format. I've tested it back and forth, but given the number of changes, I wouldn't be surprised there is a typo in there somewhere--if you see a file saying it was last viewed '54 years ago' or similar, let me know! Also, the Client API can now edit timestamps, including with the new millisecond precision. If you are an API dev, check out the new call (and 'edit file times' permission) here: https://hydrusnetwork.github.io/hydrus/developer_api.html#edit_times_set_time misc Just a side note, I have removed the sankaku downloader defaults from the program, so new users won't see them. None of them were working well/at all, especially in recent weeks. If you want to grab from complicated sites, and there isn't a clean solution on the shared downloader repo here, https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts , I recommend going with a more robust solution like gallery-dl/hydownloader or just finding the content elsewhere. If you ever run something like 'system:known url: regex = blah.com/(regex stuff)' and are annoyed at how slow it is, try adding a 'system:known url: domain = blah.com' pred in addition. This combination will now be explicitly much faster than just the bare regex (previously, it could be faster, by accident). next week Now we have ms timestamps and nicer code, I'd like to try tackling an 'edit timestamps' dialog that works for multiple files, including a 'cascade' command for force some clever sorts.

I was doing some work and wanted to sort files so that similar ones were next to each other. Would adding phash sorting help with this or is phash too simple for that? We already have hash sorting, so I think adding a phash option would be nice. Maybe even blurhash, since we have that now. Not sure how that one works though.

>>14397 >I was doing some work and wanted to sort files so that similar ones were next to each other. I wish. I think we're still waiting for "sort/group by alternates/duplicates".

>>14398 I think sorting by phash or blurhash would help with this a bit, unless it doesn't lol.

>>14270 Started using Hydrus 2 months ago when someone posted screenshots of it on /v/, I still have a lot of stuff to tag from my old "art inspiration" folder but it's so useful when I can type something like "pose:sitting" and get a nice assortment of references to use; or adding a couple of tags and find the exact photo I was thinking of. Related tags are a huge time saver as well, never could work that out as smoothly on other image organizers like XnView. Updating to the new v559 went smoothly, props to the dev of this great tool. There's something I'd like to ask, if I ever have to modify an entry (say cropping or rotating an image), is it automatically updated in the database or should I add the new modified image as a separate entry?

>>14400 The modified image will be considered a different image, you should re-import it and make sure to tag it. You will also probably need to manually set it as related to the non-modified version if you plan to keep the non-modifed one. If you overwrite the original from the place where Hydrus stores pictures (something like hydrus/files/f00/pic.png) Hydrus will see that as the file having become damaged.

>>14401 That clears it up, thanks!

>>14402 Note that if you do modify a file and then save it over the old one, when it finally notices it's "damaged", it will remove the file from your database and put it in a special bin for broken files. Hydrus recognizes files by their hashes and automatically changes the filename to the file hash when you first import it. If the filename and the hash later don't line up, Hydrus suspects the file is damaged. Not sure what happens if you save a new file manually into the database while also manually adding it's hash to the filename, but there's zero reason to do so, and it probably also similarly just eject illegal alien files.

>>14396 >Just a side note, I have removed the sankaku downloader defaults from the program, so new users won't see them. None of them were working well/at all, especially in recent weeks. Really? It still generally works fine for me, but I don't use it with an account. It will usually download images that don't require an account just fine. Occasionally it will download https://chan.sankakucomplex.com/redirect.png instead, but if I just wait a little bit or view the image in my browser first, that fixes it. I thought the people having issues were trying to use it to view account-only images.

Is there a solution for twitter yet?

>>14405 nitter.privacydev.net can view hidden stuff, but it's pretty slow. There is a nitter parser, but you'll have to make new url classes and gallery downloaders for this specific instance. Also I found out that the media tab doesn't show images that were posted as a reply, so it misses some stuff if you are trying to scrape whole accounts. I did some edits to the parser so it gets these images though, but it may still miss replies to other people. I could post it later when I can.

>>14406 Oh yeah, I think it also misses videos that are not "gifs", which I also fixed.

>>14405 >>14406 >>14407 Here it is.

>>14394 >So do I understand correctly that right now sankaku, hentai foundry and pixiv (explicit) are all borked and there's no way to fix any of them? Pixiv recently started filtering adult content results based on the country listed in your account profile. Far as I can tell all this does is filters out loli/shota if your country is not set to Japan. Just change your country in your profile to Japan and that may fix your issue with explicit Pixiv.

>>14395 You're the man. I never dabbled into settings that far and figured since the issue's been there for a while it was known to a dev and is unfixable. Turns out the culprit is an added /en/ in path. >>14409 I just checked and you're right, mine was set for Finland for some reason, I don't even remember changing it. Gonna test it after I'm done catching up with my sankaku subs, but the thing is on site itself I still could see explicit content, but the gallery jobs only picked up sfw art.

I have two issues. First is with identical/nigh identical duplicates. I always want one of them gone, but for tracking purposes I don't want to delete them with "this one is better", and "these are same quality" option only sets the relation without deleting either. I checked everywhere but couldn't find where to alter that behavior, is it possible? Second issue is with pixel dupes in particular. In these cases I always want the smallest one, and it should be a no-brainer automated job where I can hold the "this is better" hotkey and pick the first one. But the first one in comparison isn't always the smallest one due to tag weights and whatnot. So, obviously, I tried maximizing the size weights to maximum. It didn't work. So I set all the other weights to 0. Still it would ignore and not apply weight. For the sake of experiment I tried with regular non-pixel-dupe comparisons, and lo and behold it works as intended. But for some reason when it's a pixel dupe pair the weight is ignored. Maybe it's only when the option "must be pixel dupes" is chosen.

>>14410 When pixiv started this they said they would automatically set your country based on your location. See: https://www.pixiv.help/hc/en-us/articles/26578893289881-I-can-t-browse-certain-works-from-the-country-region-where-I-live Weird that you can view but hydrus can't though. Only other thing I can suggest is to maybe re-import your cookies for pixiv. Possible they added a region flag in there now.

>>14411 I know you can set "this is better" without deleting if you do it through gallery view by rightclicking a selection and going to manage > file relationships > set relationship and picking the "this is better" option, but that assumes you know which file is which and it can be more tedious. Another option would be using the duplicate filter and then simply undeleting the worse ones after.

>>14405 >Is there a solution for twitter yet? Sadly no, no one has managed to shut down Twitter yet.

>>14411 >but for tracking purposes I don't want to delete them What tracking do you mean? If you're worried about redownloading inferior files you already marked, Hydrus keeps track of the hashes of deleted files. I think there's even a way to see permanently deleted files, but the thumbs and images/videos/etc will be replaced with a Hydrus icon.

>>14416 >I think there's even a way to see permanently deleted files, but the thumbs and images/videos/etc will be replaced with a Hydrus icon. Yeah, but the record and the tags are still there. You would have to re-download the rest to actually get it back.

> Version 554 >the client looks for the cover image in your CBZ and uses that for the thumbnail! it also uses this file's resolution as the CBZ resolution I have several images that I add to a zip archive, which I rename as .cbz. Then I import it into Hydrus, but the preview does not appear. Am I doing something wrong? Thank you!

>>14417 Yes. But if he only wants it for tracking purposes and doesn't want the actual files, then that should be fine. Which is why I'm wondering what exactly his tracking purposes are.

Doing some AI tagging and the tagger failed leading me to look at my Hydrus log file. Error says malformed jpg file. v559, 2024/01/19 09:46:00: Problem generating thumbnail for "H:\Hydrus_Files\f79\79a34f164708ed9ad7db4dbc68d2427aa36a7a687c693fb2179f39ec1412a682.jpg". v559, 2024/01/19 09:46:00: ==== Exception ==== DamagedOrUnusualFileException: Could not load the image at "H:\Hydrus_Files\f79\79a34f164708ed9ad7db4dbc68d2427aa36a7a687c693fb2179f39ec1412a682.jpg"--it was likely malformed! ==== Traceback ==== Traceback (most recent call last): File "hydrus\core\files\images\HydrusImageOpening.py", line 10, in RawOpenPILImage File "PIL\Image.py", line 3309, in open PIL.UnidentifiedImageError: cannot identify image file 'H:\\Hydrus_Files\\f79\\79a34f164708ed9ad7db4dbc68d2427aa36a7a687c693fb2179f39ec1412a682.jpg' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "hydrus\core\files\HydrusFileHandling.py", line 251, in GenerateThumbnailNumPy File "hydrus\core\files\images\HydrusImageHandling.py", line 337, in GenerateThumbnailNumPyFromStaticImagePath File "hydrus\core\files\images\HydrusImageHandling.py", line 166, in GenerateNumPyImage File "hydrus\core\files\images\HydrusImageOpening.py", line 14, in RawOpenPILImage hydrus.core.HydrusExceptions.DamagedOrUnusualFileException: Could not load the image at "H:\Hydrus_Files\f79\79a34f164708ed9ad7db4dbc68d2427aa36a7a687c693fb2179f39ec1412a682.jpg"--it was likely malformed! ==== Stack ==== File "threading.py", line 973, in _bootstrap File "threading.py", line 1016, in _bootstrap_inner File "hydrus\core\HydrusThreading.py", line 451, in run File "hydrus\client\ClientFiles.py", line 3153, in MainLoopBackgroundWork File "hydrus\client\ClientFiles.py", line 2796, in _RunJob File "hydrus\client\ClientFiles.py", line 2573, in _RegenFileThumbnailForce File "hydrus\client\ClientFiles.py", line 1746, in RegenerateThumbnail File "hydrus\client\ClientFiles.py", line 618, in _GenerateThumbnailBytes File "hydrus\core\files\HydrusFileHandling.py", line 50, in GenerateThumbnailBytes File "hydrus\core\files\HydrusFileHandling.py", line 255, in GenerateThumbnailNumPy File "hydrus\core\files\HydrusFileHandling.py", line 66, in PrintMoreThumbErrorInfo File "hydrus\core\HydrusData.py", line 932, in PrintException File "hydrus\core\HydrusData.py", line 963, in PrintExceptionTuple ===== End ===== Looking for the file, it does not appear to exist at H:\Hydrus_Files\f79 path. I have a handful of similar errors for other files as well. Not sure what the best path forward is here.

>>14415 I hate Twitter too but unfortunately it's the only place artists I like post.

>>14356 Thanks, yeah, I'll see what I can do. >>14357 >Can't you limit your scope so you're not looking for files missing "only a handful of tags", but instead files that are obviously under-tagged which are likely to get more tags later? Make sure it only checks files with "number of tags < x" once a year? Yeah, I think that's basically going to be it. When I finally make this system, I'll guide users to maximising the efficiency with pre-built queries they can easily click, just like this. You'll be able to queue up whatever you want and set it to work whatever speed you want, but I'll have some fairly rigorous training wheels in place to start with so new/ESL/penguin-of-doom users don't run into too much trouble by accident. >>14368 I heard about this ssl stuff from another user. In that case, it was one particular CDN of a wider site, I guess that one server had a newer ssl version or a different security policy or something. I don't know what the key problem/answer is here, but I'm pretty sure that it is because ssl is pretty 'core' to the python version you have, and the built releases are still on Python 3.10, which is getting a little old now. I am hoping to have a 'future' test build on 3.11 in the next few weeks. I tried to do it last week but multiple things exploded so I need to alter the build scripts a little bit. Best answer, I think, is just to wait for that, and let me know how it goes. As a side thing, I'm also planning to move from python's 'requests' library to 'httpx', ideally within the next six months, so we can move from HTTP 1.1 to 2.0. I feel like I blinked and five sorts of webshit moved on (in truth it has been ten years, ha ha ha). >>14369 Yeah, sorry, I'd like to add custom axes and stuff to it, but much of it is still mostly 'ah I wonder if this works' v0.1 code. I'll fix the deleted box, thanks! >>14386 Thank you, I will add this!

>>14397 >or is phash too simple for that Unfortunately, the problem is the other way around. phash comparisons are very complicated, and there is no simple way to collapse their differences down them to A-Z, or smallest-to-biggest, or whatever-to-whatever. I think, when I eventually get around to the next duplicate files overhaul, that I'll try to add grouping or even a pseudo-similarity-sort that tries to isolate and group 'kings' with the most similar files or something and then sorts those groups by individual hamming distance, but I have to think about it more. We could sort phashes by hex, like the current 'sort by hash', but if any of the differences between files are within the first, say, eight bits, then the sort for that group of similar files is going to be whack. Tricky problem all around. EDIT: I have an idea and will try something. If you are interested, the way phashes are stored and searched in the database is a VPTree. This was one of the coolest things I worked on during all of hydrus. I always like to post this image with the discussion, since it visualises probably one of the 'simplest' ways of 'sorting'/ordering this data. https://en.wikipedia.org/wiki/Vantage-point_tree >>14400 Thanks, I'm really glad you like it. Feedback from new users is always helpful, so if there was anything in the help that you found confusing, or generally in the workflow of 'onboarding', let me know! >>14403 >Not sure what happens if you save a new file manually into the database while also manually adding it's hash to the filename, but there's zero reason to do so, and it probably also similarly just eject illegal alien files. This counts as an 'orphan' file in my system. There's no worries about it, but you can use the database->file maintenance->clear orphan files to scan your file folders for them and collect them in another directory. They typically happen when people restore an older database to a newer file storage, or vice versa, when recovering a backup. Or when my final file-delete code fucks up. >>14404 Please feel free to keep using it as long as it works for you. Unfortunately you are in the minority, and sank often change their site to discourage downloading, often with large IP filters or browser-detection code and similar. I can't officially keep up any more.

>>14411 1) Not sure if this helps your situation, but: This is a dirty secret of the duplicates system, but 'these are same quality' sets the same database relationship as 'this one is better', simply setting A>B, I think. On a technical level, my groups always need a 'king', so 'these are same quality' will bundle all the members of B's group under A. The main difference between these two settings is, as you have seen, the availability of delete, and the different metadata merge options. I think I've misunderstood the tracking you want, so if this is for something more complicated like assigning a particular delete 'reason' or something, please let me know. 2) Absolutely, we need some semi-automation and full-automation and better weight/scoring rules in the dupe filter. I hate the current hardcoded mess and I regret that I keep putting this big automation overhaul off, since it'll save everyone a lot of boring/simple work. Please keep reminding me! >>14420 tl;dr: right-click the thumb->manage->force filetype->set as cbz A core tenet of hydrus (currently) is that hydrus should be able to determine the type of a file regardless of its file extension. There are a bunch of technical reasons for this atm. So, if you rename a file, hydrus doesn't care. So, to determine the difference between a zip that happens to include some jpegs and a cbz, I wrote a custom scanner that has a bunch of rules. It looks for things like 'do the images seem to have common prefixes, and page numbers?' and 'does it have a random brush file?' and stuff, to discount various tool zips you might see on Deviant Art. The current code is here, if you are interested: https://github.com/hydrusnetwork/hydrus/blob/master/hydrus/core/files/HydrusArchiveHandling.py#L131 I've updated it several times in the past couple of months, and I'm broadly happy that we are capturing almost all legit typical cbz files you'd see on any comic/manga site. For weird cases, use the new 'force filetype' command (which I actually wrote specifically for cbzs!). Let me know if you run into any trouble. >>14423 Thank you, this is useful. It looks like that error is wrong, if it should be saying '404 file not found' etc... I expect that API call is working around the file load routine in an odd way, and I'll look at it. EDIT: I actually did just look at it now, and this thing should have given a 404 if the file was missing. Are you absolutely sure the file is missing, this isn't some weird Windows file-sort thing that is putting the file in an odd place because it is confused about '79' vs its neighbours '791' etc...? What if you paste 'system:hash=79a34f164708ed9ad7db4dbc68d2427aa36a7a687c693fb2179f39ec1412a682' into a normal search page? If the file is truly missing, check out the 'help my media files are broke.txt' file in 'install_dir/db'. If the files do exist but they are busted, there aren't excellent solutions. You can send it to me, and I'll check it out. Some older versions of our image library was happy loading truncated stuff, but the newer version isn't (and I turned on some safety code). Just finding the file with 'system:hash=79a34f164708ed9ad7db4dbc68d2427aa36a7a687c693fb2179f39ec1412a682' and then deleting the record is probably the best answer. Or, if it seems to be truly completely whack-broken (e.g. if you inspect it with a hex editor, its data is now all zeroes; something that never would have imported ok to hydrus in the past, so the content must have changed since then), it may be it has been damaged by HDD failure, and then we are talking a different kettle of fish. In this case, you will want to right-click it and hit manage->maintenance->if file is missing/incorrect and has URL then try to redownload'. That may fix you up. If you are confident you have'' had hdd damage, and this is not just some weird booru file that happened to slip in one day, then you'll want to review the 'help my db is broke.txt' document as well, which has background reading on how to check everything else is ok. Let me know how you get on, regardless.

>>14428 Isn't that one of Hunter's paintings?

ah shit bros, here we go again I stupidly did a "git pull" while hydrus was still running and now hydrus won't start. Here's the crash when running in terminal (I don't even get a window to open when trying to run normally) Traceback (most recent call last): File "PATH/hydrus/hydrus_client_boot.py", line 19, in <module> from hydrus.core import HydrusBoot File "PATH/hydrus/core/HydrusBoot.py", line 3, in <module> from hydrus.core import HydrusConstants as HC File "PATH/hydrus/core/HydrusConstants.py", line 6, in <module> import yaml ModuleNotFoundError: No module named 'yaml' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "PATH/hydrus/hydrus_client_boot.py", line 178, in <module> from qtpy import QtWidgets ModuleNotFoundError: No module named 'qtpy' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "PATH/hydrus_client.py", line 7, in <module> from hydrus import hydrus_client_boot File "PATH/hydrus/hydrus_client_boot.py", line 188, in <module> HydrusData.DebugPrint( 'Could not start up Qt to show the error visually!' ) ^^^^^^^^^^ NameError: name 'HydrusData' is not defined ~partialPATH

>>14431 Looks like the venv got messed up, try reinstalling it

I'm still learning how to use hydrus and i'm not a coder/programmer by any means. Is there a way to have hydrus sort through the images i have? I noticed a lot of duplicates because some of these images all have different file sizes but can i use hydrus to find and replace the inferior versions automatically?

>>14433 Hydrus doesn't know what's "inferior". Just because an image is larger in filesize or resolution doesn't mean it's better. It can have worse artifacting, or not actually be a duplicate, but an alternate from a set of images. It may be possible to automate in the case of perfect pixel dupes though, to always grab the smaller one if you don't care about metadata. Anything with the exact same hash is automatically combined, so you don't have to worry about that.

>>14429 I think I may have confused myself and the issue, apologies. I may have went through and did a delete in Hydrus on a few images that I found that threw an error before I went into the log and posted that excerpt. Here's another instance of the issue. This file does exist when I search for it, and does not open properly. Something's broken with it. v559, 2024/01/19 09:50:24: ==== Exception ==== DamagedOrUnusualFileException: Could not load the image at "H:\Hydrus_Files\fcc\ccd90fdb2a7eb885c9824e4e6a6397c05fca5ecef1b44378b8aa098a55fc6b23.jpg"--it was likely malformed! ==== Traceback ==== Traceback (most recent call last): File "hydrus\core\files\images\HydrusImageOpening.py", line 10, in RawOpenPILImage File "PIL\Image.py", line 3309, in open PIL.UnidentifiedImageError: cannot identify image file 'H:\\Hydrus_Files\\fcc\\ccd90fdb2a7eb885c9824e4e6a6397c05fca5ecef1b44378b8aa098a55fc6b23.jpg' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "hydrus\client\ClientRendering.py", line 249, in _Initialise File "hydrus\client\ClientImageHandling.py", line 33, in GenerateNumPyImage File "hydrus\core\files\images\HydrusImageHandling.py", line 166, in GenerateNumPyImage File "hydrus\core\files\images\HydrusImageOpening.py", line 14, in RawOpenPILImage hydrus.core.HydrusExceptions.DamagedOrUnusualFileException: Could not load the image at "H:\Hydrus_Files\fcc\ccd90fdb2a7eb885c9824e4e6a6397c05fca5ecef1b44378b8aa098a55fc6b23.jpg"--it was likely malformed! ==== Stack ==== File "threading.py", line 973, in _bootstrap File "threading.py", line 1016, in _bootstrap_inner File "hydrus\core\HydrusThreading.py", line 451, in run File "hydrus\client\ClientRendering.py", line 259, in _Initialise File "hydrus\core\HydrusData.py", line 932, in PrintException File "hydrus\core\HydrusData.py", line 963, in PrintExceptionTuple ===== End ===== For this file, Hydrus displays a screenshot, but when I try to open the file, I get an error in Hydrus. And the file won't open from the file system in Irfan view. However, when I look at the file in mediainfo, it turns out to be a webm. Not sure why this was saved as a jpg, but is really webm. That seems odd. Complete name : H:\Hydrus_Files\fcc\ccd90fdb2a7eb885c9824e4e6a6397c05fca5ecef1b44378b8aa098a55fc6b23.jpg Format : WebM Format version : Version 2 File size : 2.96 MiB Duration : 15 s 140 ms

>>14434 Ok thanks for the info. I can sort them myself i just didn't want to put the time in if there was an easier method out there.

>>14432 Goddamnit, I can't even resolve some python dependency problem on my own :/ PATH /home/wir/Projects/Creativity/Collection/hydrus r::::::::::::::::::::::::::::::::::r : : : :PP. : : vBBr : : 7BB: : : rBB: : : :DQRE: rBB: :gMBb: : : :BBBi rBB: 7BBB. : : KBB: rBB: rBBI : : qBB: rBB: rQBU : : qBB: rBB: iBBS : : qBB: iBB: 7BBj : : iBBY iBB. 2BB. : : SBQq iBQ: EBBY : : :MQBZMBBDRBBP. : : .YBB7 : : :BB. : : 7BBi : : rBB: : : : r::::::::::::::::::::::::::::::::::r hydrus If you do not know what this is, check the 'running from source' help. Hit Enter to start. -------- Users on older OSes or Python >=3.11 need the advanced install. Your Python version is: Python 3.12.1 Do you want the (s)imple or (a)dvanced install? -------- Creating new venv... Requirement already satisfied: pip in ./venv/lib64/python3.12/site-packages (23.2.1) Collecting pip Obtaining dependency information for pip from https://files.pythonhosted.org/packages/15/aa/3f4c7bcee2057a76562a5b33ecbd199be08cdb4443a02e26bd2c3cf6fc39/pip-23.3.2-py3-none-any.whl.metadata Using cached pip-23.3.2-py3-none-any.whl.metadata (3.5 kB) Using cached pip-23.3.2-py3-none-any.whl (2.1 MB) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 23.2.1 Uninstalling pip-23.2.1: Successfully uninstalled pip-23.2.1

[Expand Post]

Successfully installed pip-23.3.2 Collecting wheel Using cached wheel-0.42.0-py3-none-any.whl.metadata (2.2 kB) Using cached wheel-0.42.0-py3-none-any.whl (65 kB) Installing collected packages: wheel Successfully installed wheel-0.42.0 Collecting cbor2 (from -r requirements.txt (line 1)) Using cached cbor2-5.6.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.0 kB) Collecting cryptography (from -r requirements.txt (line 2)) Using cached cryptography-41.0.7-cp37-abi3-manylinux_2_28_x86_64.whl.metadata (5.2 kB) Collecting dateparser (from -r requirements.txt (line 3)) Using cached dateparser-1.2.0-py2.py3-none-any.whl.metadata (28 kB) Collecting pympler (from -r requirements.txt (line 4)) Using cached Pympler-1.0.1-py3-none-any.whl (164 kB) Collecting python-dateutil (from -r requirements.txt (line 5)) Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) Collecting beautifulsoup4>=4.0.0 (from -r requirements.txt (line 7)) Using cached beautifulsoup4-4.12.3-py3-none-any.whl.metadata (3.8 kB) Collecting chardet>=3.0.4 (from -r requirements.txt (line 8)) Using cached chardet-5.2.0-py3-none-any.whl.metadata (3.4 kB) Collecting cloudscraper>=1.2.33 (from -r requirements.txt (line 9)) Using cached cloudscraper-1.2.71-py2.py3-none-any.whl (99 kB) Collecting html5lib>=1.0.1 (from -r requirements.txt (line 10)) Using cached html5lib-1.1-py2.py3-none-any.whl (112 kB) Collecting lxml>=4.5.0 (from -r requirements.txt (line 11)) Using cached lxml-5.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.5 kB) Collecting lz4>=3.0.0 (from -r requirements.txt (line 12)) Using cached lz4-4.3.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.7 kB) Collecting numpy>=1.16.0 (from -r requirements.txt (line 13)) Using cached numpy-1.26.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB) Collecting psd-tools>=1.9.28 (from -r requirements.txt (line 14)) Using cached psd_tools-1.9.30-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.7 kB) Collecting Pillow>=10.0.1 (from -r requirements.txt (line 15)) Using cached pillow-10.2.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (9.7 kB) Collecting pillow-heif>=0.12.0 (from -r requirements.txt (line 16)) Using cached pillow_heif-0.14.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.3 kB) Collecting psutil>=5.0.0 (from -r requirements.txt (line 17)) Using cached psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB) Collecting pyOpenSSL>=19.1.0 (from -r requirements.txt (line 18)) Using cached pyOpenSSL-23.3.0-py3-none-any.whl.metadata (12 kB) Collecting PySocks>=1.7.0 (from -r requirements.txt (line 19)) Using cached PySocks-1.7.1-py3-none-any.whl (16 kB) Collecting PyYAML>=5.0.0 (from -r requirements.txt (line 20)) Using cached PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB) Collecting Send2Trash>=1.5.0 (from -r requirements.txt (line 21)) Using cached Send2Trash-1.8.2-py3-none-any.whl (18 kB) Collecting service-identity>=18.1.0 (from -r requirements.txt (line 22)) Using cached service_identity-24.1.0-py3-none-any.whl.metadata (4.8 kB) Collecting Twisted>=20.3.0 (from -r requirements.txt (line 23)) Using cached twisted-23.10.0-py3-none-any.whl.metadata (9.5 kB) Collecting opencv-python-headless==4.7.0.72 (from -r requirements.txt (line 25)) Using cached opencv_python_headless-4.7.0.72-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (49.2 MB) Collecting python-mpv==1.0.3 (from -r requirements.txt (line 26)) Using cached python_mpv-1.0.3-py3-none-any.whl (44 kB) Collecting requests==2.31.0 (from -r requirements.txt (line 27)) Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB) Collecting QtPy==2.3.1 (from -r requirements.txt (line 29)) Using cached QtPy-2.3.1-py3-none-any.whl (84 kB) ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11; 6.0.0 Requires-Python >=3.6, <3.10; 6.0.0a1.dev1606911628 Requires-Python >=3.6, <3.10; 6.0.1 Requires-Python >=3.6, <3.10; 6.0.2 Requires-Python >=3.6, <3.10; 6.0.3 Requires-Python >=3.6, <3.10; 6.0.4 Requires-Python >=3.6, <3.10; 6.1.0 Requires-Python >=3.6, <3.10; 6.1.1 Requires-Python >=3.6, <3.10; 6.1.2 Requires-Python >=3.6, <3.10; 6.1.3 Requires-Python >=3.6, <3.10; 6.2.0 Requires-Python >=3.6, <3.11; 6.2.1 Requires-Python >=3.6, <3.11; 6.2.2 Requires-Python >=3.6, <3.11; 6.2.2.1 Requires-Python >=3.6, <3.11; 6.2.3 Requires-Python >=3.6, <3.11; 6.2.4 Requires-Python >=3.6, <3.11; 6.3.0 Requires-Python <3.11,>=3.6; 6.3.1 Requires-Python <3.11,>=3.6; 6.3.2 Requires-Python <3.11,>=3.6; 6.4.0 Requires-Python <3.11,>=3.6; 6.4.0.1 Requires-Python <3.12,>=3.7; 6.4.1 Requires-Python <3.12,>=3.7; 6.4.2 Requires-Python <3.12,>=3.7; 6.4.3 Requires-Python <3.12,>=3.7; 6.5.0 Requires-Python <3.12,>=3.7; 6.5.1 Requires-Python <3.12,>=3.7; 6.5.1.1 Requires-Python <3.12,>=3.7; 6.5.2 Requires-Python <3.12,>=3.7; 6.5.3 Requires-Python <3.12,>=3.7 ERROR: Could not find a version that satisfies the requirement PySide6==6.5.2 (from versions: 6.6.0, 6.6.1) ERROR: No matching distribution found for PySide6==6.5.2 -------- Done! PATH

>>14438 Python 3.12 is the problem here; the cleverer libraries like Qt (PySide6) have very little support yet. You may be in luck though, somewhat--I saw the very very new PySide6 6.6.1 just added Python 3.12 support, although how good that is I am not sure: https://code.qt.io/cgit/pyside/pyside-setup.git/tree/doc/changelogs/changes-6.6.1 So, to get working on your situation, try again, selecting 'a' for advanced and 't' for 'test' everything, except for the Qt question: there, write 'w' for write your own, and hit '6.6.1'. For QtPy, enter '2.4.1'. WARNING: I tried this the other day and noticed that in 6.6.1, there's a bug in my multi-column list system that makes it unable to size any column widths. They will all be ugly wide. I mean to fix this soon, but I haven't put time into it yet. Ideally, I think your better solution here is to figure out how to deploy this venv with Python 3.11, if possible. 3.12 is still pretty new, I'm afraid. Maybe you can copy/edit the 'setup_venv' script to say: 'py311 -m venv venv' -instead of- 'python -m venv venv' at the critical line where it creates the venv. I guess it depends on what symlink you have 3.11 set as in your PATH, if any. I am not sure what Linux usually symlinks any older version to, but if it is there, I think it'll be something like that. Let me know how you get on--I'll see about integrating this as a selectable option in the setup venv thing.

Anyway I can set ctrl+C in the tag/parent/sibling managers, search, autocomplete, and basically anywhere I have a tag selected to copy the full "namespace:tag". Right click -> copy -> namespace:tag is getting tiresome. I can't seem to find something like this anywhere in the shortcuts, and it just seems intuitive that if you can right click -> copy something you have selected that ctrl+C ought to do the same much faster. I know there's multiple copy options, but ctrl+C should at least do the most basic one that's listed first.

>>14433 It's not automatic but you know of the duplicates system, right? https://hydrusnetwork.github.io/hydrus/duplicates.html

>>14440 Yeah, this would be very handy.

It would be nice to be able to search by domain modified time and "has/does not have domain modified time".

Hey lads, hydev here, my internet died and it could be a week before it is fixed. I don't have a good solution for my dev logins to talk to the internet, so no release or posts for a bit, will update later!

>>14444 Do those even exist? I only see one global modified time.

>>14446 They're under right-click thumbnail > manage > times

>>14447 I know, but domain times are for stuff like imported/deleted times, not modified, which there is only one. I think I'm just getting semantic here.

>>14445 Understood.

>>14445 Well be waiting!

I'm curious how hydev feels about a verify system for the PTR, some way in which when other users run into a same hash that exists on the PTR they can send a note to the PTR saying hey, this file exists in the public domain I also have it, and then (eventually) garnering some form of verification to the files that exist on the PTR, this is a bit of a spitball to an idea that came to mind, could help with cutting down on the size of the PTR

>>14454 I'm a little confused. That doesn't sound like it'll cut down the size at all, because every hash will still have to be in the PTR for users to confirm it. And what would the point of verification even be?

>>14455 The general concept would be after a certain period of time without confirmation it would get pruned automatically, not sure what that time frame would look like. The point is to remove the bloat of hashes uploaded that nobody else has the file, which I imagine users have done so.

> https://8chan.moe/hydrus/res/20352.html#21117 Added colors to some lines. #!/usr/bin/python3 import os import subprocess import sys import tempfile import pyperclip # get the exported siblings or parents (a list of lines representing a table) #exported = sys.stdin.readlines() # From stdin. Be careful not to paste dangerous commands into the terminal! exported = pyperclip.paste().splitlines() # From clipboard. # node names are enclosed in double quotes, escape them (not tested) exported = [s.replace('"', '\\"') for s in exported] children = exported[::2] parents = exported[1::2] table = zip(children, parents) # make an actual table result = [] result.append("digraph {") # a directed graph color_isolated = " [color=red]" color_nonparent = " [color=blue]" color_nonchild = " [color=orange]" color_favnonchild = " [color=green]" # list of correct top-level parent tags favnonchildren = ["species:mammal"] for child, parent in table: if child not in parents and parent not in children: # line between tags with no other parents/children color = color_isolated elif child not in parents: # the child tag has no children color = color_nonparent elif parent not in children: # the parent tag has no parents if parent in favnonchildren: color = color_favnonchild else: color = color_nonchild else: color = "" result.append('"{}" -> "{}"{}'.format(child.strip(), parent.strip(), color))

[Expand Post]

result.append("}") dotinput = bytes('\n'.join(result), encoding='utf8') # sent to dot's stdin, it returns PNG data dotoutput = subprocess.run(['dot', '-Tpng'], input=dotinput, capture_output=True) with tempfile.NamedTemporaryFile(suffix='.png', delete=False) as png: png.write(dotoutput.stdout) pngname = png.name subprocess.run(['mimeopen', pngname]) # Open the viewer. input() # The next line removes the file, so the viewer may lose it. So we wait for user's Enter. os.remove(pngname)

>>14454 >they can send a note to the PTR saying hey, this file exists in the public domain I also have it I'd really like it if Hydrus doesn't automatically send notes to the ptr saying exactly which files each account has. If you're gonna open that can of worms, why not go all the way and only download the entries for files you have?

is there a login script somewhere for lolibooru.moe?

>>14461 I'd really like if you had a more charitable understanding of what I posted. It would be opt-in, mouth breather, and the same as any other part of the PTR, it wouldn't keep who uploaded it.

So it seems my twitter posts just suddenly and silently started failing... I found what >>14408 posted and imported it, and it seems like I can download from there if I manually look up and change the url to privacydev, but considering I most likely have quite a large backlog of failed twitter posts that I was not aware of, is there a way I could potentially batch-automate this kinda job? To sorta "re-direct" all these failed twitter links to their privacydev equivalent?

>>14464 Select all the lines in the log with failed twitter URLs. Right click --> Copy Sources. Paste the URLs into notepad. Do a find and replace for twitter.com with nitter.privacydev.net Select all and paste it into the hydrus URL downloader. Enjoy.

>>14462 >lolibooru.moe >Make a booru dedicated to loli on a .moe domain >You have to login to even look at anything questionable or explicit What's even the point? ATF's booru requires no account to view anything

Is it "safe" to discuss sankaku scraping here or do their admins lurk and patch discovered exploits? Because there is still a way to get static session-independent post URLs from gallery search page.

ive followed the instructions for the tagger on github but its not mapping them to files, im not sure what ive done wrong, anyone know anything about this?

>>14467 >static session-independent post URLs Do you mean file urls? There's never been an issue getting post urls from searches, they're not tied to sessions.

>>14469 the Sankaku beta scraper still works, you just have to get the security header from the web page code and stick it into Hydrus. It's changed every 48 hours. URLs still only last for an hour as well. Then you have to rescrape them.

>>14470 Really though, compared to site like Pixiv, there's not much on Sankaku.

>>14468 I use an old one that was the predecessor to Garbevoir's version. Use Garbevoir's. It should work well. https://github.com/Garbevoir/wd-e621-hydrus-tagger Just follow all the instructions to make a venv, etc. I've tagged about 2.4 million images with the old one. I set threshold to .10 and it works good. You'll get a few false positive tags, but their usually minor things you can ignore. The tags will show up in My Tags, so you can edit them there, if you wish. I haven't edited any. I'm not about to wade through 2 1/2 million pics! :p

>>14471 lol no. there's way more on sankaku

>>14463 Make your own PTR?

>>14472 Not the poster you were talking to, but am currently using Garbevoir's version for tagging now. I see the option, --threshold, what does this change in the tagging, when set to .10?

I'm constantly getting 403 when trying to download from danbooru, any way to troubleshoot it? I tried changing the UA string to my browser's and I added cookies/login too but it didn't help. I can access the site via browser just fine

>>14474 It's a good thing you don't run the PTR, and hopefully you don't procreate. My theory would have just as much knowledge of your files as the current PTR has, I'd recommend reading the wiki.

>>14470 >Sankaku beta Why use sankaku beta? The regular chan.sankakucomplex.com website works.

>>14476 Threshold sets the amount of "confidence" the AI needs to have in a tag in order for it to actually be sent to Hydrus. Lower means more tags, but less accuracy. Higher means less tags, but more accuracy. I find the default value works fine. There will always be a bit of inaccuracy even at higher threshold values, and you may find the added tags from lower values, inaccurate as they are, help in finding the file anyways. >>14468 If you're using the .bat files, could you paste the contents of the files you're using? without your API key, of course.

>>14463 >and the same as any other part of the PTR, it wouldn't keep who uploaded it And you can verify this, how?

>>14481 The PTR already keeps hashes of every file owned, independent of user information, all this would do is add a little number of people who have sent a 'verify' check, No personal information would need to be stored.

>>14468 >>14472 i fixed my issue, i didnt have the client api service enabled in hydrus. however i cant see to get CUDA working on it but im going to troubleshoot that more today

>>14439 inredastiing, seems like there's a typo in your script somewhere (?) picrel (= vs ==) it claims So instead I tried with modifying setup_venv.sh but it says '$py_command -m venv venv' instead of 'python -m venv venv' at the place I think you meant. Should I instead just replace every instance of 'python' in he script with 'py311'? Have to give up for today sadly. YALL MOTHERFUCKERS READY TO GO BACK TO WORK TOMORROW?

>>14439 >>14484 Well, I thought I had it by installing the package called "python3.11" and then making sure to put "python3.11 at the places in the script, but alas, pircrel struck again and upon trying to run hydrus client I just get the same old error about a whole bunch of modules not being found.

Almost bare-bones parsers of the description and source for derpibooru. I suggest using one of these as an example image for the parser: the files are small, they have descriptions and sources: https://www.derpibooru.org/images/2492858 (it has a blockquote) https://www.derpibooru.org/images/113706 https://www.derpibooru.org/images/1453599 [26, 3, [[2, [30, 7, ["description", 18, [27, 7, [[26, 3, [[2, [62, 3, [0, "div", {"class": "image-description"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]]], [2, [62, 3, [0, "div", {"class": "block__content"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]]]]], 2, "href", [84, 1, [26, 3, [[2, [55, 1, [[[9, ["<div class=\"block__content\">", ""]], [9, ["<p></p>", ""]], [9, ["<div class=\"paragraph\">(.*)</div>", "\\1"]], [9, ["</div>$", ""]]], "<div class=\"block__content\"><p><em>No description provided.</em></p></div>"]]]]]]]], "booru description"]]], [2, [30, 7, ["source", 18, [27, 7, [[26, 3, [[2, [62, 3, [0, "div", {"class": "image_source__link"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]]], [2, [62, 3, [0, "a", {"class": "js-source-link"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]]]]], 0, "href", [84, 1, [26, 3, []]]]], "specified source"]]]]]

>Are you sure? >Add [tag] to the favorites list? How do I remove this dialogue that appears every time I want to add a tag to favorite? Having to confirm every time is super annoying.

>>14476 The main reason I use threshold of .10 is that it will detect loli quite well. I can't stand the stuff, so I use it to erase files with that tag after the scrape. It may get a few false positives on realistic photo's though. Other than that though, .10 does a good job on the major tags. I haven't found any really off tags yet.

>>14489 A lot of loli isn't tagged these days, so if you don't like it, you'll want to go with .10 to detect it. I usually get about 500 loli tags / 10000 files. All of them were missing a loli or lolicon tag.

ok hdev, got an issue that I caused myself a long time ago. I had a small hdd, I needed to make space to buy time for a month till I could afford a bigger drive, and the files I was able to 'delete without issue' was the tag repository files as they have already been parsed. at some point, it wanted to re look at the old ones, saw they werent there, and the tag repository was more or less never updated since. however I got larger drives and I have been tryint to get it working again for a while, I get this ------------------------- v559, win32, frozen Exception An unusual error has occured during repository processing: a content update file (12a87231d24caa8a961cfc14795e2ccab255112be2c04e04ea33d38576860009) was missing. Your repository should be paused, and all update files have been scheduled for a presence check. I recommend you run _database->maintenance->clear/fix orphan file records_ too. Please then permit file maintenance under _database->file maintenance->manage scheduled jobs_ to finish its new work, which should fix this, before unpausing your repository. Traceback (most recent call last): File "hydrus\client\ClientFiles.py", line 1599, in GetFilePath def init( self, controller ): File "hydrus\client\ClientFiles.py", line 1044, in _LookForFilePath hydrus.core.HydrusExceptions.FileMissingException: File for 12a87231d24caa8a961cfc14795e2ccab255112be2c04e04ea33d38576860009 not found! During handling of the above exception, another exception occurred: Traceback (most recent call last): File "hydrus\client\ClientServices.py", line 2195, in _SyncProcessUpdates content_update = HydrusSerialisable.CreateFromNetworkBytes( update_network_bytes ) File "hydrus\client\ClientFiles.py", line 1603, in GetFilePath self._pubbed_message_about_bad_file_record_delete = False hydrus.core.HydrusExceptions.FileMissingException: No file found at path I:\Hydrus\f12\12a87231d24caa8a961cfc14795e2ccab255112be2c04e04ea33d38576860009! During handling of the above exception, another exception occurred: Traceback (most recent call last): File "hydrus\client\ClientServices.py", line 2201, in _SyncProcessUpdates raise Exception( 'An unusual error has occured during repository processing: a content update file ({}) was invalid. Your repository should be paused, and all update files have been scheduled for an integrity check. Please permit file maintenance under _database->file maintenance->manage scheduled jobs_ to finish its new work, which should fix this, before unpausing your repository.'.format( content_hash.hex() ) ) Exception: An unusual error has occured during repository processing: a content update file (12a87231d24caa8a961cfc14795e2ccab255112be2c04e04ea33d38576860009) was missing. Your repository should be paused, and all update files have been scheduled for a presence check. I recommend you run _database->maintenance->clear/fix orphan file records_ too. Please then permit file maintenance under _database->file maintenance->manage scheduled jobs_ to finish its new work, which should fix this, before unpausing your repository. ----------------- so I do the maintenance, I clear the orphans, however network sync never re adjusts itself, it never resets, it never tries to redownload the old files, its essentially a feedback loop, so im wondering if there is something wrong on my end, did I do something wrong in resetting it, or did I do something so stupid that the un fucking yourself method has never been tested and its just not working correctly?

Hey hydev, >>>/hydrus/21085 again, the issue's happening again. Using extremely scientific and rigorous testing (spam opening the same video over and over & spam paging through a list of videos) It seems to be every other video I open doesn't work. Even if it's the same video opened twice in a row (e.g. open vid, blank, close, open vid, plays, close, open vid, blank, etc). Still no idea why it only works sometimes. Nothing shows up in the logs, it isn't dependent on any work being done. It has gone away, at least temporarily, after doing a restart. I have been running into an issue with X11 having too many clients, which restricts you from opening any more until something gets closed, one of my always open programs (not Hydrus) misbehaves after a while and it's troublesome. However the issue persists after fixing the misbehaving program and nothing else has long term aftereffects so IDK. I'll keep an eye on Hydrus as well as I can since I'll be afk for a while. One last thing about the windows, media_viewer was set to fullscreen & maximized, I turned it off because I didn't want that behavior anyways.

>>14477 They did something to prevent scraping. They loosened their rules a bit and I got it working not too long ago. Did they make the block more strict again?

I think I found a bug? When I open an image it starts unmaximized. If I maximize it and then close it the image viewer, the media_viewer setting saves that and opens all other images maximized despite the fact that I have save position & save size disabled. Using Linux & awesomeWM. If this is the expected behavior and it's not a bug could there be an option to explicitly not start the viewer maximized?

>>14501 Oh, I forgot to mention that the start maximized button gets re-enabled when I close the image while it's maximized, which is what I think the bug is. That's with the save position & save size options deselected.

>>14469 >>14479 I thought there are currently two problems with sankaku: 1) hindered subscription scraping by spamming queue with new "page" urls that are duplicates of already downloaded file urls. 2) some kind of rate limiting or mass download detection that makes some "page" urls download a placeholder logo instead of real file unless opened in browser first. Or am I misunderstanding the issues?

>>14503 >1) hindered subscription scraping by spamming queue with new post urls that are duplicates of already downloaded post urls. I solved this by just updating the gallery parser to provide old-style urls. >2) some kind of rate limiting or mass download detection that makes some post urls download a placeholder logo instead of real file unless opened in browser first. This is correct. In my limited testing it seems to mainly happen with files that have larger filesizes, something like 1 MB or more. To be clear: "gallery url" refers to a search page with multiple files, such as https://chan.sankakucomplex.com/en/posts?tags=rating%3Asafe "post url" refers to the page for a single file, such as https://chan.sankakucomplex.com/en/posts/PVaDw30G8Mb "file url" refers to the direct link to an image, such as https://s.sankakucomplex.com/data/bb/11/bb119d2eb2746a335d14bff900ec3a45.png These are the names that hydrus uses.

>>14501 >save position & save size disabled I think that only applies to custom window sizes and positions. Maximize and fullscreen are separate functions from that. Not sure if there's a setting to disable that.

Is it just me or does pixiv no longer require you to be logged in to get full-size images?

Some of e621's images are login-only. Example: https://e621.net/posts/2869855 But it turns out they can be downloaded without an account because you can frankenstein together the image url from stuff in the html. Here's a parser that does that. Note that this just a file page parser, not a gallery search parser - searches will still exclude login-only images. Details: there's a bit of json inside the html that contains the md5 and the extension among other things. This parser adds a subsidiary page parser that builds the url from that json. Also, I recently found out about this website: https://e621.cc/ If you clear the global search blacklist at the bottom, it's capable of displaying login-only images. I may try to make a parser for this.

Also, is it possible to export a subsidiary page parser alone to clipboard/image like you can with a content parser? I don't see a button for it.

If Hydrus crashes while importing files from a folder, it doesn't lose the files, but it does lose the import progress. This causes it to read all the files it got before the crash as already in the database, which makes it difficult to see what actually was already in the database when it throws you a "234 files already in db".

These two files were ignored on import. They're both very large gifs, both explicit. They both are too large for catbox as they specifically restrict gifs to 20MB, but I can keep them up on litterbox for 3 days. https://litter.catbox.moe/ki3xg4.gif https://litter.catbox.moe/qtiqcq.gif

>>14510 Click on the file log button to see the reason a file was ignored. The import options have a maximum gif filesize setting, which defaults to 35 MB, so that's probably what's happening here. You can change the default import options in file > options > importing. You can change the import options for a particular page by clicking on the import options button.

>>14509 >crashes cause minor issues whoda thunk it

I can bet this question was probably answered already on the FAQ but if it was, I couldn't guess the wording to locate it. Basically, I have a NAS attached to my computer (which is running a Windows OS). Currently in my NAS I have a location with a folder where I store all of my pictures. Now, in my head, I have this image of the perfect way that Hydrus will work for me, in that I can install Hydrus on my computer but the database it references is located on the NAS, which is where the pictures are also located. That way I don't have to worry about my computer drives dying or being destroyed as long as the Hydrus database is alive on the NAS along with the pictures, and I am unlikely to move those pictures anywhere so it should stay in place and not get messed up with a parent folder moving or some other form of restructure occurring. That way if I get a new computer, all I need to do is install Hydrus and reference the existing database on the NAS. Is this possible with Hydrus? Hopefully I was able to convey what I want succinctly.

>>14513 I'm not sure about a NAS because I have no experience with those. But it's certainly true that the pictures don't have to be right next to the install. I have my pictures on a separate drive that I plug into my computer which has hydrus installed. This help page has an overview of the database and splitting it up, as well as the launch arguments to use. https://hydrusnetwork.github.io/hydrus/database_migration.html

My internet came back today, and despite the problems, I had a great couple weeks of work. The manage times dialog can now operate on many files at once, and it can even set incrementally increasing times to files to force particular file sorts. There are also some miscellanious bug fixes and quality of life improvements. The release should be as normal tomorrow.

>>14508 I needed this a bunch of times before and I don't think you can. It would be great if you could do that. >>14509 There's an option to display only new images in file import options (where you filter filetypes and filesizes etc.) at the bottom. You can also pick the silent preset.

>>14511 Danke. I didn't notice I could right click -> copy notes. Both Hydrus and Catbox set lower size limits for gifs than other files. Why do some places not like large gifs?

>>14489 Thanks for the update. Do we know what the default threshold is?

Does anyone know why a bandwidth limit that I set for a domains is just being automatically overridden by download pages? I checked and I don't have the "auto override after 5 seconds" enabled, but it's just doing it anyway. I know really even know how to troubleshoot this.

https://www.youtube.com/watch?v=UBYThAP_C5A windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v560a/Hydrus.Network.560a.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v560a/Hydrus.Network.560a.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v560a/Hydrus.Network.560a.-.macOS.-.App.dmg linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v560a/Hydrus.Network.560a.-.Linux.-.Executable.tar.zst I was suddenly without internet for a while, so there was no release last week, but everything is back to normal now, and I had a great couple weeks' work. The 'edit times' dialog can now handle multiple files. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html editing times for multiple files at once If you launch 'edit times' on a single file, everything is the same. If you open it on a larger selection, it will now show, in a summarised way, the range of each particular time type, and how many files in the selection have or do not have a time set there. The controls all work as before, but in general, when you set a time, all files are now locked to that new time. Give it a play and you'll see how it all works. SInce we don't always want to set exactly the same time to a set of files, but rather ideally a similar, staggered time, there is also a new 'cascade step' system in the multiple file version of the dialog. Every time-edit you open up has the ability to enter a little step, say 100ms, which will cause the dialog to set each successive file in the selection to be that much later (or earlier, negative values are allowed!) than the last. This way, if you have a bunch of files of something contiguous, like a comic chapter, that all have different, merged, or otherwise awkward import times, you can now manually sort them in the UI using tags or hackery and apply a new import time based on the first file plus a few milliseconds each time. Thereafter, any time you sort those files by import time, they'll also be nicely in page order time, just as if you had imported them all together from one directory. I am quite happy with the new dialog, and I plan to copy some of the new techniques I figured out to other dialogs for similar better multi-file handling. I'd also really like, as we have discussed before, to tackle a 'cascading' tagger in 'manage tags', so we can quickly set 'page:n' tags to many files at once. misc In v559, it turns out I did make a '54 years ago' mistake, on moving files from one local file domain to another. This is fixed, and on database update, any borked timestamps that were saved should be corrected too. Let me know if you encounter any more issues. Mr Bones and the File History Chart are now under the database menu. There's a new checkbox under 'files and trash', 'Remove files from view when they are moved to another local file domain'. This actually re-introduces the unintended behaviour we had a couple weeks ago, which was happening under 'remove when trashed', but targeted to the correct logical trigger. The gallery downloader cog button now has a 'don't add new dupes' option, which discards the query_text/source pairs you add if they are already in the download list. If you need to paste a hundred queries from a big mishmash list, I think this will help. As a very brief test, I've added 'sort files by pixel hash'. If you have some pixel hashes in your current query, this will make them stand out, but the rest of the sort data is meaningless. I had wanted to try out some similar hacky new tech with 'sort by perceptual hash', but thumbnails don't have quick access to that data yet, so it'll have to wait. I wasn't sure when my internet was going to come back up, so I dumped a bunch of time into some longer cleanup jobs. There's a ton of updated code behind the scenes, particularly around the way files get informed about new metadata, but nothing you need to care about. I have some new plans here and can see the path to getting all sorts of new metadata commands integrated into the shortcut and undo systems. I changed a couple thousand lines, so there may be a typo bug somewhere--let me know if you try to set any unusual content changes and it throws an error! next week Now we have tech that relies on file order, I think it is time we get the ability to conveniently reorder thumbnails. This will also help when I figure out cascading tagging in 'manage tags'. I'd love to have reordering by simple drag and drop, but that may be too complicated to slip into one week. I'll see what I can do. I have quite a few messages to catch up on, which I will try to tackle Saturday.

Hi. I'm making a parser for a website where multiple post pages will refer to the same image link. (Not the same image from different sources, but the exact same image from the exact same URL.) I want to associate data from ALL the multiple posts (e.g. their urls, tags, etc) onto the same image link/download. However, the behavior I currently have is as such: The first post downloads fine with associated post data added to the downloaded image. But, when downloading/parsing a second post, it 'recognises' the image URL as already downloaded, which is fine, but it doesn't associate the data from the second post with the image. Is there anything I can do?

>>14499 I ran the login script test and the HTML looks like the Cloudflare challenge page Another downloader which I used to use is running into same issue, and it identifies the issue as cloudflare too sucks https://danbooru.donmai.us/forum_topics/25845

>>14522 I remember you liked the idea of indicating which tabs in manage tags have tags.

>>14524 I think you have to copy a cookie from your browser into hydrus. I don't know what browser you're using, but in waterfox (firefox) you can inspect element, go to the storage tab and copy paste the cookie values into hydrus under network > data > review session cookies and set the expiration date to something like 3000 days.

Is there any way to set certain tags as default for new file search pages? I want certain tags (for example -meta:wip) to always be excluded on new searches, unless i remove it manually.

>>14529 Those closest I can think of is favoriting a search. It's not that useful if you're only doing it for a single tag though, just possibly very slightly faster.

>>14531 Try adding your session cookies to Hydrus, it's possible that gelbooru is filtering some content to "public" viewers. This has happened to me with downloaders several times where about 10% of content are missing without logging in/setting a "everything" filter.

>>14532 I deleted my post after digging a bit more. It's delayed for 24 hours due to bandwidth limitations.

On certain files in the duplicate filter, I get these error messages: v560, win32, frozen TypeError object of type 'ContentUpdatePackage' has no len() File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 3261, in ProcessApplicationCommand File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2776, in _MediaAreAlternates File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2896, in _ProcessPair File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 3130, in _ShowNextPair File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2981, in _ShowCurrentPair File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 3331, in SetMedia File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 1287, in SetMedia File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2691, in _GetIndexString File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2730, in _GetNumCommittableDeletes File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2730, in <listcomp> v560, win32, frozen TypeError object of type 'ContentUpdatePackage' has no len() File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 858, in paintEvent File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 501, in _DrawBackgroundBitmap File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2666, in _DrawBackgroundDetails File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 1699, in _DrawBackgroundDetails File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 1714, in _DrawIndexAndZoom File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2691, in _GetIndexString File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2730, in _GetNumCommittableDeletes File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2730, in <listcomp> v560, win32, frozen TypeError object of type 'ContentUpdatePackage' has no len() File "hydrus\client\gui\canvas\ClientGUICanvasFrame.py", line 38, in closeEvent File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 3356, in TryToDoPreClose File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2730, in _GetNumCommittableDeletes File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2730, in <listcomp>

>>14535 The errors seem to trigger on duplicates with more than 1 potential pair.

>>14523 From what you're describing it sounds like it should just work but with no example I can't test anything. In the post page url class, checking "post page can produce multiple files" might work?

Getting a serious error in duplicate processing. v560, win32, frozen TypeError object of type 'ContentUpdatePackage' has no len() Traceback (most recent call last): File "hydrus\core\HydrusPubSub.py", line 140, in Process except HydrusExceptions.ShutdownException: File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2232, in CloseFromHover File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2225, in _TryToCloseWindow File "hydrus\client\gui\canvas\ClientGUICanvasFrame.py", line 38, in closeEvent can_close = self._canvas_window.TryToDoPreClose() File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 3356, in TryToDoPreClose def qt_continue( unprocessed_pairs ): File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2730, in _GetNumCommittableDeletes return File "hydrus\client\gui\canvas\ClientGUICanvas.py", line 2730, in <listcomp> return TypeError: object of type 'ContentUpdatePackage' has no len() The progress text for duplicate processing turns into a single hyphen, and the information from the previous comparison will not update to the new comparison. It always happens at the same place, and I've committed my decision on the file just before that place. It will crash after a few moments, or if you try to see the second file in the comparison, and will still crash if you pick an option and move on to the next comparison. I have no trouble loading the file and its potential duplicates in a normal page. I brute forced it by setting the relationships there through right click menus, which cleared the comparison from queue. However, every subsequent comparison seems to have the same issue. I've manually cleared out three comparisons using the view potential duplicates option, but I still have a few hundred to go. Thinking the issue might be with the files from which the old comparison info will not update from, I also tried brute forcing that in the thumbnail viewer. This resulted in the that first comparison getting cleared from the queue. Instead of immediately crashing, the next, now the first, comparison functioned just fine, but the duplicate processor still crashes when I try to go to the next comparison after that. I tried resetting potential duplicates and re-running the dupe filter preparation function, but the issue still persists.

>>14542 Oh hey, someone else is having the same issue. >>14535 >>14536 Did you find a way to get passed it so you can process other duplicates?

>>14535 >>14536 >>14542 >>14543 Hey, sorry for the trouble with the duplicate filter! I screwed up a line in all the rewrites and the error is particularly bad. There's a hotfix here: https://github.com/hydrusnetwork/hydrus/releases/tag/v560a Let me know if you have any more issues!

>>14545 Thanks a ton for the quick fix. Any other program and I'd probably be waiting a month for someone to tell me they're "working on it".

Is there any way to have individual tag services treated similiar to a namespace visually and via search? I'm pulling in some external tags, and I have my own manual tags but it doesn't really work when they're both mixed together indistinguishably.

Hey just catching up with my messages now I have internet back properly. I may have to finish tomorrow, thanks for your patience. >>14435 For this file, I think you'll want to right-click it and say manage->maintenance->regenerate file metadata. If it then comes up as a webm and all works, great; if it doesn't, then you can delete it as just some crazy file, or I'd be interested in seeing it so I can see why hydrus is screwing up. If you can upload it here or just post a source URL, that's great, or send me an email or hit me up on discord and we'll figure out a private way to transfer it otherwise. >>14484 >>14485 Thanks, sorry--I messed that up, exactly the = vs == as you say. Should be fixed in the v560 now. I haven't looked into the custom-python-from-script idea yet, but I'm still hoping to fit it in so we can handle this stuff better in future. >>14440 >>14443 Thanks, I'll slip this in, should be easy. I have a longer term plan to move the taglists over to the customisable shortcut system too. >>14444 Great idea, I also want domain-filtered sort at some point. >>14446 >>14451 Yeah, any time you download a file from a website that parses a 'source time', which is usually something like the booru post time, that gets added as a new modified time for 'safebooru' or whatever. These are aggregated and in UI you only ever see/sort/search the most early of them. >>14454 I had some similar ideas here when I once planned out a 'ratings' repository where we'd be able to share and normalise an average rating for files. In general, although I think this stuff is broadly a good idea, I just don't have the coding time to build or maintain anything but the basics. Any time I think about diving into the hydrus network protocol again for an update like new file metadata or something, I shudder at how much of a pain it will be and how many new problems will spin out, so even if an idea is great, I often just can't entertain it. Ultimately, in future, I would prefer to move away from large scale sharing tech like the PTR. It has been a fantastically successful experiment, but it isn't sustainable, and I want to grasp this giant corpus we have worked on and hone it down to more useful systems. (e.g. training AI auto-taggers on it). Many tags like nouns are becoming easier and easier to recognise quickly, and the situation will presumably be bonkers in five years, so rather than build a very efficient 'list of samus aran files' repository, I think I'd like to move to a better 'samus aran manager', which involves better processing workflows and tech like deduplication. That said, in terms of sharing data, I want to move to friend-based client-to-client communications, through the Client API. Getting a few thousand Anons to agree on things in a central repo is tricky in a number of ways, but when you share your Goku fan pics with your friend, he knows what he is getting into.

>>14456 Oh yeah to add to this specific idea, I have had this same rough idea for purging old hashes. It is tricky, but I think there's a way of doing it. I wouldn't do it in an ongoing process, but I'd have the admin trigger a 'recycle reset' and users who wanted to help out would let the server know what was still a good file, and then after 180 days or something, any hashes that were not pinged by anyone are now delisted. EDIT: Sorry, I realise now this is what you were actually talking about, I didn't read it properly and thought this was more about multiple clients verifying tags through multiple 'voted' upload. >>14460 Thanks, will try! >>14477 Ah, this is a shame. I noticed it was seemingly working again a week or two ago. Even if things loosen up here and there, I guess the danbooru admins are moving in the direction of tightening up access to automatic downloaders, so things are probably going to be rocky for us going forward. I don't personally keep up to date with any booru politics, so if anything important happens, please keep me updated. Since this is Cloudflare tech, and it seems to be the CF tech that hits some people and not others (it is a few days later, but works fine here), my general experience with it is it is region/ISP based, so moving your VPN to a different location can be helpful. Otherwise, you basically just have to wait it out--the rules appear to be applied dynamically by some automated system, and they are undone similarly. >>14487 Thanks, I'll roll these in to the defaults! >>14488 Thanks, someone else mentioned this too, I will remove it. >>14494 I am confident this is a fixable solution, so no big worries. Although it sounds dumb, I know that big errors like that apply some rare pause states. I was fixing a guy earlier on today with the same issue. Check services->pause menu to see if all your repo work is paused. Otherwise, check the multiple pause/play buttons in the review services PTR page--any of them paused? If it isn't as simple as the thing being paused, what is the service's status text in review services? Normally it says 'service is functional' in black text, but if there is an error (e.g. 'all services are paused' or 'Download cancelled: blah blah connection error'), it'll says that in red text. What does it say there? Once you have things set up you think to how they should be, hitting 'refresh account' is the best way to 'jog' it back into action.

>>14495 Interesting. I think if you ctrl+f the last thread or two, you'll see me talking about how I secretly have two always-on mpv windows in most clients, and switch back and forth as needed, and so when you see 'one video works, the other doesn't, over and over', like you are seeing, it tends to be because mpv window A is broken but B is fine (and thus a restart fixes it). I wonder if this is what is happening. These issues can often be aggravated by GPU drivers sperging out through a system hibernation or similar, failing to restore one or the other mpv windows after wake. If it helps you play around with this, you can sometimes force loading a particular mpv window by opening multiple media viewers at once or going back to the main gui after the media viewer is open and clicking on something to load it in the preview viewer. Imagine that hydrus simply has a pool of mpv players, and any time it needs a new one, it grabs from the pool (or if none, creates one), and every time it moves on from a video, it gives it back to the pool (after a very brief delay). Most of the time, this means two mpv players, but you can force three or more if you try. Although it sucks, if you notice this happening in your X11 after system sleep or just long use or whatever, the best solution may be just to restart your client every day. I am planning better and more safe mpv integration in future, but it'll be a long way off, so I can't promise much from my end to fix the current system. I basically just call a library to implant the mpv window in my UI and pray that it works. Note that the reason I recycle mpv players like this is if I ever try to destroy on mpv window, most of the time I get a crash within a second, wew. I'm interested though, when you have an mpv player that is presumably borderless fullscreen or otherwise sperging out, can you alt-tab your way somehow to seeing the media viewer 'beneath' the hydrus media viewer below it? Maybe at a different size, or not borderless? I know that when mpv goes wrong on Linux, it is often de-embedded from my actual UI often, and is just a floating window somewhere. Anything like that?

>>14522 >>14550 >Thanks, I'll roll these in to the defaults! If you meant to include the parsers, then the HTML paragraph thing does not seem to be working.

>>14551 >Although it sucks, if you notice this happening in your X11 after system sleep or just long use or whatever, the best solution may be just to restart your client every day. NTA, I have two displays and if I turn a display on and run xrandr --output ... --primary --auto --above ... and move the window to the new secondary display, menus do not appear. The display works without that, so maybe I should only run xinput and not xrandr.

Version 558 needs a 'special update' note stating you need to update to 558 before updating to 559 or later. If you don't, earlier database migrations can fail due to the timestamp_ms column not yet existing. This happens at least for the database update to 548 when running 559 or later, but it's possible it happens for more versions between 548 and 558 too. See GitHub issue 1512 for someone else who also encountered this error (and included a full traceback).

Can Hydrus not snag ugoira animations from Pixiv? I dropped a url in and it got ignored with the note, >veto: ugoira veto

>>14501 >>14502 >>14505 Thank you for this report! I played around and see you are just talking about window size (stuff in options->gui), not the actual image inside the window (stuff in options->media). Yes, it looks like I am saving maximised status even when remember size/position are off! I will fix this. >>14507 Thanks, I will play with this and see if I can roll it nicely into the defaults! >>14508 >>14516 It is strange you cannot. I bet it is because the separation formula and parser proper are separate objects and thus I don't have a nice single thing to serialise to the clipboard (my import/export buttons are programmatic, not hardcoded every time, and often need things coming in and going out in a certain way). I thought, though, that I had hacked some code to make it easier to do this sort of thing, so I'll have a look. >>14509 Yeah, I'm sorry, I can't guarantee nice behaviour here, after a crash. The import folder only saves its progress when the whole run is done, so if the program shuts down non-cleanly, you'll get this. Rather than rewriting the import folder to save more atomically, making things nicer after a crash, can you talk more about your crashes? I'd much rather try and relieve that so you aren't having to worry at all. What's your platform, and does any area of the program or action tend to aggravate the crashing, as far as you can tell? If you are on Linux and sync with the PTR, do you notice the client suddenly eating 16+GB of memory after some maintenance operations, and is that correlated with crashes at all? Unfortunately, crashes--where the program suddenly halts and exits, with no clean shut-down--do not write anything to the log, and due to the language I write hydrus in, it is tricky to get good information out of OS crash dumps, but if you check your log (in your install_dir/db folder), there may be a bunch of unusual error messages in the lead-up to certain crashes, particularly Qt 'paint' stuff. Let me know if you see anything in there! >>14510 >>14518 >Why do some places not like large gifs? This doesn't happen so much any more, but when webms were still new and not always supported by either codec support or CPU power, some geniuses would convert 10 second 720p overwatch sfm loops and the like to gif (often through dumb automatic converters) so ye olde phoneposters could still see them. What was a fairly neat 30fps 7.9MB webm would become a 12fps 93MB gif with a garbage colour palette. They got re-uploaded to boorus and it was a spammy hassle. It is a bit like when an artist puts out ten variants of a picture of an anime girl on their patreon rewards in 16,000x22,000 png format--just a wasteful format put out by someone trying to do something helpful but who just doesn't know enough about the levers they are pulling. Some months ago, I turned off that that 'max gif filesize' default for new users, since this isn't such a big deal any more. >>14513 Just as a side thing, while hydrus can absolutely store its files and stuff on a NAS, and several users do this and are happy, if you did not know: hydrus does not manage your files and folders as they currently are. It sucks them up and copies them into its own technical non-human-friendly storage system. If you haven't tried hydrus out, I recommend you just spin up a copy on your desktop or something and drag and drop a thousand files onto it. By default, it won't delete the files from their original locations, so this is 100% safe and non-destructive to do. Have a play around with the program, go through the getting started guide, and see if the program is for you. If you absolutely need a human-friendly copy of your files, with regular files and folders, some hydrus users figure out some duplicated solutions for certain precious files where they need it, but in general hydrus is not for that sort of thing. >That way if I get a new computer, all I need to do is install Hydrus and reference the existing database on the NAS. Yep no worries, hydrus is completely portable and as long as you can still see your files through some normal directory, it is simple to relocate to another computer. Let me know how you get on! More reading on the specific filename question, if you weren't sure on this and want to know why: https://hydrusnetwork.github.io/hydrus/faq.html#filenames https://hydrusnetwork.github.io/hydrus/faq.html#external_files

>>14521 Is it overriding on the 'gallery download' step, where it walks through all the gallery pages of a search to gather Post URLs? I think I have a ~30 second override on this, forced, because gallery contents will shift by a file or two whenever a user uploads a new file that matches that search. If I were to allow a four-hour break on a search like that, and some user uploaded 150 files in the meantime, we'd lose our position on resume in a complicated way. I don't think this time is editable, but if you like you can force a mandatory wait between gallery fetches under options->downloading. Maybe adding 90 seconds would fit your rhythm better? Let me know if this isn't--I can expose these values for editing, although I'd be wary of allowing anyone to turn it off completely. If you really are getting bandwidth override on file downloads and don't have that hacky 5-second override set, let me know, since in that case something is definitely screwing up. >>14523 >>14539 Yeah, odd situation. It is strange that in your 'already in db' situation it isn't associating the metadata. Normally, hydrus will assign all parsed data on an 'already in db' result, BUT as soon as 'already in db' is set, it won't do any more network requests, so if you have a complicated download 'chain' here that relies on multiple URLs being downloaded, maybe that's what's happening here? Setting the URL class to 'multiple files per URL' might get the behaviour you want here, even if it sounds a little wrong. That thing basically says 'get all metadata, keep downloading html and always do download html every time you see it even if you think you've seen it before, just don't download files again'. If you right-click on a file log entry, you should get pic related. In your second attempts, which end up 'already in db', do you have any parsed tags, and if it has a hash, is it the correct hash? If the hash is good and there's content to add, it should all fire correctly on 'already in db', so let me know if it isn't! I'm currently investigating a complicated subsidiary page parser situation where it apparently doesn't, also, hope to have it cleared this week, so check the changelog on next version too! >>14525 Thanks! >>14529 >>14530 Yeah, I'd recommend setting a favourite search with a template. Have that negated tag and set it as 'search paused' (i.e. click on 'searching immediately' in your template to save), and then load that or other templates up. I think Ctrl+I flips 'searching immediately', if you can get used to it. Then again, if WIP specifically is so important to you, you might like to also play with multiple local file services. If you made a new file service called 'complete' or something and then mass-copied the non-WIP files to it, then you could just search that file service when you wanted it, or you could have other variants like expelling all the WIPs to their own box, whatever worked best for your situation. Multiple local file services are often used to make strict sfw/nsfw barriers, and this isn't completely different.

>>14547 Yeah, I'd like this. I want some of my personal unnamespaced tags to stand out in gold lettering or something. I could just namespace them all, but bleh, and I might like just some normal kino tags on any service to stand out in bold or something anyway. I want more tag display rules. I don't honestly have excellent tag-service-separation tech yet. I was talking with someone about this the other day--when I did multiple local file services, we got a load of individual-file-service tech on the database end and custom unions and so on when doing searches, but tags are still stuck on either one particular service or 'all known tags', which is a clever optimised hardcoded union of everything. It'd be nice to have some more settings here, although there would be some fun stuff to deal with when two services had the same tag but different display rules. So I know better where you are coming from, what sort of display are you thinking? Showing the text a different way, grouping different services' tags separately, or something else? >>14553 Yep thanks, I rolled in some new tags, from here >>14386 , in v560, and I will try and roll in that description note parsing stuff in for next Wednesday's release, v561. >>14556 Thanks, I have had a couple of reports. I hate it when this happens, and there isn't a clever way around the bitrot. I'll have another look to check, but I think the critical points are v448 (can technically update to v559, but misses some downloader updates), and v551 (can safely update to v559), so I'll write some version checks and tell the user if they need to do another step, and to what version. >>14557 We are in slight limbo with Pixiv Ugoiras specifically. The thing is that Pixiv offers the frame timings of their Ugoiras in javascript or JSON or something separate to the images/zip, and if we are going to have proper native rendering, it would be nice to record those frame timings when we finally set the Pixiv Ugoira floodgates open. Another user has been playing with some extra download tech to figure this out, but I am not sure what the status is. I'll have to do some database tech at the same time so we can save it, too, and we'll probably break the Ugoira standard and save the frame timings in the zip as a .json, too, since it is so weird to have them so virtually pseudo-sidecar'd. Anyway: we want it, and we want Ugoiras to play properly, but we aren't there yet.

>>14550 ok I made sure everything is unpaused just in case there was a weird pause I missed, got the same error. the problem at least as far as I can tell is that network sync never resets, I can do everything it asks me for, the program knows those files are gone, or at least should know, but for some reason it never registers that they are gone or it never tells network sync they are gone. I have tried reset downloading hoping that would make it forget that anything was downloading, but that didn't work either. so I have no clue what's not doing whatever its suppose to do correctly.

>>14560 For my tag usecase I've manually tagged a bunch of files with wide genre tags alongside a score tag. With other tag repositories and ai tagging looking like it can streamline the wide genre tagging + more specific tagging I'd like there to be a distinction visually between them. Maybe it's literally having the option to have a tag repository effectively be a namespace, with or without a prefix but able to have it's colour/boldness etc changed. Ideally with the ability to have an images tag list sorted first by tag repository, then alphabetically within that repository so my tags aren't mixed in with the 1000 extra tags added by other repositories. This would also apply to search suggestions if possible, it's a lot easier to search through my limited tag set first than having it mixed in with the thousands added by external repos. For the problems you'll run into yeah there's a lot of decisions to be made. If a person elects to have individual repositories act as their own namespaces that would at least fix the multiple services using the same tag issue since the tag would simply appear multiple times in the list, one per service (throwing shit at the wall but maybe you can set a service priority where the highest priority service with the tag is the one that displays it, with an asterisk next to the tag to indicate other services also have that tag).

>>14559 >If you really are getting bandwidth override on file downloads and don't have that hacky 5-second override set, let me know, since in that case something is definitely screwing up. Yeah. It's saying "overriding bandwidth after 5 seconds" but the override checkmark is unchecked. It's happening for the file downloads, not the searching step.

I had a mixed week. I added thumbnail rearrangement, and I cleared some bugs and quality of life stuff, but I didn't get as much as I wanted done. The release should be as normal tomorrow.

>>14566 >I added thumbnail rearrangement noice

I'm a casual user getting frustrated with all the fiddling with 403's and cookie/header stuff. What's the best couple of sites with lots of content and little to no upkeep to scrape from? I was using lolibooru.moe & sankaku, then switched over to rule34.paheal and danbooru+gelbooru. I looked into hydownloader, but that looks like a high maintenance program to use, so it doesn't seem feasible for really reducing time spent fiddling with stuff.

>>14568 I mostly grab from pixiv.net, tbib.org, and e621.net. Only issue I ever have is the occasional (like once a month) sending cookies from e621 to hydrus to stay logged in, but that's literally 2 clicks two fix with hydrus companion. Cannot recommend hydrus companion enough, it makes so many things much simpler. >lolibooru.moe pixiv.net would be the place to go for that sort of thing. You'll need an account and to manually set your location to Japan in your profile, then send cookies to Hydrus. Again, super easy to send cookies with hydrus companion. The only issue with pixiv is that they made their own animation format called "ugoira" that hydrus can't handle yet. It'll probably happen one day, but until then I've been using the Pixiv Toolkit browser extension to convert them to webm in the mean time. Extension is at: https://github.com/leoding86/webextension-pixiv-toolkit

>>14569 >convert them to webm That's a lossy conversion I believe. If you want something lossless, I think it also lets you convert to APNG which is close to what ugoira actually is.

>>14570 It can do GIF, APNG, or WEBM. I believe you are correct that WEMB is lossy but I've never been able to discern a difference to my eyes with it set the best quality though. Gotta manually download, convert, import and tag everything anyways, may as well pick the space saving option while I'm at it since I can't visually see any difference between the APNG and WEBM anyway. Though if Hydrus could automagically do all that for me, I wouldn't care what format or how much space it used. Convenience would trump the space saving for sure. Maybe one day.

https://www.youtube.com/watch?v=GuVnZZ1sFIc windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.macOS.-.App.dmg linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v561/Hydrus.Network.561.-.Linux.-.Executable.tar.zst I had a mixed week. Thumbnail rearranging is added, and some bugs and quality of life issues are cleared. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html thumbnail rearranging So, if you right-click any thumbnail, you'll now see a 'move' menu. This makes it simple to rearrange your thumbnails. You can move your current selection to the start, the end, left one, right one, or 'to here' if you have multiple selected. If your multiple selection is non-contiguous, it will be made so on move, with the move focusing around the position of the first selected item. You can also map these commands to keyboard shortcuts under the 'thumbnails' shortcuts set. I haven't added any default shortcuts for this yet, but let me know if and what you would prefer--I've been playing around with ctrl+numpad numbers on my dev machine, and it feels nice to me. In future I'll try and figure out mouse drag-and-drop rearranging. It might have to wait for a larger pending rewrite of the thumbnail grid though--we'll see. other highlights Unfortunately, there were a couple of stupid typos in the content processing changes last week. One in the duplicate filter (which the v560a hotfix addressed), and another fixed today that was causing 'already in db' results to not get metadata updates correctly. Sorry for the trouble, and thank you to the users who reported these. Ctrl+C/Ctrl+Insert is now hardcoded to copy tags from the taglist. The thumbnail and media viewer menus should now be much thinner. I hate how wide they can get, how annoying it is to hit their many nested submenus when they get like that, so let me know if they still go crazy in some situations. There was a bitrot issue in v559, the millisecond timestamps conversion, that made it impossible/bad to update from a much older version. This has always been a tricky technical issue to talk about and communicate to the user, so I've now written a better in-client error reporting process that stops the user before the update is even attempted. The upshot this week is that if your client is v551 or older and you try to update to v561 or later, you will be told to update to v558 first. In lieu of a proper rewrite, I've made some semi-hacky newline processing improvements to the parsing system. If you make downloaders and hate having to deal with extra whitespace in multiline content, notes or otherwise, check out the full changelog and let me know how you get on with it all. next week I want to work on github bug reports, which I haven't put proper time into for too long!

>>14572 >Ctrl+C/Ctrl+Insert is now hardcoded to copy tags from the taglist. Noice.

>>14462 just saw this post. seconding >>14466 they both have different stuff

>>14572 The Ctrl+C and thinner context menus are some of the coolest changes to me in a long while, great! Also just curious, what's the use case for thumbnail rearrangement?

"The 403 error has recently returned to me. And I found a new solution on the Danbooru forum. Instead of your user-agent you have to write your nickname (Danbooru) in hydrus. It worked for me, maybe someone can use it too." This is in regards to fix 403 errors. Any clue wtf he means by putting your nickname in hydrus?

>>14575 >what's the use case for thumbnail rearrangement? When you imported a set of images and you need to order them, manually putting them in order before actually labeling them is probably much easier. I know I often want to just see a small set in order that's out of order, but changing the sort order to make it work is often a bit cumbersome.

>>14575 >>14578 Oh, I didn't read the update right. This is moving thumbs either forward 1, backward 1, to the end, or to the front, via right click menus. I thought is was the more intuitive drag and drop rearrange being implemented. In this case, I can understand you seeing little use cases for this. Really, if I need to temporarily see something grouped together in order, I can just drag and drop each file in order to a new page and it would be faster and more intuitive. This is really only useful if you only need to move one or two files, and those files just happen to be one or two spaces away from where you want them or need to be moved to the front or back of the list.

Can I ask if there are any plans to hook subscriptions and import folders up to the API? Designing the right API for subscriptions might be difficult but it's mostly import folders I care about. These days I am running Hydrus headless like this: env QT_QPA_PLATFORM=offscreen hydrus-client --boot_debug I mostly interact with https://hydrus.app and rarely touch the GUI except for subscriptions and import folders. Subscriptions I don't mind using the GUI for but sometimes I want to reload all watched folders (for example gallery-dl has just finished downloading a bunch of files, I'd like to tell Hydrus about this rather than wait for the timer to trigger). It'd be nice to have an API call for that or maybe attach a signal handler (e.g pkill -SIGUSR1 or -SIGUSR2 could trigger a reload). Another option could be hooking up inotify into Hydrus instead of polling but that introduces more complexity compared to a simple curl call to the API at the end of my script.

Exporting a file fails with "Filename too long", so {tags} is not usable in the template.

>>14577 You can change headers such as User-Agent under network > data > manage http headers.

>>14579 To be honest I was thinking the idea was the rearrangement was saved, which would be super handy to me. I understand why that's not the case (i'm sure this would be difficult to produce given the variable things one can sort by) but as it is I'm not thinking I'll get much use out of it, I rearrange images in chronological order primarily and would rather just edit the times so they are arranged as such permanently. Still neat though.

artstation gallery searches always 403 now, even when given cookies. What do?

>>14561 Sorry for the continued trouble. I can verify that file is a legit PTR update file that you should have, so please try this: - make sure _help->advanced mode_ is on - open a new search page, change 'my files' to 'repository updates' - paste in "system:hash = 12a87231d24caa8a961cfc14795e2ccab255112be2c04e04ea33d38576860009" - if it does not appear, change the file domain from 'repository updates' to 'all local files'. Then, try 'all deleted files'. Does the update file appear? Presumably it won't, since you don't have it, but let's see. Then let's fix another common error here, hit database->regenerate->local hash cache. Then, to test another issue, let's hit up database->file maintenance->manage scheduled jobs->add new work tab and click the 'all repository update files', and add the top 'if file is missing, then if has URL try to redownload, else remove record' job. Go to the 'scheduled work' tab and force it all to clear again. Then hit up database->db maintenance->clear/fix orphan file records. Then try unpausing the repo again. Let's see if we get the same error or what. >>14562 Thanks, these are good ideas. I already have some options structure here, and some service precedence, in the options in tags->tag display and search and the sibling/parent stuff. I wonder if I should move the namespace colour options there? The current options would be 'all known tags' as normal, but then each service could override it. I'll have a think. >>14565 Damn, thank you. I will look at this. I guess restarting the client doesn't fix it? And is it for one page or all download pages, new and old? And stupid thought, but does turning that 'cog' button menu option ON actually turn the behaviour OFF? I wonder if the initialisation is flipped somehow. >>14568 The big three boorus--danbooru, gelbooru, and safebooru--are good for run of the mill animeshit. Much of western-friendly pixiv is reposted there. Rule34.xxx hosts a lot of western content and SFM stuff. e621 has beast/furry stuff if that's what you like. All are easily accessible to a guest user (although danbooru got stricter recently). If you want more spicy content, there's very few places that'll host it without a login these days. Gelbooru allows spicy stuff just with a one-time cookie, I think. I haven't done it myself, but I think it should be simple: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Downloaders/Gelbooru

>>14570 >>14571 We hope to have native playing of Ugoiras this year, just need to figure out some frame timing parsing stuff into the download pipeline and database storage, and then I'd be interested in figuring out the first in-client file format conversion. APNG and Webm are good candidates for different technical reasons. >>14575 >>14578 >>14579 >>14584 >Also just curious, what's the use case for thumbnail rearrangement? I recently updated the 'manage times' dialog to handle multiple files at once, and one feature is it can apply a 'cascading' time to a set of files, so file 1 in the selection gets 'blahblah and 00 seconds', the next 'blahblah and 01 seconds', then 'blahblah and 02 seconds', and so on. I have also, for ten years or more, wanted to add the same thing to the manage tags dialog, so you can spam 'page 17 -> page 54' to a bunch of files in one operation. This new tech will benefit from better ways of reorganising files before we set up the job. I'd like to figure reorganisation via mouse drag and drop too. The thumbgrid is all my garbage custom code from a million years ago and coordinateshit can sometimes be a pain. We'll see! >>14581 Yes, absolutely! I need to do some backend stuff to make subscriptions more instantly accessible (think about the delay before manage subscriptions opens), and some more 'this is what this sub is and what it has done and does do' tech, but yeah, lots of people want access so I won't forget this. Import folders is less desired, but as I add subs, I think the access calls will be similar, so I think it should be doable. If I forget, please remind me! >>14582 Thank you, can you give more of an example, so I can reproduce it on my end? I thought I had pretty good filename 'eliding' tech now, where it should clip the filename to something that'll fit for your file system, no matter what. Do you actually get the 'filename too long' error? Can you post it? What list of tags cause it? >>14585 Yeah I don't think it is reasonably possible any more. I think it is a strict CloudFlare block that we just won't pass. Maybe in the future when I move to HTTP 2.0. Maybe gallery-dl/hydownloader can handle it?

>>14587 >I recently updated the 'manage times' dialog to handle multiple files at once, and one feature is it can apply a 'cascading' time to a set of files, so file 1 in the selection gets 'blahblah and 00 seconds', the next 'blahblah and 01 seconds', then 'blahblah and 02 seconds', and so on. Ah, that would be very useful.

>>14572 >thumbnail rearranging YAY!

>>14587 >>>14582 https://twibooru.org/2951748 {hash} {tags} A sidecar file didn't get saved. Most of the tags are in two services, but there is one that stars with "downloadreason:sub:mares definitely" in a separate one, that got cut off. Traceback (most recent call last): File "/mnt/hydrus-561/hydrus/client/gui/exporting/ClientGUIExport.py", line 851, in do_it metadata_router.Work( media.GetMediaResult(), dest_path ) File "/mnt/hydrus-561/hydrus/client/metadata/ClientMetadataMigration.py", line 193, in Work self._exporter.Export( file_path, rows ) File "/mnt/hydrus-561/hydrus/client/metadata/ClientMetadataMigrationExporters.py", line 657, in Export with open( path, 'w', encoding = 'utf-8' ) as f: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ OSError: [Errno 36] File name too long: '/tmp/3a94a9abc717a496ffe22eb1d7e4741ce184591bcdf76105428fdf49408e89b3 best friends until the end of time, applejack, fluttershy, pinkie pie, rainbow dash, rarity, spike, twilight sparkle, twilight sparkle (alicorn), derpibooru import, sub:mares .png.all tags.txt'

>>14583 Oh sorry, I understood that part. I meant I was confused by what he meant by "write your nickname (Danbooru) in hydrus." What nickname?

Copy pasting timestamps in the manage times menu seems to be broken. I simply click on the copy button and then the paste button and get this error.

Hi @Hydev, i got some questions. Hope you can answer them. im still testing with a very small number of files to get the grips, so maybe the answer wouldn't be clear to me for some questions. 1. im up to date with the PTR. i exported the repository update files which are around 11GB, just to have them backed up in case. is it correct that people can use those files to update their PTR tags to the latest state, means all the mappings/parents/siblings are contained in those 11GB? if that is the case then: a) why was it that those files were not offered compared to a whole client with 30GB+ that was linked in your documentation? is it because they have to be processed first? b) there is the import repository update files option. can i import those in an offline client or do those files need an online connection with public account to work at all? didn't try yet. if you don't need an online connection, i guess it would be feasable to just copy the new update files to the offline client and import them, instead of creating huge Hydrus Tag Archive files that take hours to be created for all PTR mappings. i tested to create a HTA with all the mappings from the PTR to see how big it is. after it grew to 30GB after 1-2 hours, i stopped. i assume it would grow to 60GB like the client.mappings.db. c) can i delete the old processed update files from the "repository updates" domain after i am up to date? or are they necessary to stay in the client for the PTR to work? also if i can delete them, i assume i should leave the latest in there in case i haven't processed them yet, correct? 2. i could not find out what the "tags" sorting button (see screenshot No.1) is for. maybe i have not enough own tagged files to see a difference. can you give me examples when it is used? also i only can see which tag domain is selected when moving with the mouse on the button to show the tooltip, because i assume there should be a checkmark icon like for the other buttons but it is not showing here. bug? 3. im not sure if the tags autocomplete suggestions works as intended in the manage tags dialog or when adding favorite tags in "options -> tags dialog" for example. maybe other places too. i mean specifically the file domain button does not change the behaviour and seems to not be necessary at all, because it is not considered. let's say i have only one file in hydrus and it is in the "my files" domain and i tagged it myself with the tag "test1", then changing the tag service to "my tags" and the file domain to "trash" shouldn't show that suggestion because "trash" is empty, but the suggestion does show. let's say that file would have been also tagged by PTR with "cool1" and i change the tag service to "PTR" and the file domain to "my files". instead of showing me only "cool1", the autocomplete shows every PTR tag. no matter which file domain you chose, it either shows all "my tags" or all "PTR" tags, if you understand what i mean. in summary: the option to change the file domain seems useless? is it intended? (see screenshot No.2 -> i have only 13 files in hydrus "my files" and no zelda content at all but i get the suggestions anyway). it looks like the file domain is set to "all known files with tags" all the time, no matter what file domain you chose. if it would work like i thought it would, you could create a "SFW" file domain, select it and then you would only get SFW tag suggestions. Of course only if you put files in there. for the normal search pane it works just as i imagined though! the file domains there are considered. 4. right-click into the "selection tag" window after a search shows the "experimental" option with three different display options. can you explain where they apply, how they differ and if there are settings to change the behaviour. in general: where does it make sense to use those at all? i couldn't find out what they do. it is not the same as the "manage tag display and search" setting afaik, i kinda understand how this one works after playing with it a bit. but i don't get the "experimental" one. 5. feature request: can we have the option to toggle the visibility of the local file domains in the popup that are shown when you left-click on the local file domain button in the search pane (see screenshot No.3)? im in advanced mode so there are some that i find distracting. maybe when you click on the file domain "multiple/deleted locations" and the new window opens, you could add an eye icon next to the file domains to toggle the visibility in the popup as you like. for example, the "all deleted files" location is annoying me right now because it shows files that were put into a new test file domain and then deleted again, even tho they are still in "my files". not useful for me. i know it is intended for this one, as the tooltip for it says. but i rather switch that file domain visibility to a "deleted from all local files/all my files/my files" domain. would be nice if we could customize to show whatever file domain we want in the popup, maybe even rearrange the order. thats all for now. sorry for the wall of text. i appreciate it alot if u try to answer them. you are a TRUE LEGEND!

I had a tepid week. I fixed some bugs and improved some quality of life, that's all! The release should be as normal tomorrow. >>14591 Thanks, should be fixed tomorrow, let me know how it goes! >>14593 Same, thanks, should be fixed tomorrow!

>>14569 do you use the pixiv parser from the cuddlebear repo?

Does anyone know what the "hatate:not found" tag is? I got banned for it but I've never seen it get parsed from anything.

>>14599 It's from this: https://github.com/nostrenz/hatate-iqdb-tagger

>>14600 Ah, I see, yeah that'd explain why I've never seen it before, I must've gotten caught in the crossfire due to duplicate processing.

>>14601 oh hey, welcome to the duplicate processing ban club! XD hatate:not found has never gotten me, but sankaku urls as tags has plenty of times. >>14598 Yeah, I just use that one and send pages to hydrus via hydrus companion.

https://www.youtube.com/watch?v=XnwHtzBf-c8&t=17048s (go to 4:44:00 for a wild fight!) windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v562/Hydrus.Network.562.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v562/Hydrus.Network.562.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v562/Hydrus.Network.562.-.macOS.-.App.dmg linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v562/Hydrus.Network.562.-.Linux.-.Executable.tar.zst I had a tepid week, but there's some decent fixes and quality of life improvements. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html all misc this week I fixed a stupid typo error in the manage times dialog when you go into a single time and try to copy/paste the timestamp. The buttons also add millisecond data now. Fingers crossed, drag and drops of thumbnails and page tabs will feel snappier this week. You might see some heavy 'analyze' database maintenance coming down the pipe. Let me know if it proves annoying, or if you even notice it at all--my hope is to iron out some super slow PTR-based tag updates that hit some users, but we'll see how it goes. If you are an AUR user or otherwise updated to a very new Qt version (6.6.1) and suddenly the column widths of multi-column lists all went ~100px wide, I think I've fixed it! If you were affected by this, I can't recover your old settings, but recall that you can right-click any list header to reset the widths to the default. next week I was short on work time this week, so I'll try to hack away at github bug reports again.

>>14603 >https://www.youtube.com/watch?v=XnwHtzBf-c8&t=17048s <direct linking Dev-anon, you should know better.

>>14602 weird, i'm getting this every time i try to add the pixiv tag parser from github despite having deleted all pixiv stuff in my hydrus client and restarting it "All 1 downloader objects in that package appeared to already be in the client, so nothing need be added."

>>14605 i'm referring to the "pixiv tag search api parser 2020-10-16.png" on cuddlebear btw

Forgive me if this is already implemented and I haven't noticed it, but could we get a "favorite gallery downloader setting" or something? I have a few gallery downloaders that I mainly use and many that I use only occasionally and it'd be nice to have the high traffic ones at the top or highlighted in a different color or something.

>>14606 Did some double checking and found this, which i believe I got somewhere in one of these threads about a year ago. Maybe this will work for you?

>>14606 Isn't that one out of date? Try clicking "add defaults" at the bottom of network > downloader components > manage parsers.

I want to host a public booru on my server, anyone got some advice on whether Danbooru or Shimmie2 is better? Sorry for off-topic, but this is the best place I could ask.

>>14610 >Shimmie2 Huh, this is new to me. Are there any examples of boorus that use this?

>>14609 when was the last time the default hydrus pixiv parser was updated?

>>14613 >>14608 ok I successfully imported this and it looks like it's just the parser without a gallery downloader, so this is quickly getting into DIY territory. is there one with a premade gallery downloader and stuff to save me the trouble? also since i'm new to pixiv, will i be able to scrape all the content using this file page parser as compared to the pixiv search API parser that comes with hydrus by default?

>>14614 NGL i know nothing about how downloaders work lol. I have no issues with downloading galleries or pages. Almost everything I get is automatic thru subscriptions to artists and they all work as expected. Sending individual pages works fine too. The only issue I have is that nothing works with the bookmarks pages. The following is like 2 minutes of work a week tops. I only use this for this for thing I bookmark on mobile. Anything on the desktop is just click hydrus companion --> send this tab to hydrus. Works on artist gallery pages or individual posts. That covers like 95% of my pixiv stuff. Below catches the random mobile stuff. Bookmarked items doesn't work for anything but the first page. For bookmarks I use the Link Gopher extension (https://sites.google.com/site/linkgopher/) and have it extract all links on the bookmarks page with the term "art". This gets all the URLs for each item on the page. Paste that into the URL downloader, Then go back to pixiv and move to the next page of bookmarks. Rinse repeat. Takes like 5 seconds to grab an entire page of 50 bookmarks like this, though it's a bit manual. If you don't use pixiv's bookmarks (the little heart icon) then you don't have to deal with that at all. Aside from the bookmarks page not working right, hydrus interacts with pixiv exactly the same as it does for any other site for me. Is there any kind of way to see what versions of a downloader you have? Possible I've got something else from a random post here that's not on cuddlebear. Or a way to just export everything pixiv related so I can post it here? >>14613 assuming the stuff on cuddlebear is "official", then nothing has been updated for pixiv for 4 years. Sometime around January 17, 2024 someone posted the downloader I put in >>14608 on a thread here and I grabbed it. There may be some technical discussion if you wanna go back and try to find it. I just saw pixiv downloader, said "I use pixiv!" and grabbed it. All the network related stuff is way way over my head.

>>14615 Typos, ugh. I meant 2023, not 2024

>>14613 >>14614 >>14615 >>14616 Fuck it, I'll stop being lazy and just find it. Post where I got that parcer is at https://8chan.moe/hydrus/res/18976.html#19058 Not anything technical there. Just an anon asking for it, and a kind anon just posted the parcer.

Can the OR search dialog file domain reflect the file domain of the page it launched from? It always defaults to "my files" no matter what is selected and it's somewhat annoying having to change it every time.

>>14597 >Same, thanks, should be fixed tomorrow! It works now, but there's another issue now and that is if you copy a timestamp and then paste it somewhere else, it won't actually change when you hit the apply button, you have to manually edit it and then it works, but it has to be different than the time you're pasting. If you want to change the value to be the one you're pasting, you have to edit it twice (first to change it to something else > apply, then change it back to what you want > apply).

>>14615 >assuming the stuff on cuddlebear is "official" It's not and plenty of stuff there is very out of date. >>14613 https://github.com/hydrusnetwork/hydrus/tree/master/static/default/parsers The tag search parser was updated 4 years ago, the file page parser was updated 8 months ago. >>14614 What's wrong with the default pixiv search? Are you not seeing NSFW files? Pixiv won't show nsfw files if you're not signed in. Hydrus companion can make it easy to send login cookies to hydrus. >>14617 One of the replies to this post is hydev saying "I'll add this to the defaults" so it's probably no different from the pixiv parser that comes with hydrus.

>>14586 ok, in that search there is only 1 file. now, when I deleted those files, hydrus still went on like normal updating it like normal and getting new ones till something happened and it wanted to see the old ones again, there should probably be more than just one file. ok, I did everything else, there was some weirdness, finding more missing files then it searched, but I did that and got back and no change, then I restarted in case, no change, so I hit go on the first one thats paused, and it came back with this v559, win32, frozen Exception An unusual error has occured during repository processing: a content update file (12a87231d24caa8a961cfc14795e2ccab255112be2c04e04ea33d38576860009) was missing. Your repository should be paused, and all update files have been scheduled for a presence check. I recommend you run _database->maintenance->clear/fix orphan file records_ too. Please then permit file maintenance under _database->file maintenance->manage scheduled jobs_ to finish its new work, which should fix this, before unpausing your repository. Traceback (most recent call last): File "hydrus\client\ClientFiles.py", line 1599, in GetFilePath def init( self, controller ): File "hydrus\client\ClientFiles.py", line 1044, in _LookForFilePath hydrus.core.HydrusExceptions.FileMissingException: File for 12a87231d24caa8a961cfc14795e2ccab255112be2c04e04ea33d38576860009 not found! During handling of the above exception, another exception occurred: Traceback (most recent call last): File "hydrus\client\ClientServices.py", line 2195, in _SyncProcessUpdates content_update = HydrusSerialisable.CreateFromNetworkBytes( update_network_bytes ) File "hydrus\client\ClientFiles.py", line 1603, in GetFilePath self._pubbed_message_about_bad_file_record_delete = False hydrus.core.HydrusExceptions.FileMissingException: No file found at path I:\Hydrus\f12\12a87231d24caa8a961cfc14795e2ccab255112be2c04e04ea33d38576860009! During handling of the above exception, another exception occurred: Traceback (most recent call last): File "hydrus\client\ClientServices.py", line 2201, in _SyncProcessUpdates raise Exception( 'An unusual error has occured during repository processing: a content update file ({}) was invalid. Your repository should be paused, and all update files have been scheduled for an integrity check. Please permit file maintenance under _database->file maintenance->manage scheduled jobs_ to finish its new work, which should fix this, before unpausing your repository.'.format( content_hash.hex() ) ) Exception: An unusual error has occured during repository processing: a content update file (12a87231d24caa8a961cfc14795e2ccab255112be2c04e04ea33d38576860009) was missing. Your repository should be paused, and all update files have been scheduled for a presence check. I recommend you run _database->maintenance->clear/fix orphan file records_ too. Please then permit file maintenance under _database->file maintenance->manage scheduled jobs_ to finish its new work, which should fix this, before unpausing your repository.

>>14617 ty. is there anything wrong with the default pixiv parsers that come with hydrus? actually, it looks like hydev said he'd as that parser to the defaults for hydrus >>14621 thanks. what's the difference between the tag search parser and the file page parser? also idk if anything was wrong with the original since i haven't used it yet, it's just the guy i was talking to further up said he used the cuddlebear version, so i figured that was the most up-to-date.

>>14623 ok i'm using the default gallery downloader that comes bundled with hydrus called "pixiv tag search." i also imported cookies using the companion. i'm glad i checked out pixiv since it looks like the japanese have a lot of unique stuff. however, i'm noticing pixiv doesn't tag their stuff as well as the boorus do. anyone got any leet tips for blacklisting stuff like SFW, amateur doodles, and such? also i'm noticing hydrus doesn't scrape anything for artist queries even though i can search artists via the search bar on the site.

Is it possible to search deletion records? Apparently some bad urls got added at some point and now HC is highlighting things it shouldn't, which is annoying.

>>14596 Hey, thanks for reaching out. 1) Although the update files offer almost everything, they aren't 100%. There's a tiny extra bit of data you need which is still synced from the server (basically it is the list of these files). If we ever needed to wangle complete offline sync of a repo, I could figure out a technical solution here, but for now these update files and their import/export capability is mostly only useful for fixing weird problems. a) Yeah, some people don't want to wait for all the CPU-intensive processing, so that QuickSync pre-processed client became popular. I always recommend a 'natural' sync tbh--people who do quicksync or forced megafast processing in other ways tend to run into maintenance problems as certain maintenance timers don't have time to kick in. (I tried to fix this a bit last week, if you check the changelog for the analyze stuff, it has been a long time problem). I don't offer the files because the PTR sync already grabs them. That's the best way to import them to your client, and future iterations of the hydrus network protocol may well retroactively change the files, so they aren't super worth backing up. b) You can import them, but you'd need to do one single metadata sync to the server. Import the files to your client, set up some PTR credentials, then put it online and hit 'refresh account' once. The progress bars down below will populate and the client will see it already has all of the update files it needs to catch up to a week ago or whenever your export happened. Yeah don't make an HTA of the PTR I think. Too much data. c) You can delete them, but I think hydrus will just redownload them. They will be needed again if you ever run into a database problem and need to reprocess some data. 2) That thing is pretty experimental, I think it is hidden behind 'advanced mode', right? It changes which tag service the collections math works on. I did it with a user who has a very complicated namespace collection system, but we didn't go beyond this. I'll fix the checkbox thing, thanks. Just leave it on 'all known tags', which is the default, unless you want to go crazy. 3) The file domain filter does change which results you get, and the tag counts should be accurate, and particularly if a tag has 0 count in that file domain it isn't supposed to show up, but there are some hacks that ensure that (tags that match what you typed because of a sibling relationship) will show up despite having zero count. In your pick it is obviously all zelda stuff showing up. I had thought I'd fixed this to work in a less confusing way, but I guess it is still a bit of a mess. Normally these sibling tags get shunted to the bottom of a list, below results with actual count, unless you type enough that they are actually useful suggestions in some contexts. That said, if you really can add an unsiblinged 'test1' to a 'my files' file and then switch to 'trash' and see 'test1' in your results with a count, that is a problem. If you do it, what's the count of the result, (1) or (0) or blank? 'blank' should be correct. 4) Like the tag service collections thing, this was a one-off debug experiment really. I might work on it more as I do some other tag display and search infrastructure. Shortly: 'multiple media views' - the tags you see by default in that list 'display tags' - the files' tags with siblings and parents applied (no 'multiple media views' filtering, if you have tags set to hidden) 'storage tags' - the files' tags without siblings and parents applied, should be basically what you see in 'manage tags' But again, if you are a new user, ignore this. I hide it behind advanced mode because it can be confusing. 5) Yeah I think I agree. This tech is mature enough that we should have some display options and favourite domains. I'll make a job, thanks.

>>14607 Great idea. I'm sure someone has mentioned it before, but I don't remember. I'll see what I can do. >>14612 The Katawa Shojou Mishimmie used to, now looks like a dead subdomain. :( http://shimmie.katawa-shoujo.com/ Afaik it turns up here and there, but it was never that popular. >>14619 Thanks, I'll fix it! >>14620 Thank you for the report, and sorry again! I'll fix it and let's hope I'm not being stupid anywhere else. >>14622 This is a little concerning. It thought all those files were missing, I wonder why. In your second picture, where it basically says everything it found is missing/incorrect, did the 'review services' window immediately say that it had none of its update files, and did you see a popup appear within a few seconds/minutes as it redownloaded everything? As far as I know, if that file maintenance job sees missing files, it'll remove their records, causing the redownload. But why were they marked as missing/incorrect in the first place? Do you remember deleting them manually? Why, I ask myself, is your client losing updates, downloading updates, and then being unable to find them again? I am not sure. I suspect we either have files going missing or something is seriously screwed in the file storage so it doesn't know what it has, or doesn't know how to purge a bad record. This is a stupid thought, but do you have anything like an anti-virus that might be pulling files out of your client file storage? These update files are in your [install_dir/db/client_files] directory normally, they have no extension, just the hash as filename, and are gzipped json. >>14625 When you say 'deletion records', do you mean deleted files? If so, turn on help->advanced mode and then in a normal search page, click the 'my files' button under the place you type search tags and then select 'multiple/deleted locations'. Set the domain to 'deleted from my files' or whatever, and then the search page will work like any other, it'll just deliver a variety of 'virtual' results with weird/no thumbnails. You can edit these deleted files' metadata, so I reckon search up your known urls here and then manually delete them. Let me know if you need a cleverer solution here.

>>14624 Pixiv posts are limited to 10 tags. So don't expect much in the way of tags unless it's something in the PTR someone else has already tagged. The tradeoff is there is a stupid amount of content there that's not available anywhere else. And it's really, really good about recommending new and similar content/artists to you. >>14612 I was able to find a very small shimmie2 booru at https://booru.oke.moe/ after a quick google. there's surely more, but it's the first thing I found.

>>14624 Everything in pixiv is in japanese. Enter the tag you are looking for in English slowly, and you will see suggested japanese tags start showing. After that, you just have to start perusing what they are tied to.

>>14629 A few tags can get weird, like ギャグ, which is "gag" in Japanese. It seems to have a double meaning in pixiv. On one side, it means gag as in bondage, but in the other, it means comedy manga or doodle. So, would have to make a blacklist that kills any files tagged with manga or doodle, if you just wanted the bondage files. Put the tag into pixiv and start looking the files, and you'll see what I mean.

I have been trying to use the premade downloader for Inkbunny. Current client is 559. It is stuck hitting the homepage and will not log in successfully. Advice?

>>14626 Thanks a ton for all your answers. You pretty much answered everything. 3) Oh you are right. I didn't think of tag counts in this case. I thought that the autocomplete suggestions should only show the tags that are actually in the file domain, just like the normal file search pane. The tags that are not there, wouldn't even appear in the autocomplete. "test1" wouldn't appear there for example in the trash domain. But seems you handle those two autocomplete locations (search pane & migrate tags) differently. So yes, in "migrate tags" it shows "test1" in trash but without a count -> blank. Im completely fine with that. So i know that i could distinguish which tags are actually in the domain by checking if they have a count at all. Good to know! 4) I checked again and understand now, thanks. They might be very useful in future, so dont get rid of them ;) 5) If i may add two other ideas that came into my mind. a) it would be very cool if i could fast forward/backwards in videos with the mouse wheel when being on the timeline with the mouse cursor. In VLC media player for example, i configured the video to jump forward/backwards in 3 second steps. It is super convenient and it would be cool in hydrus too. If the seconds would be configurable even better! b) my default image viewer is irfanview right now and i like the option to rotate the picture in there with the R or L keys. sometimes you may just want to rotate the image. some buttons on the top bar in hydrus would also be fine if shortcuts are not possible. Maybe you like to consider those ideas. I bet the wishlist is huge :P Thanks alot.

>>14624 >>14628 thanks, i've been slowly adding stuff to search for. there's just soooooooooooooooooooo much useless SFW, doodles, and other stuff getting scraped because the pixiv stuff is poorly tagged. even just filtering out all the SFW stuff would be a massive help. also are either of you able to do queries for artists using the default pixiv tag search that comes with hydrus? i've tried several artist queries now and they all return 0 results.

>>14633 looks like i had deleted the "pixiv artist lookup" gug without realizing it (disappointed though that you gotta look it up by an id number rather than the artist name--30423409232 doesn't exactly roll of the tongue) also it seems like adding R-18 to the query is the best way to find nsfw stuff (doesn't work with the artist lookup obviously though). now the question i'm asking myself is whether pixiv is worth scraping from given the increased effort level due to bad tagging. anyone have a workflow/thoughts on this?

>>14634 I would suggest just browsing around and bookmarking stuff you like. It will start suggesting things based on that. Pixiv tags do kinda suck, but there's some AI system behind the scenes that they use for recommendations and it's pretty damn spot on. Their recommendations clearly don't just rely on tags. Find a post you like, scroll down past the comments, and go peruse the recommended works. Bookmark anything you like. Follow artists you like. You can also go to Discovery and well, discover works based on what you've told pixiv you like with works viewed, bookmarks, and followed. Most of the popular tags have english translations, so searching in english is often possible, but using japanese is usually better. There has also been a steady influx of westerners using pixiv as an alternative to places like twitter and deviantart because of pixivs general lack of content restrictions. As long as it abides by Japan's weird censorship laws, there's not a lot of banned content. Some of the most popular pixiv works end up on the various boorus. Check the booru source to look up the artist on pixiv. Follow that artist, in my experience it very rare for all of an artist's pixiv work to end up on boorus. Pixiv is an absolute gold mine of content if you spend the effort. As far as tagging goes, the most popular things might already be in the PTR from getting tagged by some booru already. Hydrus grabs the post tags fine for whatever they may be worth lol. My workflow is to do a quick pass to identify artists and characters that were not properly tagged and the most basic tags for me. Things like colors, positions, fetishes, obvious locations (like "bedroom" or "beach"). Then I export everything into Waifu Diffusion and let it do the fine detailed tagging and re-import everything with the AI tag sidecars. Tags are sent to a separate "AI Tags" services to keep them separate from PTR and my private tags. I also add a tag "meta:ai tagged" for good measure. This keeps things searchable but makes sure nothing accidentally bleeds over into the world of human tags. AI tagging isn't perfect yet, but seriously, it's like 85-90% there. With a weight threshold of 0.35 there are very few errors. It's so much faster to AI tag everything and then do a sanity check for anything wrong. Once I review the AI tags are all correct, I copy them to the PTR and add any tags that the AI missed.

If I want to start subscribing to pixiv and and kemono party, I need to give cookies to Hydrus companion. But the whole point of subs is that it's automatic. Do I need to go open pages regularly on sites requiring cookies in order for the subscriptions to work? Or is Hydrus Companion only really good for one time downloaders?

>>14637 Once you have hydrus companion setup and working with the Hydrus API, all you have to do is click Hydrus Companion --> Send cookies from this tab to hydrus. That's it. You may have to do that again occasionally but it's very rare. LIke once every few months. If there's ever any issues the subscription log will let you know and you can easily resend cookies and retry errored downloads.

>>14638 Thanks. I wasn't sure exactly how long cookies would last. Now to make a few dozen more subs for every artist I can find.

I'm trying to download via url download a bunch of 8moe files, I'm using direct links, for example: >https://8chan.moe/.media/8filename.jpg But it seems that the site content waning is popping up for hydrus when it tries to access the urls. I tried importing my session cookies but it didn't work. Any ideas?

>>14640 I have Hydrus Companion now, but I have no idea how to make my login cookies apply to my downloaders/subscriptions so they can get the R-18 posts. I've both sent the gallery to Hydrus using Hydrus Companion as a one time downloader and made an artist subscription in Hydrus using the pixiv artist ID, after having "Successfully sent cookies to Hydrus" while logged into Pixiv using Hydrus Companion, and both the one time downloader and the subscription skip over R-18 files.

>>14635 >AI tagging isn't perfect yet, but seriously, it's like 85-90% there. With a weight threshold of 0.35 there are very few errors. It's so much faster to AI tag everything and then do a sanity check for anything wrong. Once I review the AI tags are all correct, I copy them to the PTR and add any tags that the AI missed. automation is where it's at, thanks for this recommendation. i won't be browsing anything manually via the website though, that's just so slow and inefficient, especially when there's an ocean of stuff out there that i'll never even see as is. >>14637 as a fellow noob, what i'm gonna do is bookmark all of the porn sites to a bookmarks folder, and then open them up once a week to let hydrus companion automatically send the updated cookies to hydrus (there's an experimental feature in the hydrus companion options menu to send cookies automatically) >>14642 make sure you delete the login script and cookies for those sites first. also make sure you've set your region to japan on pixiv, and also make sure you're actually able to view the r-18 stuff on the site itself. you can append R-18 to your pixiv tag search downloader to make sure you're only getting nsfw stuff

>>14643 >make sure you delete the login script and cookies for those sites first. I'm running around in the dark here and haven't really even messed with cookies or html in something near a decade. I can find the cookies, but I have no idea which need to be deleted and which need to stay as I don't know which is the "logged in" cookie. It may be the one that's set to expire with my current browser session? But there's one each of that on both www.pixiv.net and imp.pixiv.net. I have no idea where or how to delete the login script. I went ahead and set my region to Japan, which I thought I already did, but it was blank when I checked. However the missed files are those I could see without changing my region setting in the first place.

>>14633 There really is no "safe" tag in Pixiv. Everythng from "safe for children" to "downright pornographic" is all mixed in together. Even pics from a single artist can really span the range of safe levels. You just have to dig. Most of the stuff your wanting is tagged in japanese. If you find something you like, search for the japanese tags in the tag list, and then look for more japanese tags in the search results. That's how I've been finding stuff. The English tags don't lead to much.

>>14645 Pixiv has a TON of stuff. Out of the 2.7 million files are so I have downloaded, about 1.6 million are from Pixiv. Sankaku has a lot too, but it is really hard to get, and you have to know how to do it all manually with Hydrus, which makes it kind of semi-automatic. Rule34.xxx is surprisingly good. Xbooru pretty much has nothing. Gelbooru is OK.

>>14631 Same guy here, but I think I found the issue. I just don't know how to resolve it. Outstanding network job displays an incorrect https. I don't know how to modify it to be correct. Advice?

>>14635 do you have any thoughts on using hydrus-dd for autotagging? also i haven't touched any diffusion stuff in a long ass time, so do you just use waifu diffusion inside a1111 to autotag stuff? i can't seem to recall which tab in a1111 allows you to add tags to pre-existing images also how exactly do you import stuff that waifu diffusion tagged back into hydrus?

>>14648 I am using this: https://github.com/picobyte/stable-diffusion-webui-wd14-tagger Adds a new tag to Automatic1111. Can tag either single images or an entire directory and export to .txt sidecars. Pic related. You'll need to grab the models as well as the above extension, but it has links to them. If you want the e621 model but don't wanna go thru stupid discord, the actual link to the model is at https://pixeldrain.com/u/iNMyyi2w I've had fantastic results with the WD14 moat tagger v2 and Z3D-E621-Convnext. The workflow is pretty straightforward. Put stuff into to a folder. Point the tagger to that directory. Make sure "Save to tags files" is checked and let it work it's magic.Then just import that folder into Hydrus and have it use the sidecar .txt files. Highly recommend adding a new tag service in hydrus though to keep them separate. See services-->manage services-->add-->local tag service.

I have images with filename and account tags, I want to retag them so the account prefixes the filename for example, >filename:2024-02-19_6 >twitter:sachiko would become >filename:sachiko-2024-02-19_6 is this possible? am I missing something obvious? >>14603 >column widths fantastic, thank you. that's been bothering me for a long time but I never got around to asking about it

>>14649 Too bad it doesn't add ratings (general/sensitive/questionable/explicit) to the txt files. Though there's an option that shits out a json of all the the tags with weights, which also includes the ratings. I tried making a script some months ago that imports the tags from the txt files with the ratings in the json straight into hydrus if your filenames are hashes. Not sure if it still works though, but I don't think there were any major changes to the extension to break anything. https://files.catbox.moe/luvqjr.py There's also a standalone that doesn't need the webui and supports hydrus, but I think you need to install the cuda kit for it to work with gpu. https://github.com/Garbevoir/wd-e621-hydrus-tagger

>>14652 Oh yeah, I forgot to mention that importing through the script using hashes as filenames is a lot faster than going through normal hydrus import, since it doesn't need to calculate hashes for every file.

>>14650 >Export your files with "[twitter]-[filename]" filename pattern >Delete old filename tags >Reimport files back with new filenames as tags Requires some extra free space and import can take a long time if you have a lot of files, but it's the only way I can think of without using the api. There's of course also the api, but I'm too dumb for that.

>>14652 Yeah, I'm using a slightly older version than Garbevoir's that got taken down off of Github, but it works great. Ai tagging is definitely the future of tagging.

>>14649 Is there a way to get it to focus on a specific tag like human/humanoid? I only want stuff with humans in the image so I can easily filter and get rid of furry stuff.

I had a good week. I fixed some bugs and made some shortcuts a bit nicer to deal with, and there are a bunch of macOS improvements. The release should be as normal tomorrow.

>>14662 Thank you for all your work. Hydrus has changed my life. Maybe one day I'll be able to fully trust AI tagging and stop manually tagging thousands of files for years on end.

>>14661 There's a filter where you can exclude tags, not sure if there's one that adds only specified tags. Just have it tag everything and filter it in hydrus.

>>14661 What >>14665 said. AI tag it, shove it in Hydrus. Search for some common furry tags like "furry", and "anthro*". Then manually delete the stuff you don't want either from thumbnails or the archive/delete filter. Or I suppose you could just shotgun it and just blindly delete anything that gets detected as furry by the AI. Depends on your level of fault tolerance.

>>14667 Just did the "delete all furry crap". I discovered the Ai tags it as "no humans". I had 63000 "no humans" garbage filling my collection. Everything from pictures of houses and fruit to furries, all of it not tagged as such by the booru's. I love the Ai tagger. It's not 100%, but it's close to it.

>>14665 >not sure if there's one that adds only specified tags. Whitelist.

>>14669 What this does is override the blacklist for any tag in it. I'm sure it can do other things as well. You just have to look at the logic of using it combined with the blacklist.

https://www.youtube.com/watch?v=XXlzWhBvYwg windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v563/Hydrus.Network.563.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v563/Hydrus.Network.563.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v563/Hydrus.Network.563.-.macOS.-.App.dmg linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v563/Hydrus.Network.563.-.Linux.-.Executable.tar.zst I had a good week. There's a mix of small fixes and improvements. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html macOS A user has improved the macOS release in many ways, mostly in brushing up the App to normal macOS standards. The menubar should now plug into native global bar, with some standardised command labels. The program icon is better, some colours should be improved, dialog menus are no longer a crazy hack by default, and the system tray icon is turned on. In a side thing, I also added Cmd+W to close pretty much any dialog or non-main-gui window in macOS, just like hitting Escape. I think we can expect more work here, so let us know how you get on! all misc In more shortcut news, I've added Alt+Home/End/Left/Right to the 'thumbnails' defaults to perform the new thumbnail rearranging. Existing users will also get these! (assuming it doesn't conflict with something you already have set) Also, the shortcut system now, by default, converts the 'numpad' variants of non-number key presses (think 'numpad Home' or 'numpad Return') into the non-numpad 'normal' ones. You now only need to map 'Left' or 'Delete' once when setting a shortcut, and, no matter how crazy your keyboard's internal mapping is, it should just work. If you do need to differentiate between the numpad and normal variants of these keys, you can turn this new behaviour off under options->shortcuts. The menu on a 'file log' button now lets you delete all the remaining 'unknown' items or set them all to skipped. I fixed another damn problem with copy/pasting timestamps into the manage timestamps dialog. When you paste a timestamp, it should 'stick' better now! If you have had problems with mpv sometimes going silent on 'every other video' or had your windows being rescued from 'off screen' even though they were supposed to appear just on some monitor, check the changelog for some special new BUGFIX options. next week I want to put some work into system predicates, specifically startng with 'system:duration'. The aim is to eventually get rid of all the +/-15% '~=' stuff and replace it with actual customisable values, along with better behind the scenes storage of that data. We'll see how it goes!

I noticed that my idolcomplex downloader wasn't working, I updated it and it seems to be alright now.

>Could not find a file or post URL to download This is the note I see on URLs in a TBIB gallery downloader > file log > show file log. There's no 403 code or anything detailed, it just says it's ignored. Any way to fix this?

>>14677 so I followed the URL's, and it redirects me to a gallery page rather than a post, implying that the post does not exist. the question then becomes why is TBIB gallery downloader fetching URL's for posts that don't exist?

>>14678 fyi, this isn't happening for all TBIB, just 90%+ of them.

>>14677 >>14678 >>14679 I think there's a bug with TBIB, if you click the first image result under the search query "blush" (picrelated), it should link to: https://tbib.org/index.php?page=post&s=view&id=13851516 but instead it redirects me to: https://tbib.org/index.php?page=post&s=list&tags=all for some reason. I'm not going to click 500 images to get a proper analysis, but a bunch seem to have issues like that.

>>14680 oh wow nice find. i guess i'll have to stop using tbib then

are wildcard tags supposed to have an implicit * at the beginning? for example, searching "hair*" returns "hairband", as expected, but also "red hair" or "filename:16*" returns "filename:2022-01-16_04-33-28" it's not really a problem, you can use ^ to denote the beginning of the tag. I've just never noticed before >>14654 thanks, it was a bit painful but I managed to do it through the API

Does someone know how to force Hydrus window to appear in the main monitor when you are using two monitors. I recently downloaded Hydrus (Hydrus.Network.563.-.Windows.-.Installer.exe) and after the installation the main windows always appears in my second monitor (left) and not the main one (right) even when I move the Hydrus window I still have this problem like the one you can see on pics related, were the rest of the dialogue text and windows still appear on the second monitor instead of the main one.

>>14683 It ca be finicky. I have this issue myself. I can't recall the exact process, but involved, >Changing the main monitor in Wangblows settings >Unmaximizing the window >Dragging it to the preferred window without maximizing >Dragging it to the top of the monitor to auto maximize >Repeated testing by minimizing and re-maximizing the window see if it took Furthermore, due to some QT issues I can't get popup windows to appear on the correct monitor either, my lower one, and they would always snap back to the top regardless of saved location settings. But I could rig it up so that 98% of the tags management window, save for the title bar, is on my bottom monitor. I have to open the tags management window, inch it down, close it to save it's position, and repeat. Moving it too far at once will cause it to reset. However is does still move of it's own accord slowly to the left, which is frustrating, but I can move it back to the right all at once without issue. Hydev says I can resolve these issues with some experimental QT stuff I could fiddle with, but it's usable right now and I'd rather not break it further. I'm fairly certain this issue will be resolved when I get a real computer instead of a craptop so that my monitors are identical.

>>14680 I'm getting this on some subscriptions and in browser too the last couple days. I guess tbib must have broken something pretty good. Seems to happen randomly, but seems most common with any wildcard searches. Sometimes you get the same 3-4 results over and over and over for dozens of pages before it fixes itself and starts displaying proper pages.

>>14652 what do you mean by "importing through the script"?

>>14689 Nevermind, ignore this. What alternatives are there to using the a1111 tagger extension? The lack of rating tags for images is driving me nuts.

>>14689 >what do you mean by "importing through the script"? I mean using the script in the catbox link I posted. Basically you're supposed to use it like this: >in webui settings, under "Tagger", find the "Store images in database" option and turn it on This should add image paths in the db.json that gets generated when you run the tagger, which the script uses for assigning ratings to the correct file. I think the json should be generated by default, but in case it doesn't, try looking for an option that turns it on. >drag and drop your files from hydrus into a folder, this should make the files have hashes as filenames >run the tagger on this folder, txt files will be created for each file along with a db.json that has a list of all tags and scores and file paths >open command line in this folder and run the script from it (don't forget to use the -k and -s arguments for hydrus api key and tag service name) The script will then look in both the txt files and the json and sends the tags and ratings from them straight to hydrus using the filenames as a sort of an "address" to the correct file in the hydrus database. Doing it this way is also a lot faster than importing normally, because when you import normally, hydrus will calculate hashes for every file so it knows, if the file is already in the db or not, which can take a while. You're basically skipping that part. >>14690 The link at the bottom of that post is pretty much the only alternative I know of.

>>14690 Do you use the PTR at all? If so, it has many tags that have ratings paired (e.g. anus --> rating:explicit). If so, you can migrate the PTR mappings/parents/siblings to wherever you're sending your AI tags. This way you can get at least some of the ratings automatically added. See Tags --> Migrate Tags. Set the source as the PTR and destination as your AI tag service. The PTR has a ton of parent/sibling associations, many of which pair with rating:explicit. This can help reduce the workload there. Rating:safe and rating:questionable are pretty subjective, so I don't think there's many useful siblings for those. If you don't use the PTR, or if the PTR doesn't have enough siblings for you, you can always make your own.

>>14632 >it would be very cool if i could fast forward/backwards in videos with the mouse wheel when being on the timeline with the mouse cursor Amazing idea! >my default image viewer is irfanview right now and i like the option to rotate the picture in there with the R or L keys. sometimes you may just want to rotate the image. some buttons on the top bar in hydrus would also be fine if shortcuts are not possible. Yeah, I think I am about ready to do this sort of thing. The media viewer code has been crazy in eight different ways until very recently, but there are definitely times you just want to do this for whatever reason. >I bet the wishlist is huge Yeah, it used to be several thousand items, and then I stopped counting, ha ha ha. I am always drowning in work, always accepting more to-do items than I can clear in a week, so I have generally fallen into a marathon of 'just get some decent things done every week', and when things aren't so crazy, I can plan out a multi-week or multi-month project. Although it is a problem, it is a nice one to have. Btw, if I say I'll do something that you care about and it doesn't happen in a few releases, please do remind me. I'm culling what I can achieve every tuesday night, and many things just slip to the eternal back burner. >>14641 iirc 8chan needs to referrer URLs in order to work. As long as there is a referral URL, and that URL is 8chan.moe (just as if you had clicked in your browser), you skip the click-gate. It stops direct linking of spicy boards I think. My internal network engine handles this automatically these days with the particular sort of 'API' downloader we use in the 8chan watcher, but if you are downloading URLs direct, I think you might need to hack it. Maybe you can change the 'url class' under network->downloader components->url classes to always force a certain fixed URL class? Not sure if it'll work. If you can, I'd say just try to watch the respective thread URLs in the normal watcher, I think it'll work. >>14663 I'm glad you like my program! Changed my life too, lol. I'm really looking forward to the next ten years. I think teaching a model to recognise 'hair_ribbon' is only the tip of the iceburg here, and, if I can, I am determined to do my part in propping up the open source/unshackled side of AI.

>>14677 >>14678 >>14679 >>14680 >>14681 >>14688 Thank you for this report. I can't promise anything, but since I roll TBIB into the hydrus defaults, I'll see if I can fix this. This stuff happens every now and then. Sometimes stuff like if the booru inserts an advert in the place of a normal thumb, or a 'buy premium to see this' swap-in, that sort of thing. Just the parser sperging out over a change in html, usually. I'll check it out. >>14682 It basically follows these rules: https://www.sqlite.org/fts3.html but there is additional bullshit I've added. It essentially has 'start of string or a whitespace' at the start, so 'hair*' will match your 'red hair' but not 'redhair', with the additional caveat that in the search code I collapse all weird punctuation like '[' or '-' into whitespace, so that's how your '16*' is matching the date. I never knew you could prefix with '^' to say 'start of string only'! I should think of a nice 'hydrus' way to integrate that. >>14683 Sorry, we have had a bunch of multi-monitor issues recently. Newer versions of Qt are having trouble doing screen coordinate calculations for some users, and I am hoping to have it magically fix when I next update Qt in the build (although I know the newer version still breaks some stuff for some people). I added a BUGFIX setting in v563, under options->gui, called 'Disable off-screen window rescue'. If this helps you at all, please give it a go. If this continues to be a super pain for you, running from source lets you try different Qt versions, so you may be able to discover a better version for you. Check this document out if you want to think about it: https://hydrusnetwork.github.io/hydrus/running_from_source.html

>>14643 >make sure you delete the login script and cookies for those sites first. I'm confused. Doesn't Hydrus need my login script and cookies to for subscriptions on account locked content to work?

When a sub gives me notifications on a number files it just added, is there any setting I can use to make it not count files I already have? I keep getting notifications that are essentially about how Hydrus found new urls for hashes I already have.

>>14695 all the login script is doing is dealing with cookies. you're bypassing the need for the login script by import the cookies yourself via the companion (the latter is more reliable)

>>14692 i'm a very casual nooby. is there any risk of my breaking/overwriting my local tags and stuff by connecting to the PTR?

>>14694 regarding the TBIB breakage, it sounds like it's on their actual site itself, so IDK if you can correct things on the hydrus end or not.

>>14698 If you run the 'add the PTR' command it makes a new tag repository, no worries there. The main issue is that the PTR is huge, it's like 60gb and takes a week or two to fully catch up since it has to download and process in order.

>>14698 No. by default there is only a "My Tags" service. If/when you add the Public Tag Repository (PTR) if will create a second tag service, unsurprisingly called "Public Tag Repository". It's completely separate from My Tags. You can find out more at https://hydrusnetwork.github.io/hydrus/PTR.html As >>14700 says, it does take a pretty good chunk of space. And only do it if you have hydrus running on a SSD. If you have no interest in the PTR you can manually make parent/sibling associations on the default tag service. It'll be a little bit of work, but could save you a lot of time. Parent tags in particular. Tag parents let you automatically add a particular tag every time another tag is added. (anus adds rating:explicit automatically as in my prior example.) Tag siblings let you replace a bad tag with a better tag (lotr gets replaced entirely by "series:lord of the rings"). Migrating siblings/parents from PTR to another tag service just copies them over for use in another service. For specifics see: 1) https://hydrusnetwork.github.io/hydrus/advanced_siblings.html 2) https://hydrusnetwork.github.io/hydrus/advanced_parents.html

>>14697 Companion says it successfully got the cookies I sent while logged into Pixiv, but my pixiv sub that I made and ran immediately after still misses everything R-18.

>>14702 a few things: 1. delete the login script as discussed 2. reset the login cookies under 'manage logins' 3. on the pixiv site, change your account region to japan 4. make sure you're using the pixiv tag downloader for tags and the pixiv artist id downloader for artists. these are the downloaders that come with hydrus by default. 5.make sure your query is good. the pixiv search will autosuggest the japanese version of your tags as well, so maybe search for that instead since the results can be better 6. give it time. there's A LOT of sfw & otherwise garbage images depending on what your query is.

>>14703 >1. delete the login script as discussed >2. reset the login cookies under 'manage logins' I have no idea where to find these, or if I'm supposed to be looking for them in Hydrus, Hydrus Companion, or my browser's developer mode. I've looked for a bit through all three. I'm using Ungoogled Chromium which is one of the suggested browsers for Hydrus Companion >3. on the pixiv site, change your account region to japan >4. make sure you're using the pixiv tag downloader for tags and the pixiv artist id downloader for artists. these are the downloaders that come with hydrus by default. >5.make sure your query is good. the pixiv search will autosuggest the japanese version of your tags as well, so maybe search for that instead since the results can be better >6. give it time. there's A LOT of sfw & otherwise garbage images depending on what your query is. All of this is fine. I'm using the artist ID downloader, and I'm successfully getting only the SFW art by any given artist. My issue is purely not understanding what step between >Send cookies from this site to Hydrus and >Make a subscription like usual must be taken and how. I've solved the issue. I was in incognito mode out of long ingrained habit, and that effects the cookies. Sending cookies from a regular window and then making a subscription functions correctly and gets R-18 content, contentious tags and all.

I might be super autistic, or just biased towards other boorus, but whenever I add a new namespace and it comes in-between "series:" and "character:", it bothers me to no end. I wish we could have a custom namespace order for tag display. I know you can do a dirty hack by prefixing namespaces with a number to get a custom sort order, but it makes them look very grotesque. And hiding them is like brushing a turd under a carpet. Also, hiding namespaces of the same colour like "page:" and "chapter:" that 99% of the time contain only numbers, makes viewing tags with them very confusing (pic related). So hiding/showing individual namespaces would be a cool feature. >inb4 just change the namespace colour

>>14705 >So hiding/showing individual namespaces would be a cool feature. You can already do that under tags > manage tag display and search.

>>14706 Sorry, I didn't mean hiding the entire namespace with subtags, just showing some namespaces, when this box is unticked. So when everything else is hidden, namespaces like "page:" would still be visible. I know it sounds stupid, but seeing only numbers (instead of "page:5" for e.g.) with all namespaces hidden, is not very informative.

My subscriptions are getting a little unwieldy. I have 120+ now and I still have many more to add. Is there any way I can group them up? My subs are all artist subs on various sites for each artist. Just being able to group by artist I could put 2-5 subs per group.

>>14627 they were deleted manually years ago, at least a number of them were, I believe at this time there was no way to find them in program, or I had no idea how to find them in program, essentially it came down to I needed to buy time to new hdd because I was completely out of space and could not parse enough fast enough to make it to the finish line. nah, its not my storage that messed up or anti virus, it was my dumb ass desprate to make it to new hdd. like I said here >>14494 this is entirely a problem of my own making and I have 0 idea if anyone else was dumb/desperate enough like I was to do what I did and no failsafe was in place for it... I do have an idea that potentially could work. I could run a bare client and redownload all the files that was, then merge them into the main clients folders, think that would work or would there be a way to reset this in the main client?

Having a minor issue with Hydrus companion. If I send a whole pixiv page to Hydrus with the send current tab button, it works just fine, but if I send just one file through right clicking, it's ignored with a 403 note.

>>14710 You can't direct link pixiv images.

Would it be reasonable to have the "copy small bmp of image for quick source lookups" work for videos/animations as well? Such as by grabbing the same frame used for the thumbnail, or the thumbnail itself?

Is the deviantart parser broken for anyone else? Tried importing cookies from my account to hydrus, still doesn't work

>>14712 What source lookup site or tool works even with videos? I'm still eagerly awaiting partial duplicate processing functionality for videos that uses the first frame of a video while ignoring any videos that have a solid color first frame.

>>14713 Did you import the cookies from an incognito window? That breaks it, as I've recently found.

https://nekohouse.su/ just popped and seems to use the exact same structure as kemono party, made to be more JP art centric and go down less. You think kemono party downloaders will work for it out of the box since it appears to be the same site infrastructure, or will new downloaders need to be made?

>>14713 just use gallery-dl and set up an import folder with sidecars

I had a good week. A bunch of system predicates now support customisable +/- ranges, both by absolute and percentage values. The release should be as normal tomorrow. >>14694 >>14699 Thanks, yeah, I looked just now and it seems the site is 302 redirecting to the main index from a variety of normal Post URLs. It works in the browser too, for instance this, https://tbib.org/index.php?page=post&s=view&id=13954530 I've seen this sort of thing before in some boorus to block guest user access to spicy content. I don't know if TBIB hosts this sort of thing, but maybe it is similar. Maybe you need to make an account and click a 'allow x content' flag somewhere. Normally though, when this sort of thing happens, they won't show you the thumbnails though, they'll either obscure them or hide them from results completely. Maybe the site is just sperging out.

>>14720 i'm logged into TBIB and have an empty blacklist, but it's still redirecting me to the main index page for the link you posted. they must've broke something

I heard someone mention kemono blocks scrapers and that you need Hydrus Companion to send to cookies to Hydrus to circumvent this. How will I know it's working though? Does kemono immediately recognize my subscription and block it without cookies, or does it take a certain amount of fast activity to trigger it? How do my cookies prevent these triggers?

>>14722 >kemono blocks scrapers I haven't heard this but I'd assume that if they do block scrapers, it's because they already have a public API. just use that. works for me with no cookies needed

https://www.youtube.com/watch?v=Oo0o84-TJTU windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v564/Hydrus.Network.564.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v564/Hydrus.Network.564.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v564/Hydrus.Network.564.-.macOS.-.App.dmg linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v564/Hydrus.Network.564.-.Linux.-.Executable.tar.zst I had a good week, several system predicates have better range-based searching. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html highlights The system predicates for width, height, num_notes, num_words, num_urls, num_frames, duration, and framerate now support two different kinds of approximate equals (≈): absolute (±x), and percentage (±x%). Previously, the ≈ secretly just did ±15% in all cases, but now you set how and how far they go. Also, any 'system:framerate' predicate that was '=' will now be converted to ±5%, which is what it was secretly doing before, and any 'system:duration' '=' predicate will also be converted to ±5%, which is what it really should have been doing before. 'system:duration' also allows hours/minutes input, for longer videos. This predicate overhaul was an important cleanup job, replacing a ton of hacky ancient code with something that is easier to update and maintain. I've collapsed all these preds down to a lot of shared UI and logic, so let me know if there are any display/search quirks, but once we have it nailed down, I hope to replicate this work for the more complicated system predicates. I reworked what 'Space' does in the media viewer by default. I am updating existing users too, so you'll probably get a little popup about it when you update. Essentially, if you are still on the default shortcuts, Space will now only send 'pause/play media'. It no longer does 'yes' on the archive/delete filter or 'this is better' on the duplicate filter. If you want to go back to how it was, sorry for the trouble--hit up file->shortcuts to set it back. Thanks to a user, space also does a new 'Quick Look' for macOS users on thumbnails. Try it out! If you are a 'running from source' macOS user, make sure to rebuild your venv this week, or it won't work! next week I'd like to figure out incremental tagging on the manage tags dialog, so you can select 20 files and tag them page:7 through to page:26 in one step. Let's see how it goes.

>>14726 >I'd like to figure out incremental tagging on the manage tags dialog, so you can select 20 files and tag them page:7 through to page:26 in one step. Let's see how it goes. This would help me with some filename tags. For ordering and exporting purposes, I keep filename tags on ordered sets of images, but with subscriptions, lots of ordered sets could come in out of order with no filenames. If I could select a group of files I had just manually ordered and then tag that "filename:big-ol-tiddies 0" through "filename:big-ol-tiddies 324", that would be great.

Would it be feasible to add a function for editing pictures without following the export, edit, import, set this file as better, and then delete the original, process? A sort of one button solution that tells Hydrus "I just edited this file, attach all the tags to the new hash and trash and/or delete the original."?

>>14726 Thanks for the helpful "I shuffled around mappings" warning at first launch. Thanks for not being a "fuck you, read the whole changelog" dev and caring about UX, anon! Even paid, proprietary software usually gets that wrong.

can anyone save me some trial and error and tell me what the ideal bandwidth settings are for lolibooru? i keep getting pauses due to "serious domain errors" and i have it set to 1 request every 3 seconds.

>>14732 I have one request a second and 10gb a day and rarely if ever have problems. Could it be an overall network issue on your end? Say your network card goes to sleep, could that trip Hydrus' "domain error" flag?

>>14733 i'm not sure, i don't have a dedicated network card. i use a VPN and stream stuff, so surely I would lose internet connection if my "network card" or the equivalent was going to sleep. i don't appear to have this issue on other domains either, just lolibooru. lolibooru doesn't seem to block my VPN either, so i don't think its that

>>14671 >If you have had problems with mpv sometimes Hey hey, anon from >>>/hydrus/19467 here - I was still reproducing the issue sometimes, without being able to figure out anything more about it that I could share with you, so I didn't message again about it. It just happened again with my client fully up to date, so I was able to verify that the debug -> gui -> isolate mpv thing fixes my issue without restarting the client. Thanks devanon!

Hydrus crashes when I drag and drop stuff into firefox. Has anyone else experienced this? Arch/Wayland.

It's probably a dumb error on my part, but I can't for the life of me get hydrus-web/hydrus.app to work properly. Every other thing using the client api works just fine so I don't really get it. I just get "0: Unknown Error" for every request. Any of you anons run into & fix this issue? This happens regardless of local or remote, so 127.0.0.1, and I can access the client api page from my browser, CORS support is enabled too.

>>14696 Yeah, I call it the 'presentation options', which is inside 'file import options'. It just governs what a downloader broadcasts when asked to show what it has. You can set the default file import options under options->importing. I like to set my subs (and downloaders tbh) to 'new files'/'must be in inbox', and then I'm just looking at legit new stuff. >>14705 Yeah I totally agree. Unhelpfully ordered namespaces drive me nuts too. I always want namespace x, y, z at the top. I'm going to do a little some tag sorting work this week, maybe it is time to finally pull the trigger here. Hiding/showing some namespaces is something to consider. Maybe I can figure a hacky system that doesn't eat up too much CPU to actually calculate. >>14708 For technical reasons, it is much easier to group subscriptions by the shared downloader. Hydrus wants to have subscriptions like 'safebooru artists', where you'd have 60 artists all on safebooru. You might then have 'danbooru artists', which a similar artist list, all on danbooru. Nested GUGs, if you know what that is, allow you to play around and merge stuff, but it can get complicated and the juice generally isn't worth the squeeze imo. The tools you want are the 'merge/separate/deduplicate/lowercase' buttons in the edit subscriptions dialog. I recommend you gather all your subs on site x and y and then click 'merge'. It should collapse them all down, and they'll then all share the same options, much easier to manage. If you have done some complicated NGUG or custom downloader gubbins to somehow gather 'one artist, many site' subscriptions, rather than 'one site, many queries', I haven't got a good solution other than moving to the simpler technical format. Let me know how you get on! >>14709 Ah, thanks, I see your problem better. It is odd that the client is not fixing itself in your case. Normally, if update files are simply missing, it'll run its scans and figure out what it needs to redownload. I'll do some tests here, maybe I broke this scan recently. Yes, you can absolutely set up a fresh client extract on your desktop, add the PTR, and then export/import those update files. I think and hope it'll fix you right up. Another option is to get the quicksync download, here: https://breadthread.gay/ which I assume only has the update files in its file store, but it is 30GB, so maybe more trouble than just setting up a new one yourself, since you just want to download the files, not save time on processing.

>>14712 >>14714 I know some Apps that do video de-dupe, but yeah they all rip a more complicated 'similar files' hash than just one bitmap frame (most basically just grab a phash for every nth frame). I'm going to do the same when I implement this myself. If you are feeling clever, you might want to explore this, which does external video de-dupe via the Client API: https://github.com/hydrusvideodeduplicator/hydrus-video-deduplicator >>14716 I just did a little test myself, copying the URL template and trying https://nekohouse.su/api/v1/fantia/user/292282/post/2528700 https://nekohouse.su/api/v1/twitter/user/shrimp3528_mmd/post/1760846417289937043 https://nekohouse.su/api/v1/fanbox/user/66119767/post/7392693 instead of https://kemono.su/api/v1/patreon/user/3506306/post/52615502 but it gives me a 404. As you say, the site looks like it is using the exact same engine, so maybe they just haven't turned the API on yet. Or maybe the kemono/coomer guys are a separate outfit and hacked their own API together? Or maybe instead of 'fantia'/'twitter'/'fanbox', they use different site codes for their API. >>14727 Hell yeah. >>14728 Yeah, in future I'd like this. It'll work when we eventually add 'convert this ye olde shit to AV1 mp4'/'Jpeg XL' tech, too. I need to write better metadata-merging tech first. The shit in the duplicates system is go unwieldy atm, I hate it. Once that is better at doing automated stuff, and is generally nicer to work with, I can see us going in this direction. >>14730 Great. I'm slowly learning this myself, so let me know when I do get it wrong--it is often surprising, being on the inside, what most affects people on the outside.

>>14735 Great, thanks for letting me know. This fix is a bullshit hacky thing, so let me know if it starts giving you other trouble. We are basically just opening up new copies of mpv every time that command is set. There's another guy who is going to put time into mpv, maybe figure out non-crashing mpv window destruction, so fingers crossed we'll have nicer solutions here in future. >>14739 I've been working with a guy whose client freezes when he tries to drag and drop page tabs, but a crash when dropping things onto a browser is new. When you say crash, do you mean the program actually halts instantly and disappears, or do you mean it freezes/hangs indefinitely? Unfortunately, when I initiate a drag and drop, I kind of give up a bunch of my ownership over what's going on. I basically bundle some data up and attach it to a 'clipboard' on the mouse for the OS, and the OS handles the rest, so it can be really tricky to debug this stuff. I assume you are ok dragging and dropping files to other programs, and between hydrus page tabs? On Windows for a long time, discord has been a pain in the ass, because DnD on Windows has some unusual security/permission stuff going on in the background. A 'move' DnD will work, but a 'copy' one won't. I wonder if something similar is going on with you, although normally the result would be 'fail' than a full crash. That said, I'm sorry to say, I added a section to the 'getting started' help this past week basically saying 'Wayland does not work well with hydrus (Python Qt) right now, sorry' since we have so many problems with embedding mpv windows and stuff. Some things just seem to be broken, and our only hope is waiting for a new Qt/Wayland version to fix them. That said, I'd like to fix you if I can. What happens if you try to drag and drop a page tab onto firefox? It should do nothing. Can you drag and drop page tabs to reorganise them inside hydrus, or does that cause problems too? Can you drag and drop files from firefox onto hydrus (it'll probably give you a bullshit bitmap in your temp dir, so don't try to import it, but does it work at least?)? >>14740 The guy who makes that is active on the discord, if you would like one-on-one help from the expert. I know some people have had trouble setting it up, and the solution tends to be variants of 'oh, my OS has some bullshit firewall', so going to 127.0.0.1 is the typical first step to reduce the possible number of walls in the way, but you've already done that. If you put https://127.0.0.1:(your client api port)/ into your browser, do you get a naked ascii lady? (you might have to click through an ssl warning page) The 'open client api base url' button in review services should do this for you. If you get the lady, then you have visibility, and I'd have to guess the hydrus-web setup is some bullshit like http vs https or something.

Is there a way to connect a Hydrus client to a database hosted on a different PC? I run my Hydrus off an Ubuntu server and I already have remote access to it through hydrus.app, but when I'm at home I'd like to be able to use the normal Hydrus client to access it. The github docs seem to say it's not possible, (the "don't run a database from a network location" thing) but I want a second opinion before writing it off. I don't even know if Hydrus can do multiple clients accessing the database at the same time. I mainly want this so I can do some much needed de-duping without having to hook up a monitor to the Ubuntu machine or have to do it all in a VNC window.

hydrus is great, but the sankaku downloading sucks. what's the best alternative for downloading from sankaku, or does it all suffer from the same issue that hydrus does of needing to constantly refresh headers every 48 hours?

Is there any way to easily copy hydrus settings from one database to a new one? I have it set up as I like it but it's a lot of options to change on a new database... talking about options and shortcuts mainly

>>14741 >'presentation options', which is inside 'file import options' Danke. >I like to set my subs (and downloaders tbh) to 'new files'/'must be in inbox', and then I'm just looking at legit new stuff. This gets me halfway, but that still gives me notifications every time it finds something I already have, but have yet to process and archive. Would it be feasible to add an option for notification only if a "new" file is completely new, i.e. "not already in db"? I don't see an option that fits this.

Forgive me if I'm just retarded and overlooked this in the doc, but am I able to essentially "reverse search" my existing, unorganized image collection in say gelbooru and automatically tag them all? I see tag options for different sites but will the program actually pull the tags from the copy of the image on the booru, or will I just need to manually tag them? Would prefer this as opposed to the AI tagger I see people mentioning.

>>14748 see >>14385 you could also look into installing the PTR, since most files from boorus will be in there.

Looks like Nitter is kill. There goes my twitter artist scraping. https://github.com/zedeus/nitter/issues/1175

>>14746 I would like to know this too, as I would like to move some files to a separate one purpose db. The only thing I thought of would be just copying the db folder without the files and then clean up the missing files and tags and such from inside, but that might not be very clean.

Hydev, >>14495 again, I'm having the issue once again, this time I definitely don't have an x11 client limit causing the problems. I have seen the mpv media viewer floating occasionally, usually it's when hydrus has frozen (I need to de-clutter my session), the mpv player still plays but I can't interact with it. I haven't seen mpv fully detach from the UI like you mentioned in >>14551. Reading the changelog for 563, I don't think the mpv debug option applies to my issue, but I tried it and it didn't fix the issue unfortunately.

>set gallery download with a few thousand images >forgot I didn't empty trash, eventually get full disk I/O errors >Hydrus handles it by pausing and tells me how to unpause in the error message Perfect, this probably saved me a whole lot of headaches. I've had programs that become unusable due to disk space errors before so this is refreshing.

>>14750 selfhost with an account

>>14755 I'm retarded, so no idea how to do that.

>>14741 tell me if you you see something broken, ideally I can do it all from in my own client, but if you dont find anything ill set up a temp client to download from ptr as for that file, little iffy because at some point hydrus wanted to recheck all prior repository files, so if it wants to do that again its just going to break in the same way, it may solve the problem for now, but its kicking the issue can down the road.

>>14755 that could get you banned and also has no privacy advantages

God damn, kemono has damn near everything. I should have been browsing this and making Hydrus subscriptions with it before looking at boorus.

I had a simple week, mostly fixing a bunch of bugs. The release should be as normal tomorrow.

https://www.youtube.com/watch?v=lgpD6OsHCKU windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v565a/Hydrus.Network.565a.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v565a/Hydrus.Network.565a.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v565a/Hydrus.Network.565a.-.macOS.-.App.dmg linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v565a/Hydrus.Network.565a.-.Linux.-.Executable.tar.zst I had a simple week. Lots of small changes today. The update step may take a couple of minutes. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html highlights You are going to get a couple yes/no dialogs on update this week talking about deleting some mis-parsed URLs. If you do not manually store weird data in your 'known urls' store, just click yes. If you have lots of URLs, the work will take a couple of minutes. options->sort/collect now has four different default tag sort widgets! You can set the default tag sort for search pages, the media viewer, and the manage tags dialogs launched off them. There's a new 'media' shortcut action 'copy file known urls', that copies all the known urls of your current media selection. Sidecars set to import to file notes now have an optional 'forced name' field, so if you ave a .txt file with only note text, no name, you can now force it. Some of the UI is less jank here too. The tag filter UI also got a little polish. There's less logic jank, better labels and tooltips, and you can now copy listed namespace entries to the clipboard and get something you can paste back in elsewhere. next week I did not get to the manage tags dialog incremental tagging thing this week, so I'll try again.

>>14762 can't wait for the incremental tagging, that'd be gigasick and very helpful perchance

>>14745 Each gallery page body contains all data required to generate permanent static post and file urls without resolving the session-dependent dynamic urls. I just do not know enough js to make a userscript that would fix the links. Also there's still throttling that prevents mass downloading so you can't subscribe anyway. >>14504 >I solved this by just updating the gallery parser to provide old-style urls. Can anyone share it please or better yet give some pointers how to do it.

There is a pull request open to add read support for Jpeg-XL (JXL) files to Pillow. This is the image library that Hydrus uses, right? If this is added, would you be able to add JXL support to Hydrus, or do you need to wait for write support too?

anyone have issues with hydrus finding far few images for a tag on pixiv than is available on the site itself? I tried restarting the search from the URL before no new links were found, but to no avail.

>>14765 >Can anyone share it please or better yet give some pointers how to do it. The md5 is in the src attribute of the img tag of a thumbnail on the gallery page. You just have to parse it out of the url. Then you can avoid the new alphanumeric urls like https://chan.sankakucomplex.com/en/posts/60rvLD5GZM3 and instead make the older md5 urls like https://chan.sankakucomplex.com/en/posts/b427a29e4efbae0559946ff4ebf433b0 or https://chan.sankakucomplex.com/posts/b427a29e4efbae0559946ff4ebf433b0 or even https://chan.sankakucomplex.com/post/show/b427a29e4efbae0559946ff4ebf433b0 As far as I can tell there's no way of getting the super old number ids anymore.

>>14743 I just updated to Plasma 6 and am having similar issues. Does Hydrus work well under Xwayland?

>>14767 fixed it. retrying from the most recent gallery URL didn't find any new results for some reason, so I had to redo the entire gallery downloader to get it to find the rest of the files.

Was in the middle of updating a few versions at a time and thought I'd run an integrity check, it's been going for 20 hours now. Is this normal or did it crap out? DB is around 80gb, UI stopped responding but it's still reading the db constantly with a single rowid out of order error in the console.

>>14744 I know some guys who do this, talking to a database over a network connection, even on different platforms. I don't recommend it, but it _can_ work. A) don't run the same database from multiple clients at once B) be careful, make backups, have a reliable NAS connection My dream here is to write a proper interface using the Client API so you can just dial into another client within a client. I'll be able to support full file domain searching using a file domain that just happens to be on another client, and we'll live stream metadata and thumbnails and files as needed. We'll see how it actually goes IRL. >>14746 >>14751 Not really, I'm sorry to say. If you are feeling brave, you can poke around client.db with SQLite Studio and extract the dump_type = 6 json object in 'json_dumps'. That's the core 'newer options' structure, which holds about 98% of your options. Replace the one in the newer database, and I _think_ things will be mostly fine. You MUST make a backup beforehand if you try this, because it could well fuck up. Make sure it survives two options dialog open/close cycles and a restart and another two open/close cycles. >>14747 Interesting, I thought that filter setting was supposed to do exactly this. 'all files' does successful and already in db, but 'new' is supposed to just do 'successful'. I wonder if I am screwing something up here when it combines with 'inbox'. I'll check it out! >>14752 Damn. I was talking to an X11 guy earlier today who had an issue where one mpv window worked while the other was blank, and the new debug thing worked. Thanks for letting me know. Although I'm sure my code can be cleaned up, I think most of the answer here is to wait for newer Qt/X11/mpv/whatever library versions that don't clash so much. I'm still hoping to get a 'future' release with a newer Qt version out in the next month. I'm sorry that I forget if you are running from source or not, but that may be the next thing to try here. Running from source means one less layer between the code and your OS, which can often smooth out .so file conflicts and such. https://hydrusnetwork.github.io/hydrus/running_from_source.html Let me know if you try it out, and if so, how it goes! >>14757 Yeah, sorry for this. I had a good look and tried replicating various 'this update file suddenly went missing, what does the client now do?' situations, and I could not reproduce your problem. I think I cleaned up a little of the code in v565, but I don't think I fixed anything serious. I can't claim this is anything appropriate for you, but I've been working with a guy with a similar problem recently and we discovered his hard drive was completely fucked and had damaged his database. I think your problem is my code being dumb somewhere, but you might like to give 'install_dir/db/help my db is broke.txt' a skim to see if anything stands out. Maybe run an integrity check on client.db, which will be nice and fast.

>>14766 Hell yeah, I'm waiting and hoping for PIL to just one day support it. A guy I know was playing around with the Jpeg-XL plugin a while ago, he said it wasn't ready. If it is now ready, then yeah, what'll happen is: - PIL merges the pull - I see/someone tells me - I add that new PIL version to the advanced test requirements.txt and advanced source users play around with it. I probably only have to write about ten lines of code to get it actually working in hydrus. - I make the next 'future release' of hydrus use it, people test it - I update everyone to use it PIL is a great and very stable library (used to be python only, not sure if it still is), so I imagine this process, when it happens, will be super easy. We are mostly just waiting for them to do their thing. After that, we'll need some applications/workflows that actually produce Jpeg-XL, not to mention in general the whole internet community has to keep putting pressure on the big guys, Chrome and Firefox, to support it. They do not like the format, even though it is pretty much our best option, so we need to actually use it for stuff and start yelling about it, and we might just get a good end here. Think, if you are old enough, to when 4chan added webm to /gif/. If the nerds start using Jpeg-XL instead of wanged out HEIC or whatever, then the browsers will be forced to support it for real. >>14769 I have been hearing a variety of bad reports recently, I'm sorry to say. Whatever combination of Qt, mpv, python, X11/Wayland, and my shit code is causing a bunch of issues. Is XWayland a Wayland-like that runs under X11? I had understood X11 was generally well supported but Wayland was not, but now I'm getting reports all over the place. Maybe something important just updated somewhere, and it broke my mpv embedding? I recommend trying to run from source, as in my post above, but I can't promise much. Please let me know how you continue to get on, and if we discover an angle I can attack, I'll try and clean how I initialise or whatever is actually going wrong here.

>>14772 >I'm sorry that I forget if you are running from source or not.. That's fine, I don't think I've mentioned it, I do run from source already, but I haven't updated the venv in a while, these are the libs, I'll try to remember to include them in the future.

>>14768 Yea I know where to get hashes, I don't know how to make hydrus use them. Hash urls give "temporary redirect" http response code pointing to new alphanumeric session urls. Is it even possible for hydrus to save the first url and add it to image metadata together with new alphanumeric url?

Regarding dreaded "site logo" placeholder image files that some sites push instead of throwing http errors, I have an idea / feature request: Manually designate already downloaded file as "error" 1) Deassociate urls, tags etc, maybe all metadata entirely 2) Ask user whether to copy and/or redownload all previously known urls for that file A more complex method that should work on sites with static placeholder: 3) Store file hash in a separate category, (similar to deleted files?) 4) If same file is downloaded ever again, treat it as failed download Would that be sensible and feasible approach?

>>14776 Works on my machine.

>>14778 Oh that's great. Now I just need to figure out how parsers work...

Hello, just a reminder about that small feature I really wanted where you could excludes a set of namespaces from being counted when you add a "number of tags" search-term. You mentioned a few months ago that you could hack up something quick for me in the next few weeks or so. I don't need the UI to look good, just a way to do it would be helpful for my tag maintenance. I mentioned when you asked about a temporary UI that you could just reuse the namespace text box that's already in that window, and use hyphens before the namespace to say "exclude instead of include" and separate them with semicolons. And if a namespace contains a semicolon, use backslashes to escape it. It's hacky but something like that would probably be easy for you to implement UI wise, and it would work well enough for me.

>>14772 For the bad url search, how many is "normal"? I got some 7900 or so and I'm a little concerned about it so I cancelled it. Is there a way to check the urls by eye?

>>14781 well for comparison at least, I got about 260 bad urls in my 260,000 file database

>>14782 I see, if my math is right, that's about 0.1%. my db is much larger, at about 3.1 million. so that gives me about .25% percent with bad urls. Thanks for the data point.

I'm getting some weirdness with Japanese text here, it's not every picture, and it seems to be something with the title namespace. title:skeb〰 & title:skeb⛓️💙 as copied from hydrus. https://www.pixiv.net/en/artworks/116728128 https://www.pixiv.net/en/artworks/115026506

>>14779 Here are mine. Here's what each of them mean: alphanumeric urls: https://chan.sankakucomplex.com/posts/60rvLD5GZM3 en alphanumeric urls: https://chan.sankakucomplex.com/en/posts/60rvLD5GZM3 md5 urls: https://chan.sankakucomplex.com/posts/b427a29e4efbae0559946ff4ebf433b0 en md5 urls: https://chan.sankakucomplex.com/en/posts/b427a29e4efbae0559946ff4ebf433b0 legacy md5 urls: https://chan.sankakucomplex.com/post/show/b427a29e4efbae0559946ff4ebf433b0 Use whichever one your subscriptions already recognize so it won't think everything's a new url. Pick which kind of url you want under network > downloader components > manage url class links, and then sankaku chan gallery page.

I had a good week. The 'incremental tagger' system is working, allowing you to tag page:3, page:4, page:5 ... page:17 and similar to a selection of files, and .docx files are now importable. The release should be as normal tomorrow.

>>14786 >incremental tagger Can you append the number to some other text?

Feature idea: Let me set the outline of a file based on a tag/namespace such that rating:explicit has a red outline, rating:questionable has a blue outline. if namespace support was added you could have an artist be purple or something, obviously you'd be able to customize the colors. It would probably need a priority list to determine what tags override others. Alternatively, allow more customization on the little text display. My main idea is supporting shortening tags, eg rating:safe becomes r:s or something. Pic related isn't from my main Hydrus install, but on my main I have the rating show up as r:s,r:q, or r:e. Which I did by making rating:s my ratings on my local tag repo, but it sometimes still shows rating:safe if my local tags and the PTR aren't in agreement on the content rating. Also maybe support adding text on the bottom left side?

https://www.youtube.com/watch?v=PlvK2pabBqI windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v566/Hydrus.Network.566.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v566/Hydrus.Network.566.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v566/Hydrus.Network.566.-.macOS.-.App.dmg linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v566/Hydrus.Network.566.-.Linux.-.Executable.tar.zst I had a good week. The long-awaited incremental tagger is ready, and the program supports some more document types. You will get a yes/no on update, but it isn't a big deal. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html incremental tagger When you open the manage tags dialog on several thumbnails, it now has a '±' button that lets you tag the files 'page:1', 'page:2', 'page:3' and so on. You can set the namespace (or no namespace), the starting point (so you can start at 'page:18' if you need to), the 'step' (so you can count by +2, or even -1 to decrement), and even prefix/suffix for the number if you need to decorate with 'page:x (wip)' or something. I'm quite happy with how this worked out. There's some live text that gives you a preview of what's about to happen, so the best way to get to grips with it is to play with it. Just click and poke around, and let me know how you get on. Namespace will be remembered between opens, and if the first file in your selection has a number tag for that namespace, it'll set the 'start' position to that value. If you are setting page tags to a bunch of chapters, or gaps in a larger body, a bit of prep/overlap may help things here. We've wanted this for years and years, and while I'm not expecting the program to handle paged content beautifully now, this is a decent step forward. docx and friends The document types .docx, .xlsx, and .pptx, which are the newer Microsoft Office 'Open' formats, are now recognised by the client. These are secretly zips, so there's a chance you have some in your client already. On update, you'll be asked if you want to scan your client's existing zips to see if they were really one of these. You probably don't, so it isn't a big deal, but if you think you do, hit yes and they should appear. next week Cloudflare are rolling out some annoying new cache-optimising tech that is causing file dupes, so I need to work on URL handling so we can patch the most-affected sites. >>14787 Yes, prefix and suffix! Let me know how it works for you.

>>14790 >incremental tagger Awesome. Thanks.

>>14270 I really don't get the Parser Formulae tutorial... Is there a more step by step one available ?

>>14773 >Whatever combination of Qt, mpv, python, X11/Wayland, and my shit code is causing a bunch of issues. At this point it might be worth it to have a workaround option to treat video files as "open externally" only like Flash player files as it's better than having these dead mpv windows spawning every time. It might be worth someone's time to make a minimal python-mpv program that has the same bug to throw at whoever's fault it is upstream.

>>14793 You mean like this? >options -> media -> video

>>14790 >incremental tagger Fucken schweet.

Just got a >Hydrus grabbed over 100 files from a sub, either there's a fuck ton of new files, or there's been a url change If there was a url change, all these files would have the same hash as previous files and not need to go through duplicate processing, right?

>>14796 Yeah, but you will still have to download all of them to calculate the hash, unless the source page has file hashes somewhere and the downloader checks for them.

Is it a sorting issue with Hydrus or Hydrus Companion that causes files to be imported out of order when I use "send all tabs to the right to hydrus"? >>14797 Thanks. Seemed like that made the most sense.

thanks for adding the incremental tagger feature, been wanting that for a while! question, though: i usually sort my thumbnails from latest to earliest in the viewer, so a 'tag these files in reverse order' checkbox in the incremental tag dialog would be super useful. (some shortcuts for the dialog/action would be nice too!) also, any rough ETA on the parser rework/possible gallery-dl integration? i like to download a lot of art from twitter and the like, and seeing as nitter recently broke and gallery-dl/hydownloader are a pain to use compared to hydrus, it would be nice to have a built-in solution for that

>>14799 >i usually sort my thumbnails from latest to earliest in the viewer, so a 'tag these files in reverse order' checkbox in the incremental tag dialog would be super useful. (some shortcuts for the dialog/action would be nice too!) Couldn't you just reverse the sort from newest to oldest with two clicks before incrementally tagging? I find it's the most common sorting change I regularly make.

>>14762 >You are going to get a couple yes/no dialogs on update this week talking about deleting some mis-parsed URLs. If you do not manually store weird data in your 'known urls' store, just click yes. If you have lots of URLs, the work will take a couple of minutes. Can I check which files it were? It found 15 files and I picked no, because I would want to check manually myself. What would the search be like?

>>14801 I think I got it. I simply did ".http" in regex url search.

I was looking to set this up to import various collections, doujins, etc. Is there a way to save page orders within cbz files, booru pools, numeric file names within folders, etc? Are there options to search by names of collections, doujins, etc? I see options for scraping tags from boorus or even the public repository so even losing a doujin name isn't as big of a deal, but I don't want to re-order everything from scratch.

>>14803 You can use filenames as tags during import, which you can prefix with a namespace. So for example if your files are named like 01.png, 02.png etc., you can turn that into page:01, page:02... on import. There's also a counter that will tag your files in the order they are imported by a specified increment, which may be useful for files that don't have numbers as filenames, but you would have to probably import these folder by folder or maybe it applies to a selection in the ui. You can also import folder names as tags, either by using the same premade system like filenames, which lets you use 3 first and 3 last folder names, or you could use regex to format the entire path (including filename) into anything you want.

>>14804 Perfect. My current general folder structure for collections is >artist_name/comic_name/page_number.jpg So I can regex the path to import those as the following? >artist:artist_name >title:comic_name >page:page_number Or whatever the respective tags are?

>>14805 Yeah. If your folder structure keeps this exact pattern everywhere, then you don't even need to use regex, you could simply use the premade fields like in picrel. By the way, use "creator" instead of "artist", that seems to be the standard in Hydrus. Of course, regex can help you do extra shit, like ignoring leading zeroes for pages, if you don't want them and so on. Regex and the incremental counter are in the advanced tab.

https://hydrusnetwork.github.io/hydrus/server.html https://hydrusnetwork.github.io/hydrus/youDontWantTheServer.html I'm currently using Hydrus on my desktop, but want to migrate everything to a headless NAS on my same home network. I only need to view media from my phone, but from my desktop I'd like to import files and modify tags without saving a full database locally. Does the server instance support that, or do I just keep using the client instance from the NAS and remotely connect to that?

>>14774 Unfortunately, these libraries all look great! So, we can't easily blame this on 'update your venv m8'. Well, let's hope an upcoming version of Qt or whatever smooths some of this out. >>14777 Yeah, these files are tricky. I quite like the idea of having a special table that just lists, 'oh, this file is a login placeholder' or whatever, although what to do next is not so simple--I'm probably bumping it up to human eyeballs no matter what. That said, an important push I want to get to fairly soon in the downloader system is proper 'retry later' tech. For individual file imports and domains in general. If a site is overloaded, a download queue should detect that and try again later. If we had that sort of tech, we'd have more controls here and could maybe offer a nicer lever to the human that said 'try again in eight hours' or ' once you have logged in' or whatever. I think I'll have to keep this in mind. >>14780 Thank you for the reminder! >>14781 >>14782 >>14783 Yep these seem fine, I got something like 60k bad URLs on 3.5m total and I deleted them. >>14784 Thank you, I'll see what is going on! Looks like zalgo stuff, probably some crazy unicode characters messing up the coordinate calculations. >>14789 Hell yeah, I super want this. If you search previous /hydrus/ threads for I think 'Metadata Conditional', you'll see me talking about my general ideas here. I want an object that eats a file, examines its metadata, and spits out True/False, and then I can tie that into a dozen places all over the program, such as 'make the border red when X'. Then I won't have to hardcode all this logic, just the metadata conditional object and the priority queue of colour tests.

>>14792 Sorry, that's as detailed as things get I think! I can answer questions here if you like, or you can email me, or hit me up on discord so we can talk live on a Saturday, or if you'd like you can just write a little about what you found difficult/confusing with the language in the help document and I'll see if I can make things clearer. Also, if you can say more about your background with web tech like HTML/JSON and if you are at all ESL, that helps. >>14799 >>14800 I won't go more complicated than this, since I don't want the new tagger to get too many bells and whistles, but sure I will add a checkbox to say 'apply from last to first'. I can see how this would be annoying if you are by default set to that sort of file order. I was going to say 'if you feel clever you can set the step to -1', but then you'd have to fiddle around figuring out what your last number, the new starting position, is, and boring math shit is what computer code is for, so I'll do it. Let me know how it works. >>14801 >>14802 Yeah the regex I use in the update is: http\S+\s+http Since you only have 15, you can delete those manually, no worries. >>14803 >>14804 >>14805 WARNING FROM HYDEV: Although hydrus has 'page' tech, and I even did this incremental tagger thing last week, I don't really like how hydrus handles paged content. I've never been good at handling it, hydrus is optimised strongly for single media, so: please do play around with this tech on a couple of comics, but don't jump in with both feet and import a hundred comics and committing to hydrus 100%. As a new user particularly, you might find it a complete fucking mess, and I broadly recommend just staying with ComicRack or whatever, which will have nice bookmarks and all the stuff you want for actually consuming a comic. >>14807 Server does not support that. You want to set up a client on the NAS and then VNC or Client API into it. This will not be a beautiful way to run the client, so don't marry yourself to it before you try it. There's a Docker release of hydrus here: https://github.com/hydrusnetwork/hydrus/pkgs/container/hydrus if it helps, but I know almost nothing about Docker, so I can't say anything too clever about it. Let me know how you get on!

>>14789 >Alternatively, allow more customization on the little text display. My main idea is supporting shortening tags, eg rating:safe becomes r:s or something. Couldn't you theoretically set these short tags as parents to the longer tags and hide them from multi/single media view? Hidden tags will still show up on thumbnails I think.

Please add an "Unsaved changes, are you sure you want cancel?" dialog when clicking cancel on the manage url classes / manage parsers / etc. dialogs... I just modified a ton of url classes and then accidentally clicked cancel and now I have to do it all over again...

I noticed that some of my files from boorus had source twitter urls that weren't original size urls. So I made some file url classes for twitter image media. They cover 3 styles of image media urls: >normal https://pbs.twimg.com/media/Er6DNJ0XYAEhdoY?format=jpg&name=orig >alt https://pbs.twimg.com/media/Er6DNJ0XYAEhdoY.jpg?name=orig >old style https://pbs.twimg.com/media/Er6DNJ0XYAEhdoY.jpg:orig There are 3 url classes for when it's an "orig" url (original size) and 3 url classes that cover any non-orig urls that may be resizes. There are a lot of different non-orig urls. I've seen small, medium, large, 360x360, 900x900, and 4096x4096. Also, I noticed that the non-orig urls actually aren't always lower resolution. For example, https://pbs.twimg.com/media/Er6DNJ0XYAEhdoY?format=jpg&name=orig and https://pbs.twimg.com/media/Er6DNJ0XYAEhdoY?format=jpg&name=4096x4096 and https://pbs.twimg.com/media/Er6DNJ0XYAEhdoY?format=jpg&name=large are all the same file. It depends on the particular file. After I went through all my images with resized urls and redownloaded them as orig urls, only 15 out of 187 were actually higher resolution.

>>14762 >>on rule34.xxx and probably some other places, when the file has multiple source urls, the gelbooru-style parsers were pulling the urls in the format [ A, B, C, 'A B C' ], adding this weird extra string concatenation that is obviously invalid. I fixed the parsers so it won't happen again >>on update, you are going to get a couple of yes/no dialogs asking if you want to scan for and delete existing instances of these URLs. if you have a big client, it will take some time to do this scan. the yes/no dialogs will auto-yes after ten minutes, so if you are doing a headless update via docker or something, please be patient--it will go through Just to be clear: it will find files with 'A B C' and delete those, okay, but does it then add A, B, and C separately to the file's urls? I did a search with the regex "http\S+\s+http" from here >>14810 and I noticed that I have some files with the concatenated urls, but without the individual ones. As in just [ 'A B C' ], without [ A, B, C ].

>>14784 Hey, I looked at this just now and I'm sorry to say things all render good here. There are no hidden zalgo characters causing a coordinate shift on purpose, only the 〰, ⛓️, and 💙. These unicode emojis have had spotty (but increasing) support over the years, and in bad situations can make the 'fontMetrics' that calculate size and stuff sperg out, all depending on your OS/font/etc... Notice that when your system cannot find the characters (and gives you []), it calculates the size fine. I don't know the correct solution here, really. Qt is giving me the wrong size prediction, probably because the character mapping for 〰 and ⛓️ on your system has some whack coordinate space (its bounding box is probably ( -20, -20 ) to ( +40, +40 ) or whatever, when it should be ( 0, 0 ) to ( 20, 20 ) ). I am not sure if I can even 'fix' the alignment, since I think it is already aligning in the top-left corner (it is just the 'top' of the text is too high up). I also wondered if I could make a debug mode that replaces unicode characters with [] or something, although I can't obviously find a lookup table to determine emojis vs regular kanji or whatever. EDIT: Ok, ChatGPT to the rescue, check out this monster regex: emoji_pattern = re.compile("[" u"\U0001F600-\U0001F64F" # emoticons u"\U0001F300-\U0001F5FF" # symbols & pictographs u"\U0001F680-\U0001F6FF" # transport & map symbols u"\U0001F700-\U0001F77F" # alchemical symbols u"\U0001F780-\U0001F7FF" # Geometric Shapes Extended u"\U0001F800-\U0001F8FF" # Supplemental Arrows-C u"\U0001F900-\U0001F9FF" # Supplemental Symbols and Pictographs u"\U0001FA00-\U0001FA6F" # Chess Symbols u"\U0001FA70-\U0001FAFF" # Symbols and Pictographs Extended-A u"\U00002702-\U000027B0" # Dingbats "]+", flags=re.UNICODE) I'll add this as a DEBUG mode in 'tag presentation' or something. Please give it a go and let me know if it fixes the issue here, and also whether it adds ridiculous lag. >>14814 It will not add the separated URLs. The problem I fixed always had redundant URLs, but it seems hydownloader and some other places are producing this problem without the redundant URLs, and now we are seeing them. If the number is small here, I think you should click 'no' and fix them yourself. If the number is big, I guess let me know. I don't really want to do some ass-backwards retroactive update-update, or hack out some weird tool to fire this once-off, but maybe I should just suck it up and figure something out. >>14812 Sure, sorry for the trouble.

>>14822 Just a side/additional thing, this initial regex worked on the heart but not the chain because the chain was actually U+26D3 (chain) + U+FE0F (make the preceding shit an emoji). Let's see if these render differently: ⛓️ vs ⛓️. There's all kinds of bullshit 'variation selectors' in unicode these days apparently. I guess this is how the skin tone stuff works. The eternal phoneposter strikes again. Anyway, what do you know, in your tag banner there, the chain displayed, but not the heart. So I bet what is happening here is your local font engine can't do the unicode, but it thinks it can do the emoji version, but then it fails to figure out good coordinates for it. I've adjusted the regex to capture this, but I'll bet there's fifteen other 'slightly tan skin tone, with liver spots' 'selectors' to eventually capture, so let me know how you get on. Again, the probable ideal end state here is for your Linux/Qt/Font/whatever to update its font tables to handle symbols better, but I imagine it will be an endless goose chase until the whole wretched international standards edifice collapses under the weight of pregnant-male-doberman-fursuit-polycule-with-2043-pride-flag-coloured-fireworks-behind.

Is there a way to configure Hydrus separately to use a VPN or proxy service? I don't see any network options for this. The documentation implies I should use a VPN on my computer to route all internet traffic, but that would just compromise my privacy as I connect to personal email, bank, and work stuff. At the very least is there a recommended 3rd party downloader that does support proxies I can integrate with Hydrus? I'm not sure running "torsocks hydrus_client" is a good solution long-term.

>>14824 Aren't there options on some VPNs to route only the traffic of specific programs through specific nodes?

>>14824 network namespaces

>>14792 >I really don't get the Parser Formulae tutorial... >Is there a more step by step one available ? I don't get it either. That said, while researching the answer I stumbled onto a humongous parsing clusterfuck called REGEX and it doesn't look pretty at all. I'm afraid spoon feeding won't work as those formulae hardly can be explained, but at least they can be studied. F https://en.wikipedia.org/wiki/Regular_expression https://www.amazon.com/Introducing-Regular-Expressions-Step-Step/dp/1449392687/

Minor issue with the system search predicate parser: it lowercases your inputs. Case actually matters in some cases, like with the regex in pic related.

>>14822 >If the number is big, I guess let me know. About 3000 files come up from the regex search "http\S+\s+http". >I don't really want to do some ass-backwards retroactive update-update, or hack out some weird tool to fire this once-off, but maybe I should just suck it up and figure something out. I realized I could export the urls as sidecars, edit the sidecars with a script, and re-import to add the urls. I also used the script to check how many were actually missing the separate urls. Out of the 3000, only around 50 were missing the separate urls, so I don't think it's a big deal.

Situation: Running in KDE Plasma (5/6, Wayland, would expect the same situation with X11) with a global menu widget. Normally Qt applications will put their menu bars there instead of in the window. With a global menu, Hydrus doesn't create its own menu bar inside the window (correct) but doesn't populate the global menu (not correct). As a result anything in those menus is inaccessible. The workaround is to remove the widget, launch Hydrus, then re-enable it. Can I have the best of both worlds or does this need to be fixed in Hydrus itself?

>>14831 Screenshots?

>>14832 As picrel, but in the process of assembling the image I found Krita has the same issue on Plasma 6. Digging in a bit more it's a combination of Wayland issues, Plasma 6 issues, and the disjoint between Qt 5 and Plasma 6 (which uses Qt 6). Seems like it's not worth trying to address at this stage.

>>14835 >Plasma 6 and Qt6 Ah, the bleeding edge. I suppose you are using an unstable rolling distro and therefore bugs are expected. >Seems like it's not worth trying to address at this stage. Yup. A 'stable" distro may suit you better. Adopting the 'new' just because is new is sure recipe for trouble.

I had an ok week. I reworked some URL handling in ways that are mostly important behind the scenes and cleared some small jobs. It'll just be a simple release. The release should be as normal tomorrow.

>>14811 That's what I did on my desktop in a sense, they aren't hidden from view, they're just what "my tags" uses as safe/questionable/explicit, although it doesn't apply everywhere because I use the PTR so you often get duplicates like pic related.

After I updated past version 565, my rule34.xxx downloads don't get the "rating" tag anymore. I think the fix for the concatenated urls broke that. Luckily it can be easily fixed: I just went to the "gelbooru 0.2.5 file page parser", selected the "rating tags" content parser, exported it to clipboard, then went to the "gelbooru 0.2.0 file page parser", and clicked import from clipboard.

>>14837 The v567 release is cancelled! We found a problem and I need to do more work on it. I will move it on to next Wednesday, sorry for the inconvenience!

I had a bunch of borked urls, I'm sure most were dupes but I'm super paranoid about losing any. I was initially going to just document what commands I used, but I decided to go the extra mile and make a shell script, I used bash, find, ripgrep, wc, and sed. I'm not very familiar with bash so it was fun to learn the basics. Please please please read the script before just running it and I take no responsibility for any issues that might occur. The script doesn't have much error handling but I tried to make it as informative as possible. It's probably full of antipatterns or whatever but it's a one-time thing so I figured it was ok, I've already spent way too long on this. It should mention any important information. when you run it, but you should probably read the comments anyways. https://pastebin.com/MPAKgvGS

>>14840 Take your time anon. So far v566 is working fine. Thanks.

If I want Hydrus to still occasionally check "dead" subscriptions, is there any reason to not go into checker options and raise the dead sub limit from the default 180 days to something like 365 or 730 and set "never check slower than" to the same high number?

>>14720 is tbib still having this 302 redirect issue? i tried it for the link you gave and it still redirects, but i can't reproduce the issue for any of the other images i click on the site.

>>14838 Oh so it's the other way around. I don't use the PTR so I'm not sure how it works, but does it allow you to set personal parents/siblings for the PTR tag service or do any changes get merged? If you can, you could just put your personal short ratings as siblings to the long ones, so that yours get displayed instead of the long ones, event though they still exist and are searchable. Not sure if that affects the thumbnails though.

>>14836 On the subject of Plasma I found a minor bug but haven't checked if it can be reproduced in a clean environment yet. If you change colourscheme back and forth between BreezeClassic and BreezeDark the menu text / background doesn't change properly: plasma-apply-colorscheme BreezeClassic plasma-apply-colorscheme BreezeDark Restarting Hydrus will fix the colours. Other Qt applications don't have this issue but I'm not sure how they react to colourscheme changes properly. It's a very minor bug all things considered. Just something I found by accident because I have some software that changes the colourscheme during sunset/sunrise.

Wanted to download a few threads off 4chan archives (archiveofsins.com and archived.moe) using my custom downloader I made a while ago to get catboxed AI images, but they seem to have started using some cloudflare shit now and I'm getting 403s even if I copy cookies. How can they still block me if I use the cookies, even though browser works fine? Is there anything else I can do?

>>14848 Cloudflare turnstile is the devil. They use JavaScript and CORS checks on every page load now and do so very aggressively. Just to read some text and images you now have to prove you're a human. I wonder if one day Hydrus will have to communicate with a headless Chrome or Firefox over web driver in order to even be able to fetch anything. These can be detected too but can also be bypassed. It's a constant game of cat and mouse.

>>14849 Damn. I guess I'll have to return to just downloading the threads one by one as html files, bulk regex filtering the links I need and putting them as images into a new html file which I then save with the files.

>>14850 If you use the Hydrus Companion extension I think it can send links directly to Hydrus (the direct link to the image, not the HTML page). The only downside is it won't have URL Associations that way unless you manually add them.

>>14848 >Is there anything else I can do? Are you setting your User-Agent header as well?

It seems to be impossible to switch between tag manager tabs using the keyboard.

>>14851 Well that's kind of useless for entire threads. >>14853 Only the default one. I don't know how it's supposed to look like. >>14854 Ctrl+Tab works for me.

>>14855 >>>14854 > Ctrl+Tab works for me. Thanks!

>>14824 You can try the stuff in options->connection, which is all SOCKS4/5 iirc, but I'm dependant on what the python 'requests' library supports, which is not excellent. If you need per-program filtering, I think(?) most normal consumer VPN software will let you filter by app (it usually works on the actual executable path, just like Windows Firewall), so that's the best angle of attack, I think. Just as a side thing, I moved all my IRL personal stuff to an old ~$250 NUC PC + old monitor a year or two ago and it has been great. My browsing/hydrus/fun machine is all anon, my dev machine is hydrus, and all my 'real' stuff is isolated and only logs on to do email and Amazon etc.. a few times a week. No worries about accidental leaks or my normal browsing getting cookie'd-up with my IRL life, and things like VPN filtering is simple. And you obviously only need a simple office-tier PC. It would be a decent chunk of money if you bought it new, but if you have an old PC or laptop, I'd recommend thinking about it. >>14828 Yeah, I am sorry to say that if you are new to things like regex--and I presume things like HTML and javascript--my parsing stuff may be tricky to pick up. I'm still happy to answer any specific questions you have. By the way, I've always been pretty terrible at regex myself--I just don't have the right brain for it--but ChatGPT eats it for breakfast. You can just paste a regex and ask what it does, or describe the sort of regex you want, and it'll hold your hand as you figure it out together. In the last year or so I've become much better at regex just because I have that crutch available. >>14829 Thanks, this will be fixed in v567! Same for url class names. >>14831 >>14832 >>14835 Sorry for the trouble. The guy who figured this out to begin with intends to improve support. The 'fix', if it works for you, is to disable 'Use Native MenuBar (if available)' under options->gui, although by the sounds of it, that should already be disabled for you?!? Let me know if this is true: if you get this behaviour even with this setting off, that means we are still breaking things even when the 'global' code isn't supposed to run at all. Also, I hope to improve the command palette (ctrl+p by default) to show menubar items even with advanced mode off very soon, so we have a better emergency backstop here. >>14839 Thank you, I will update the defaults! Please note that if you edit a default, and I roll out an update, my changes will overwrite yours! If you make an edit to a default, remember to rename your edit, and ensure it is linked up to your rename in manage url class links. >>14841 Looks great, well done! As I've been saying about regex and other stuff recently, although it sounds cliche, ChatGPT can be a great tool to learn this stuff, or to just ask what the hell you are doing wrong in some place. I used to hate writing batch scripts because I would forget the damn syntax and not want to go through old docs trying to find the right escape character or whatever, but now it is much easier, and I have a bunch of things that ask me several questions and then fire off an 'encode my vidya capture' job to ffmpeg or whatever. I can just say 'hey, please write me a template that iterates through all the files in a folder called blah and does this to them', and the trudge-work is done. It makes mistakes, but if you know nothing about how to do some simple coding thing, it'll know a lot more than you.

I'm having computer problems and my dev machine blue-screened right at the end of this, and, praise the devs, 8chan saved my post. >>14842 Thanks for your patience. Good news is things seem fine. I'm using the current 567 code on my IRL client now and there don't seem to be any worries for anyone with a normal client, but it looks like certain advanced users who have some crazy URL Classes set up will run into some errors. No worries, I'll have it fixed for next Wed. >>14843 Yep, that sounds great. I do it manually, but whatever works for you. >>14844 I know nothing about it, I'm afraid. 13954530 still redirects me, too. >>14845 I can't quite parse your exact problem here, but yeah I think you can do what you want here. Hit up tags->manage where tag siblings and parents apply and you'll see how it generally works (warning: it is complicated). But if you want to sibling 'safe' on one service to 'rating:safe' or whatever using siblings on a different service, you can do it. >>14846 Thanks, interesting. Although it is lame of me to wipe my hands clean, I generally have no control over this--I just say 'hey Qt, set this stylesheet please!'--so when there are weird little per-widget bugs like this, I have to blame it on the window manager or Qt or whatever. We are hoping to have improved light/dark mode support in future, and I was also reading that the newer 6.7 version of Qt is going to have stylesheets that define both light/dark mode colours, if I understood it correct, and it'll change based on the OS mode changing. We'll see how it works out IRL. >>14848 >>14849 >>14850 >>14851 >>14853 >>14855 If you are copying cookies from hydrus companion, CF in its stricter modes needs the User-Agent to be the same as the one that got your cookies. Hydrus Companion supports this afaik, so poke around the options to see if you can send your User-Agent too. I know that has magically fixed some people in the past, but yeah unfortunately the internet is becoming less easy to access in many ways. The recent danbooru problems we had were overcome if you used HTTP 2.0 (we are on 1.1), so some of this will also be helped simply by updating our tech to newer libraries etc... (although in the danbooru example, that was likely an accident rather than by very purposeful design). Although, ha ha, I'm enough of a dinosaur that when we were talking about this, I was 'wow, do we really have to update? 2.0 is new!', and it turns out browsers are already using 3.0. The big corps are running away with the standards bodies, and everything is slowly pivoting towards OAuth-style tech. Oh well. The idea of perhaps one day using a headless browser as a web driver is not out of the question. We'll see how these things go. You'd think all the cheap hardware and bandwidth we have in the 2020s would allow things to be less restricted, but I guess that isn't how these guys think about things. That said, I do know that AI scraping has been hell for some sites (like danbooru) just recently. Speaking of, this general series of conversations has made me realise, I think, that we are about to get super-fucked by website filters in the coming years as AI agents start browsing at large scale. Remember that AI is going to make captcha obsolete, and when that happens, every tech that relies on it as an anti-bot filter is going to have to be replaced by an alternative, and all I can think of, to my own horror, is various sorts of Real ID. There will be efforts to launch nice open source or whatever versions of these things, but I fear the corporate bullshit versions will win, and then everything online will be government-tied iphones and fingerprints. Sounds like a good time to for some software that'll keep local copies of useful things safe, but how we will navigate the internet and download new content in a world of a million mumbai-based video-gen trainers and silky-voiced auto-scammers all thirsting for fresh content, I do not know. Fingers crossed it will be fun at least. >>14854 Also try hitting arrow up/down on an empty tag input. It should change services. Page Up/Down in the media viewer's tag manager will change the underlying media.

>>14858 >If you are copying cookies from hydrus companion, CF in its stricter modes needs the User-Agent to be the same as the one that got your cookies. I don't use hydrus companion as it doesn't support my browser apparently, but I tried copying cookies either by hand from the developer menu storage tab, or by downloading cookies.txt using some addon. I also tried setting up an http header user-agent by pasting what I got from https://www.whatismybrowser.com/detect/what-is-my-user-agent and nothing worked. >I can't quite parse your exact problem here, but yeah I think you can do what you want here. I don't really have a problem, I was just trying to give an advice to the other guy. Though I didn't know you could apply parent/sibling rules to other services. That sounds handy.

>>14858 >Yep, that sounds great. I do it manually, but whatever works for you. I changed the checker setting for subscriptions, then went and manually rechecked a dead one, and the dead subscription message still said "fewer than 1 file(s) in 180 days". Is it just a default message that matches the default settings, or is there some error and subscriptions are still dying at 180 days?

>>14858 >>>14854 >Also try hitting arrow up/down on an empty tag input. It should change services. Thanks. > Page Up/Down in the media viewer's tag manager will change the underlying media. That's useful, but not when the tag manager is covering the distinct part of the image!

>>14762 >options->sort/collect now has four different default tag sort widgets! You can set the default tag sort for search pages, the media viewer, and the manage tags dialogs launched off them. Noticed that the OR dialog doesn't have a way to sort tags or an option for it in the settings for that matter. Would it be hard to add?

>>14863 Wait, that's for the autocomplete, my bad. In that case, is there a way to group the results by namespace?

Does anyone have a way to download NSFW images from https://purplesmart.ai ? You log in using your Discord account, and something adds auth data to the image URL.

>>14865 >You log in using your Discord account Making downloaders is beyond me, but wow. Fuck that. Discord has a lot of privacy issues on it's own (i.e. no privacy at all), let alone Discord as a way to authenticate for porn. Maybe try https://aibooru.online/ as an alternative. Sadly purplesmart.ai looks to have a lot of good content, but that's way too high a price to pay for it.

Is there any reason the "pan file" shortcuts functions like a flight sim? Left and right will scroll a file left and right, but up and down are reversed and you have to manually change them for them to behave intuitively.

>>14866 The images at purplesmart.ai are generated using a bot on Discord.

>>14858 >The recent danbooru problems we had were overcome if you used HTTP 2.0 (we are on 1.1), so some of this will also be helped simply by updating our tech to newer libraries etc... Have you looked into using pyCURL? I don't know if it's a good fit for Hydrus but I really like that because you automatically get access to everything libCURL supports. It's a bit of a low-level library though but you can do everything Curl can. It's very fast too because it's a native C module unlike requests. >Want to make a simple request with alt-svc cache automatically upgrading the connection to HTTP 2 or 3 in the future? Well you can and you don't have to worry about the details. You only have to care about the pyCURL API to libcurl. I think replacing requests would be difficult though but maybe pycurl-requests could work (I don't know how mature that is): https://pypi.org/project/pycurl-requests/

>>14848 >>14849 >>14851 >>14853 >>14858 >>14859 I made it work. Turns out I also needed to open the api link (which I redirect to) in my browser, which triggers its own cloudflare check, after which I copied cookies again along with adding user-agent header.

I have a borked file here, feh loads it fine. It displays in the file upload dialog on here as well. https://danbooru.donmai.us/posts/4306140 https://gelbooru.com/index.php?id=5816146&page=post&s=view

>>14873 Me again, looking at Danbooru it's even tagged as "corrupted image" is Hydrus/PIL more of a stickler than other image loader libaries? Again, it works with feh, which uses imlib2 IIRC, and whatever Palemoon uses also works, so I'm not really sure what's going on here.

>>14857 >By the way, I've always been pretty terrible at regex myself--I just don't have the right brain for it--but ChatGPT eats it for breakfast. >that crutch available. That's a great idea. I can pester and abuse those canned algorithms non-stop. Let's see if I can learn something. Thanks anon.

I'm considering hosting Hydrus, I'd like to ask how easy it is to add text metadata to memes. I'd like to be able to search through memes by their text, is there a way to extend Hydrus with OCR? If not, can I at least add the text manually?

>>14876 Adding text metadata is easy, just use notes (default shortcut key is F2). As long as whatever OCR software you use lets you copy, you can always copy and paste the OCR plaintext results into the notes for the image. Unfortunately, there does not seem to be a way to search the actual contents of notes yet (or if there is it's not obvious to me). If notes were searchable they would be perfect for what you're looking for. Worst case scenario you can manually tag keywords from the dialog and/or the type of meme. Some memes are prevalent enough in the PTR to have already have a specific tag ("series:pogchamp" for example). I often use notes for translating and being able to search notes content would be pretty useful for me. Is there some way to do that I'm overlooking? If not, please consider searchable notes as a feature request Hydev.

I had a great week. I tightened up the URL storage/handling improvements that I was not confident in last week, so I am very happy to put out the release tomorrow. There are also advanced new tools for downloader makers for handling 'ephemeral token' parameters along with new quality of life UI in the manage URL Classes dialog. For normal users, there are also several bug fixes, file handling improvements, and a couple little things in system predicates, emoji tag presentation, and reversed tags in the new incremental tagger. The release should be as normal tomorrow. >>14860 I'm sorry to say it works on my machine, pic related. I couldn't find a forced 'no more than 180 days' rule, although you having said it maybe sparked a memory that I once had something like that, so I did give it a look. One thing is some of this text is kind of remembered from the previous calculation, so you sometimes have to force an update by open/applying the 'checker options' button, just to trigger a refresh. There's some asynchronous loading stuff happening in the background here, which makes some of the calculation update annoying to do on my end (the other row, the 'will recalculate when next fully loaded' bit in my pic is actually part of that). Another thing I did notice is the yyy time in 'xxx files in yyy' here might be the minimum of the maximum DEAD time and the oldest file in the subscription (I noticed when I first did a test here that it said like '1 file in 26 seconds', since it had just checked a new file), so if the oldest file in your sub has a source time of 180 days ago, that might explain it?! Let me know how you get on!

Is there a way to prioritize old urls over new links those urls generate in the url downloader? This is somewhat related to the archive thread downloader and cloudflare I talked about earlier. The archive cookies don't last very long and each thread generates 50+ links to another website (catbox), so it would make more sense to go through all the threads first to grab urls and then download the images. I think this would also work for images hosted in the archive as I could download raw image links without refreshing the cookie. >>14789 Color outlines (and thumbnail backgrounds) for rating services (favorites) would be nice too. Maybe even have the ability to put the service icon in a thumbnail corner like with the tags if it's set. Just now I was looking for new favorites, and thought it would be handy to have existing favorites highlighted just to see if I accidentally forgot to favorite a whole set of related images etc.

>>14878 >One thing is some of this text is kind of remembered from the previous calculation, so you sometimes have to force an update by open/applying the 'checker options' button, just to trigger a refresh This did it. The subscription options under downloading options appear to only apply to newly made subscriptions. Anything made before I have to manually edit the checker options on the manage subscriptions window. Thankfully, mass edits are allowed, so I just set everything to 730 days at once.

https://www.youtube.com/watch?v=PgqmqeH0iDs windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v568/Hydrus.Network.568.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v568/Hydrus.Network.568.-.Windows.-.Installer.exe macOS app: https://github.com/hydrusnetwork/hydrus/releases/download/v568/Hydrus.Network.568.-.macOS.-.App.dmg linux tar.zst: https://github.com/hydrusnetwork/hydrus/releases/download/v568/Hydrus.Network.568.-.Linux.-.Executable.tar.zst I had a good couple of weeks mostly figuring out better behind-the-scenes URL handling. Full changelog: https://hydrusnetwork.github.io/hydrus/changelog.html urls For normal users, tl;dr: URLs better now, you don't have to do anything. I made a dumb mistake when I first created the downloader engine in how I handle URLs behind the scenes, and today it is fixed. You don't have to do anything, and you shouldn't see any big changes, but in general, URLs and query texts with unusual characters should work better now. You may also notice a URL in the file log or search log will appear one way, e.g. with japanese kanji, but when you copy it to clipboard, it is all encoded to %EF kind of stuff--your browser address bar probably works the same way. It should just mean things copy between your hydrus and other programs with fewer hiccups. If you regularly use some very odd URLs or downloaders, there might be a hiccup or two. A file that relies of strange encoded characters in its known urls might redownload one time if a subscription hits it, or one extremely strange query text (it'd be a single tag query that has a % character somewhere in the middle) might need to be renamed to %25. If you have a crazy custom downloader that relies on %-encoding tech, please pause it before you update, and then do a test afterwards--it may be the hack you added for the old system is no longer needed. In any case, let me know how you get on, and if we run into a problem with a certain downloader, we'll fix it together. Overall, I'm hoping that working with encoded URLs exclusively will make more complicated downloaders easier to figure out in future, rather than having to fight with odd characters all the time. I decided to cancel this release last week because I wasn't confident on how I was handling the advanced edge cases of encoding here. I am happy I did, because while all the 'just download from a booru' style downloaders (like the defaults) were fine, the most experimental downloader situations (that e.g. needed encoded parameter keys) needed more work. If you are a downloader maker, then you will be in the guts of these changes, so please check out the changelog. There's new UI in 'manage url classes'--path components and parameters now have their own edit panels with linked text boxes that show the normal and encoded values of the stuff you are entering, and there's also new 'ephemeral token' tech that lets you decide in what ways undefined parameters should be clipped before the URL being sent to the server. The idea of the 'request URL' being different to the 'normalised URL' is broadly elevated and exposed across the program. other highlights I also added a 'tag in reverse' checkbox to the new 'manage tags' 'incremental tagger' thing. It adds tags like 5, 4, 3, 2, 1 instead of 1, 2, 3, 4, 5. And all new system:url predicates have new labels. They are all a bit simpler, and they should copy/paste into the system predicate parsing system. All existing system:url predicates still have their old labels, so if this is a big deal, you'll want to recreate them and re-save your session/search. Thanks to a user, the new docx, xlsx, and pptx file formats get some nicer thumbnails and a little metadata. It should all recalculate soon after update. The Client API is now more careful about which files you can undelete, and it also now lets you clear file deletion records. next week I want to put proper time into getting a 'future build' working. Last time I tried, I ran into some technical problems related to the newer libraries I wanted to bundle, so I'll see if I can fix it all and have a test release for people to try. Otherwise, I just want to clear out some small jobs that this URL work boshed.

>>14881 hey got an error updating from 566 to 568, instant crash on startup v568, 2024-03-27 19:40:59: boot error v568, 2024-03-27 19:40:59: A serious error occurred while trying to start the program. The error will be shown next in a window. More information may have been written to client.log. v568, 2024-03-27 19:41:05: boot error v568, 2024-03-27 19:41:05: Traceback (most recent call last): File "D:\hydrus\hydrus\client\ClientController.py", line 2219, in THREADBootEverything self.InitView() File "D:\hydrus\hydrus\client\ClientController.py", line 1320, in InitView self.CallBlockingToQt( self._splash, qt_code_style ) File "D:\hydrus\hydrus\client\ClientController.py", line 474, in CallBlockingToQt raise e File "D:\hydrus\hydrus\client\ClientController.py", line 420, in qt_code result = func( *args, **kwargs ) File "D:\hydrus\hydrus\client\ClientController.py", line 1285, in qt_code_style ClientGUIStyle.InitialiseDefaults() File "D:\hydrus\hydrus\client\gui\ClientGUIStyle.py", line 70, in InitialiseDefaults ORIGINAL_STYLE_NAME = QW.QApplication.instance().style().name() AttributeError: 'PySide2.QtWidgets.QCommonStyle' object has no attribute 'name'

>>14882 Thank you for this report, and sorry for the trouble! You are running from source in the older Qt 5 (PySide2), right? This slipped through testing. Since you are running from source, I just committed a fix to master, so please git pull again and see how it goes. It works here when I run on PyQt5, but I don't have a PySide2 environment right now, and it seems I can't easily get it for newer python, so please let me know how it works out. I'll make sure I update my testing regime to catch this stuff better in future.