Is there a good place to discuss tagging standards for the PTR? There has been plenty of talk on what tags and namespaces we should use, and I think there should be more. Maybe it's worth its own thread.

>>12638 Siblings are useful, but being able to right-click a tag in the manage tags window and select "replace tag with…" which lets you type in a tag to replace it with would be great. This would only affect the selected files that have that tag. I might still want to keep that tag on other files I haven't selected, so a sibling isn't helpful. I could manually remove the tag from the selection, but then I can only add the new tag to all of them.

i'm working on tagging approx. 18k files right now with rating tags, i.e. rating:explicit, rating:questionable, rating:safe. i'm trying to figure out if there's an effective way to do this automatically. I've set up obvious tags as parents for rating:explicit, so explicit is applied automatically with practically no problem. is there a way to say to the client "apply tag rating;questionable IF file does not have rating:explicit AND file has tags (tag1 OR tag2 OR tag3)" the reason I want to do this is because if i set a parent relationship for let's say tag breasts->rating:questionable, then i'm going to end up with a load of files with both rating:explicit and rating:questionable since the parent relationship does not check for the rating:explicit tag. also, mr dev man, i love your program. thanks a ton

>>12632 >sounds like a very subjective description of namespaces to me. Tagging is entirely subjective to begin with. Programmatically, namespaces are used for disambiguation to prevent naming collisions. What this means for tagging is that in practice you should have namespace_A:same_character_name and namespace_B:same_character_name to disambiguate two characters. People do shit like "character:Yuno (Mirai Nikki)" and "character:Yuno (Black Clover)", cluttering the tags with two variants of Yuno instead of providing a namespace for each of them (eg: "black clover:yuno" "mirai nikki:yuno"). This is probably done to avoid cluttering namespaces with is often its own namespace (series: or copyright:). In practice, almost nobody uses namespaces solely for disambiguation. They use it for meta information like "series:" or "artist:". >not sure what the difference to you is between having two namespaced tags "hair:short","hair:long" and having two regular tags "short hair","long hair". Either way you have two tags. I explained the difference. Information about the image (meta) vs information that describes what the image is (descriptors). This is also how every booru I know of works and how I choose to tag. Boorus will use the tags "black panties" and "black dress" and not namespace them under "black:" like "black:panties" and "black:dress". Mixing descriptors with meta information is messy as many things can be swapped with their namespace and still make sense. For example, which is more valid: "dress:black"/"dress color:black" or "black:dress"? Why would you namespace one and not the other? Would you use both namespaces? "hair:ponytail hair:ribbon hair:black hair:short" is clutter and even if you break each "hair:" down into more specialized namespaces like "hairstyle, hairaccessory, haircolor, hairlength" you only worsen the clutter. Compare: "ponytail ribbon black_hair short_hair" vs "hair_style:ponytail hair_accessory:ribbon hair_color:black hairlength:short"

>>12647 Character: Yuno Gasai is different to Character: Yuno, the Meal-Saving Prince. Clutter is a buzzword to excuse minute and less important tags from being tagged on images. A dress being black doesn't mean it's not in the style of Dior's iconic Little Black Dress, and so a Dress:Black and Black:Dress can be applied to the same image for the same garment and mean two completely different things. If a user believes that Hairstyle:, HairLength: or HairColour: tags are too specific, they can simply not use them. Hydrus allows certain namespaces that the user deems unimportant or uninteresting to be left out when viewing a certain tag or groups of tags. It's called Censoring. Hydrus does NOT allow for tags to appear on images where there are none. Therefore, it is the duty of any user of the PTR to overtag images and allow the user to pick and choose which tags or namespaces they see fit to read or discard according to their preferences. All of this relies on people actually using namespaces, though, which people still don't.

>>12650 >Character: Yuno Gasai is different to Character: Yuno, the Meal-Saving Prince. No shit, that's the point of differentiating them. I excluded last names for the sake of example but had a feeling you'd be insufferable enough to ignore the point. So here's a more solid example of two people with the same first and family names. Michelle Williams (All the Money in the World) and Michelle Williams (Destiny's Child) Both would be tagged "Michelle Williams". This is what namespaces are *meant* to resolve and where the name "namespace" comes from as a programming term. You'd provide a namespace for each Michelle Williams as a way to disambiguate between two otherwise identical names. >Clutter is a buzzword to excuse minute and less important tags from being tagged on images. If you noticed, I didn't remove any "less important tags". I simply didn't give them redundant namespaces that provide no additional value over a plain tag and only serve to add clutter the search terminology. >Therefore, it is the duty of any user of the PTR to overtag images and allow the user to pick and choose which tags or namespaces they see fit to read or discard according to their preferences. I don't use the PTR and am not speaking about public choose-your-own-tags-and-censor-stupid-tags group tagging. Most people are retarded when it comes to tagging things which is why once boorus grow past a point their tag quality goes downhill fast with very poorly or even misapplied tags.

>>12652 >which is more valid: "dress:black"/"dress color:black" or "black:dress"? If I wanted to namespace this tag (I don't), "dress:black" makes the most sense. "dress color" is unnecessarily specific, leading to more namespaces than needed. However, if I cared for searching by/adding/removing dress colours often, then I could more easily find all dress colours in a long list of tags if there was a "dress" namespace. This applies to a list of tags on an image, as well as search results for tags having entered only part of the query (e.g. just the namespace, or just the colour). This philosophy works well with Hydrus, and does not increase the number of tags. More namespaces are introduced, but only for what you are interested in. The only problem is that one has to add local siblings for all tags they wish to namespace, and possibly add the reverse siblings to the PTR if they want to easily mass upload tags. >had a feeling you'd be insufferable Now, now

>>12658 >If I wanted to namespace this tag (I don't), "dress:black" makes the most sense. "dress color" is unnecessarily specific, leading to more namespaces than needed. Not saying you wanted to use these namespaces - but I was pointing out that black:dress and dress:black are both equally valid ways of tagging the same thing using namespaces. "dress color:black" was just used as a more verbose "dress:black", the comparison was more between "dress:black" and "black:dress". Further examples of these namespaces, you might have "dress:wedding" and "black:shoes". >However, if I cared for searching by/adding/removing dress colours often, then I could more easily find all dress colours in a long list of tags if there was a "dress" namespace. I don't believe this is a common enough scenario or a significant enough increase in quality of life from scanning an alphabetical listing of tags for "color_dress". It takes maybe an additional second, if even that, to see if an image is tagged "red_dress blue_dress black_dress" instead of "dress:black dress:blue dress:red" even on my images with ~70 tags. It provides that small amount of QOL with the tradeoff of either increasing the complexity of the tagging system (rules about what belongs in the "dress:" namespace) or increased verbosity or redundancy when searching. Complexity: "hair:long black_hair ponytail ribbon (Complexity is defined as having to clarify "Which values belongs in the 'hair:' namespace?") Redundancy/Clutter: "hair:long hair:black hair:ponytail hair:ribbon" (Reduced complexity, "All hair related values belong in the hair namespace") Verbosity: "hairlength:long haircolor:black hairstyle:ponytail accessory:ribbon" (Removed redundancy and reduced complexity by being more verbose) No complexity, no redundancy, no verbosity: "long_hair black_hair ponytail ribbon" >…as well as search results for tags having entered only part of the query (e.g. just the namespace, or just the colour). "hair:" returns "hair:long" and "hair:black" (and all the other length/color in the hair: namespace) and "*_hair" returns "long_hair" and "black_hair" (and all the other length/colors tagged as adjective_hair).

>>12643 >Maybe it's worth its own thread. Sounds good to me, it's a bit messy to mix it in with Q&A

How do you get Hydrus to download Explicit tags on rule34hentai where you need an account to see? The Hydrus login doesn't do anything and Hydrus still skips these when downloading from that site.

How do do you undo an imported downloader from an png? I tried importing a downloader from an png and I don't really like how its getting tags so I want to go back to its default settings.

>>12667 Is namespaces the only way to sort tags? Is it possible to add them to sorting list to have them presented in a way that looks logical to humans? If it was possible and if coloring a tag was moved from namespaces to these list, I don't think we would need so many namespace wich essentally multiply the tag in the database. dog and species dog both exist. This may be stupid, though, I'm no programmer.

>>12668 These are the tag sort options. Currently there are no ways to manually give sequencing or priority to individual tags. Also only namespaces can be colored. There is not currently a way to override "unnamespaced tags" to be colored on a tag level, to my knowledge, but it is probably something that is capable of being added. May I ask why you feel that dog and species:dog are redundant tags? Personally I use "dog" for any actual dogs and "species:dog" for anthropomorphic characters. For example, I don't have a "catgirl" tag, I use "species:cat" while a pet cat would simply be "cat".

Has the Linux version of Hydrus gotten any more stable? How do I migrate my DB from windon't to linux? Will it ever be as stable, or will this always be a primarily windows-focused application?

>>12673 Im running it from source, so far only once hydrus crashed. >How do I migrate my DB from windon't to linux? Same as in Windows. >Will it ever be as stable, or will this always be a primarily windows-focused application? idk seems pretty stable to me.

My database sees only 21 images out of few 1000. Even though they are there. Everything else is considered orphans. Am I fucked?

>>12674 That is nice to hear, last time I tried it on ubuntu it would crash constantly. Maybe there was an error in my setup. I'll try it soon! Do you think there is any way to easily remove the more smutty compnents of the program (explicit references to hentaihaven/gelbooru etc.) becuase I want to use it in a way where it'll be relevant to cataloguing photos of family and friends? (i.e.: father, mother, brothers will see the program).

How come the installer fails everytime I try to download it?

>>12671 I used that example because the tag "dog" is aliased to "species:dog" in the ptr. I too would prefer it not to be a defacto alias. I wager you either don't use the ptr or found a way to blacklist alias locally? I wonder if the way we use namespace is necessitated by boorus compatibility rather than being the most efficient way to sort tags. Using a list system would however requires more user involvement though, which doesn't seems a good idea for such numbers of files. In the same vein I wonder if affix and suffix are possible in lieu of those qualifier, by which I means linking up to tags togethers. "character name" is affixed to "series it come from". It's 100% not doable now but may be technically possible, perhaps.

I'm getting a filenotfound error even though the file exists on my system, not sure why this is happening? db diagnostics say nothing is wrong I created the db on MacOS and have since imported to Windows, exact same db works on mac but fails on windows… did I miss something during my migration? (db was fully portable)

>>12688 I don't use the PTR because people tag like retards, so you're right. :) >I wonder if the way we use namespace is necessitated by boorus compatibility rather than being the most efficient way to sort tags As I explained somewhere, namespaces are used to differentiate two things that have the same name. It comes from programming, especially object orientated programming, where you'll have many objects that may share the same name but aren't the same thing. For example "second" and "second". "time:second" and "position:second" provides disambiguation. ( See: https://stackoverflow.com/questions/4211827/why-and-how-should-i-use-namespaces-in-c ) People don't use namespaces properly and end up with "(clarifications)" in their tags which, in my opinion, is quite ugly. Because disambiguation is rarely needed, it's boorus often use it for *meta* information like who drew the image or what the image is a drawing of rather than what it contains. >"character name" is affixed to "series it come from". This is already double by making use of child-parent relationships and if you don't tag like a retard. Unless I'm misunderstanding you. :)

I'm still kinda new to all this, but im wondering why theres a image limit on the twitter downloader? It only seems to stop at a point. Also will it ever be possible to download gifs and videos off from twitter?

Has anyone ever noticed booru subscriptions consistently failing to pull certain tags? I am suspecting that any tags which are replaced by a local sibling are not getting added to the image at all. For me, this is a Sankaku subscription. I'll look into it a bit more.

Where can I find other downloaders? Or are all of them already installed?

>>12684 >Do you think there is any way to easily remove the more smutty compnents of the program At the very least you can hide them from view from network -> downloaders -> manage downloader and url display. I think you may be able to entirely remove references to them through network -> downloader definitions and clicking on each submenu and deleting everything in there.

Hydrus has every feature I need, but the major deal breaker was that it wants to copy everything to it's own folder, when I rather it, like music libraries + players, just to monitor a list of directories. Is this still the case? If so, can there be changes where copying "imported" files to its own directory is entirely optional?

>>12731 >If so, can there be changes where copying "imported" files to its own directory is entirely optional? Yes, but that would fuck with the entire program. Hydrus depends on files being in it's own folders in order for half of its features to work. There might be some workaround in the future but no plans yet AFAIK.

>>12732 The only thing I really want is some aggregate, search, catalog, tagging features. Should I be looking for another (simpler) program?

Not quite sure how to ask this, but is there a way to purge all PTR data from my db?

>>12684 >That is nice to hear, last time I tried it on ubuntu it would crash constantly Sorry for late reply, DO NOT USE THE PORTABLE VERSION, RUN IT FROM SOURCE.

I could swear I asked this already but maybe it didn't send as I can't find it. Following scenario: One PC acts as a local server, nine others use that local connection to access files on the server. Would it be possible to put the DB on the server and access it from all other PCs? What happens if several clients try to change things?

>>12734 have you deleted the ptr from the services menu?

New to Hydrus, feeling like a proper retard. The public tag repo's account referenced here: https://hydrusnetwork.github.io/hydrus/help/access_keys.html seems to be maxed out. An alternative way to get tags imported seems to be to import a tag archive, BUT it looks that functionality has been removed in the current version? So how do I get tags? I've searched all around and I can't find a resource explaining how to get tags with the current version.

How should I tag variations/revisions of a illustration? I found using a combination of page and chapter (what should the chapter be even called?) with collections produces the desired effect, but it seems like a terrible solution.

>>12740 Just going to have to wait it out. The public tag repo is unaccessable until the bandwidth limit resets at start of next month. In the meantime check out https://cuddlebear92.github.io/Quicksync/#two and use the services -> import repository update files menu to get a head start on updates.

>>12739 That was the first thing I tried, but it didn't seem to do anything. I somehow got client.mappings.db from 15GB to 3.5MB (I think I exported my local tags, deleted client.mappings.db, and re-imported my tags), but client.master.db only dropped from 4GB to 3.5GB after vacuuming

>>12740 Shit's fucked. Like >>12743 said, you can only wait. >tfw the ever growing tags to petition only makes me more afraid to do one big fuck up like accidentally petitioning all tags to the PTR again and having to erase it all

>>12740 Use >>1914 or do the individual file lookup. The problem with that is if the hash doesn't match then you'll get nothing. That's why I run everything through IQDB and use the script to collect it all for import

>>12737 >What happens if several clients try to change things? I'm not sure which thread it was in (it was sometime within the last month or two) but I'm pretty sure Hydrus Dev said not to do this. Something about SQL not being able to handle multiple programs trying to lock/unlock the db.

>>12644 Nice idea, thank you. I know what you are getting at with the current workflow not being able to do this for a multi-file selection. I think I can do this, but I can't promise it super quickly.

>>12646 Great, I am glad you like it! Best way to do that for now is with a search page that you hit F5, ctrl+a, F3, ctrl+v, enter, enter, ctrl+r on every week. A search like: [ (semi-lewd tags), -rating:explicit, -rating:questionable ] Then hit F5 to refresh and see all the files that apply. ctrl+a, f3 to bring up manage tags. ctrl+v, enter to add 'rating:questionable', then enter to close manage tags and ctrl+r to remove all selected files from view. There isn't a way to do kind of 'logical' tag siblings/parents that rely on more complicated tag states and implications. I have thought about it several times in a 'wouldn't it be nice to add tag x if both tag y and z were present' way, but that is a lot of power and I can see how it could go extremely wrong if I did not wrap it behind two advanced modes. So I am a little wary of it atm and am ok that human eyes go over complicated jobs like this for now. rating is a very subjective namespace. I agree that explicit is going to be easier to set up than questionable or safe.

>>12666 I have heard the R34H login script can be finicky, just failing sometimes. If you are absolutely certain you have the credentials set up correct in manage logins dialog, and that the login script is active, try hitting the 'reset login' and 'do login now' buttons on the same dialog, just to retry.

>>12667 Hit the manage dialogs under network->downloader definitions menu and delete the new objects. You are probably talking one GUG, a couple of url classes, and a couple of parsers. If they overwrote old objects, those dialogs should have 'load from default' buttons or similar. Once you are all set up, hit the manage url class links dialog and make sure the 'url classes' you want are connected to correct-sounding parsers. You can click the 'try to link…' button on that dialog to try to auto-fix the links here.

>>12668 I would like future versions of the tagging system to support metadata about tags to permit custom sort orders. Future tag siblings will allow you to better control your various dog/species:dog preferences locally just to your client.

>>12673 If I could wave a magic wand, I'd love for the Linux version to be more stable. I'll be putting some time into improving multiplat stability in about three or four months. Unfortunately I am a Windows guy so just have less experience and time spent on different Linux flavours and window managers. The stability problems on Linux are mostly to do with my shit UI code interacting with stricter/different window managers that want things in slightly more proper ways. I'll keep pushing on the problem, but I can't promise excellence for all systems in quick time. That said, some Linux users experience no problems at all with source or the built release. I recommend you try it, see if your situation is good enough for you, and plan to be ok with staying on Windows or whatever. To migrate your db, the basic rule is just to install on Linux and then copy your install_dir/db folder across. Don't do this if you are uncertain–feel free to ask me for more details if you need help. Make a backup beforehand. Background reading here: https://hydrusnetwork.github.io/hydrus/help/database_migration.html

>>12676 This sounds like a serious problem. Please check the 'help my db is broke.txt' document in install_dir/db. If the files are legit still in your client_files folder, but just seem to have disappeared from your client's records, there are ways to recover. I am happy to help with all of this, but first you need to make sure your hard drive is healthy, which that document will walk you through. Feel free to send me an email if you want to work more one-on-one with this.

>>12684 >>12727 Yeah, delete the stuff from downloader definitions. Bear in mind this is imageboard-centric software. Anyone who googles 'hydrus network' is going to end up here pretty quick, so I recommend you sit in the cockpit and just drive some slideshows if you don't want to be asked questions about the INTERNET HATE MACHINE.

>>12687 Have you got an anti-virus that doesn't like something in it? This has happened a couple of times before. The 'upnpc' executable file in install_dir/bin are common culprits for false positive detections, and I know some scanners can see inside the archive of the installer exe.

>>12689 Can you say a bit more about your migration situation? If you open up database->db migration, does everything appear to be in the right place there? What's the first two hexadecimal characters of the file's hash, and is it in the right 'fXX' subfolder for that? If the client gives you path details in those FileNotFound errors, do they line up with what you are looking at, or are they old/incorrect to what you expect? Also, if this is a video, is the actual error about not being able to find ffmpeg, which hydrus uses to render video? Do any videos work? What is ffmpeg's apparent version under help->about?

>>12696 There are two limits with twitter–one is a ~2,800 tweet limit caused by their API, and another is if the user has a series of 20 or so tweets without media content, the downloader bails early. Twitter videos is tough. twitter deliver videos as streamed mp4s, so no easy URLs for hydrus to latch on to. I know one user is working on this problem. Apparently twitter can give you direct vid URLs from the API, but it takes an API key.

>>12700 Can you check your options->tags for 'Apply all siblings to all services'? If it is checked, would unchecking it help? I expect to extend this system in future so you can control which services can apply to which others, but for now it is all or nothing.

>>12721 Lots here: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System Check the 'all-in-ones' folder for easy-import pngs. Bear in mind the quality of these downloaders varies a lot. Test them a bit to make sure you only get the tags you want, and don't import a thousand all at once or you'll fuck up your UI with gigantic lists of shit you don't care about.

>>12731 Here are my current feelings on 'external file' management: https://hydrusnetwork.github.io/hydrus/help/faq.html#external_files It is technically possible, and if it were a simpler problem all around I'd do it, but I just don't think I can put the maintenance time into this sort of system. If you only want easy-external access to a few thousand files, you may be able to cobble together what you want with an 'export folder' that maintains a synced folder of a search and gives the files nice filenames based on tags.

>>12806 Nice. Is there a list of all boorus that allow md5 or some other type of md5 search?

>>12733 ACDSee is good. If you only want to catalogue a few thousand files and external management is critical for you, that might be a better fit.

>>12734 >>12739 >>12744 Unfortunately, my current 'clear orphans' tech doesn't do master definition recycling. I plan to add this, so it will go down eventually, but we just aren't there yet.

>>12737 >>12750 Yeah, there aren't great solutions yet for a single client accessed from multiple locations. I do not recommend you run a client.db from a network location, as the locks the network drive provides are usually fake and not good enough for what SQLite needs to be safe. Running multiple clients at once from multiple network locations will cause your db damage. The Client API is likely to open things up here. I know users who are working on phone/tablet/web browser wrappers that present you booru-like interfaces.

>>12740 Yeah, sorry, PTR bandwidth is tight atm. It is now free again but will likely block again for the last week of the month. I have a plan to fix this in 2-3 months. You can just leave it to sync and it will catch up when it can.

>>12809 I don't know. Most boorus share the same software, so if you know Gelbooru 0.2.0 supports it in some way, any booru that runs that should be good. Doing a search for '(booru) advanced search' or '(booru) cheatsheet' can usually get you to a nice page like this: https://gelbooru.com/index.php?page=help&topic=cheatsheet https://danbooru.donmai.us/wiki_pages/43049 https://e621.net/help/show/cheatsheet https://konachan.com/help/cheatsheet So looks like gelb 0.2.5, danb, e621, and moebooru are good with a flat 'md5:hash' search. I'd bet Gelb 0.2.0 can do it. I am not sure about Gelb 0.1.1.

>>12809 Check out the last few posts from the Hydrus Scripts thread: >>12604 >>12631

Does gallery downloaders check for newly added posts periodically or do I have to create a new query to fetch new posts?

>>12822 The downloader page just does one job once. If you were to manually create a search for the same query you put in before, it would start a new queue starting at the current new beginning. Obviously you do not want to do this yourself every week or so, so if you want to have a downloader that periodically re-checks and keeps up with a particular query for a site, check out 'subscriptions': https://hydrusnetwork.github.io/hydrus/help/getting_started_subscriptions.html PROTIP: Don't go fucking nuts with this. Start slow and figure out a workflow that won't nuke your available bandwidth and hard drive space.

>>12641 I suppose this happened because of a disconnect, but what do these "lg"s mean? Some files got corrupted on download? Can I check this somehow or should I do something else to remove "lg"s?

>>12829 Forgot to mention, It's a danbooru/gelbooru subscription.

>>12814 Nice, thanks. >>12816 I already use all of those. I want more tags.

>>12829 Ig means ignored, so 27Ig means 27 ignored URLs.

>>12833 Got it, thanks.

Is there a way to search only images with certain namespaces other than putting a wildcard like "character:*"? It's very slow to search with wildcards. Using only "character:" does not work.

>>12834 Ignored tends to be either 404 or tag blacklisting (say, the file had 'scat', and that is in your blacklist). If you zoom into the individual sub query and order its big file import list by status, you can review the ignored reason in the 'notes' column.

>>12835 Type in 'character:' in the autocomplete entry and wait a moment, and a new special tag should appear in the dropdown list called 'character:*anything*'. This is a special tag you cannot type in manually that uses cleverer db code to speed up the namespace search.

>>12837 Nice! Any reason why "character:" doesn't default to this?

>>12839 That's a good idea, thank you. I think it is probably reasonable to say that 'text:' is not a tag people will typically want to attach to anything and search for regularly, so I'll swap in the 'text:*anything*' version whenever this is added.

>>12803 database -> db migration said everything was in the right place, all the files existed on my system and were in the correct subfolders. Client wouldn't cite a path in the errors, not sure why. ffmpeg is also installed locally + system wide. I fixed the issue by importing a new file; not sure what this changes but after that I didn't get any more errors. not really sure how to reproduce this, sorry. for posterity: what i did was move a fully portable ~45gb database from mac to windows, using resilio sync (home version)

Is there a way to search for files that have mutiple tags in the same namespace? Main scenario i am thinking is searching for files that have mutiple 'series:' tags for crossover images but dont have a 'crossover' tag.

i have two files that i want them to appear as a group when i select sort by collection what do i do?

Will using socks5 with your application result in the DNS lookup also being routed through the proxy? E.g. querying safebooru will be resolved through the proxy setup in the settings?

Is there a pane with tags that I can click to add to an image? I hate having to right click>manage>file's tags>type tag name in just to add tags to images in a page. It isn't much better in the media viewer either from what I can tell. It's super inconvenient.

How would I go about recording a file's original creation/modified time as a tag? Maybe a script to record it in a text file? Like the ones the booru tag parser makes.

>>12855 Potentially answering my own question: Looking into the source code on page https://github.com/hydrusnetwork/hydrus/blob/master/include/ClientNetworkingSessions.py line 126, it appears you merely set the proxies to be the necessary value. Presuming a sufficiently up-to-date version of Python is installed with requests[socks], just making sure to use socks5h in the manage_options->connections should do it. In short, the user does have power over this. Consequently, one can easily run hydrus over TOR. I did notice socks5 being quicker than 5a, presumably because naming resolution can be slow. In addition, since you'd be using the proxy's DNS, you can't really cache results. It'd be interesting if you could use your cached DNS results from the host machine, but for new lookups use the proxy. However, that's a separate thing all together. tl;dr you can easily proxy over tor with hydrus. If you could confirm BO, I'd be grateful.

Quick question, should I be seeing saved webms as zips of images in Hydrus? Sort of like how you can unzip an epub and get a bunch of HTML pages. Or, should the saved webms be *.webms and viewable in Hydrus? This has probably been answered at some point or another, so, if so, sorry about that.

This could either go in the bugs or this thread: Does the Hentai Foundry downloader support fetching tags? I can't get it to work with subscriptions, nor with download tabs.

>>12852 Hmm, I am not sure what this was. I wonder perhaps if the OS X legacy db info had slashes in some path instead of backslashes, and one part of the files manager was unhappy with that? It is possible that by looking at your current paths and ok-ing the dialog put those unusual paths through a normalisation routine that cleaned them up for Windows, but it could also be something else. If you go to install_dir/db and check the relevant 'client - 2019-xx.log' file and scroll to the error time (I think you'll be able to ctrl+f for 'missing' or similar to find it as well) and check the exact path that was logged in the error, is there anything funny about it? Maybe like 'E:\my files/ab\abcde….jpg'? If you have the whole error traceback, that may be helpful to me, but if so make sure you either email it to me directly or redact the path and file hash if you post it publicly. I'll keep this problem in mind. If you aren't getting it any more, even on the same file, it sounds like it fixed itself. Please let me know if you start getting it again.

>>12853 Not yet. I'd like to add an option namespace component to system:num_tags.

>>12854 Collection works by namespaces right now. If the files have a shared 'title:' tag, you can collect by 'title' namespace and that'll put them together. If they have page numbers, you can sort by page to get them in the right order. I overall do not like how paged-content is handled in the client. Page tags are clunky to work with, and grouping many sorts of comic and mini-comic is a pain. Adding cbz support and the associated hydrus database backend to deal with file groups/pools with indices is probably the silver bullet here.

>>12855 >>12859 I know very little about proxies, so I cannot comment very intelligently. I just pass whatever you put in those input fields to requests, and it handles the rest. A bit of searching lead me to this: https://stackoverflow.com/questions/12601316/how-to-make-python-requests-work-via-socks-proxy and this: https://github.com/urllib3/urllib3/blob/a019c5e9e8ac461111552fdf9aa97899977cccbf/urllib3/contrib/socks.py#L141-L166 So yeah, it seems like using 'socks5h://' does DNS through the proxy, but 'socks5://' does not. I'll update the UI on that panel to suggest 'socks4a' and 'socks5h' as well.

>>12856 There are a few tag 'suggestion' columns you can turn on for the manage tags dialog under options->tag suggestions. Favourites/related/recent can be useful depending on your workflow and can be double-clicked (and in the 355 I just put, you can keyboard-shortcut focus to these) rather than having to type every time. The manage tags dialog that opens from the media viewer is actually a window not a dialog (i.e. it does not prohibit interaction with the parent window while it is open), so you don't have to close and open it again to edit other media. Also, by default, hitting page up/down on an empty tag autocomplete input should send a previous/next command to the media viewer parent, so you don't even have to change focus to tag and move on. That said, I would like to add some quicker 'remove'-style actions to the normal taglists when looking at thumbs or the media viewer. Something like right-click->remove this tag, which could spawn a little yes/no dialog that would handle the more complicated back-end for you without having to make the whole manage tags dialog.

>>12857 I have a long-time job to cache this information and make it searchable and sortable. I know another user wants this very much, so it will eventually happen. If you do not want to wait, yeah I think a script would do the job best. Do a stat call on each file and then output to a (hash).jpg.txt file that you could reimport with 'parse neighbouring .txt files' checked. You'd have to reimport these beside your files, which might make for a pain in the ass workflow though. You might be able to wangle it by reimporting from your actual client_files directory, with the .txt files embedded beside, but that is probably asking for trouble. If your collection is relatively small, maybe something like: system:everything export (copy) everything to a separate folder generate your .txts reimport that folder, setting to delete original files. It'll crunch that import pretty quick, since it already has all those files. The files will get your 'creation-date:2015-02-01' whatever. But that would also be a pain to keep up with. I'll add this eventually (another thing the new file mainteance manager will be good to retroactively fill in) and handle it all automatically, so if you can wait, I'd advise that.

>>12869 Are those from danbooru? They are probably ugoiras imported from Pixiv, which danbooru offers both in native zip format and a converted webm. If you would rather the webm, please hit network->downloader definitions->manage url class links and switch the 'danbooru file page' entry to 'danbooru file page parser - get webm ugoira'. If you want to reimport the zips you already have as webm, do a search for 'system:has url:danbooru file page' and 'filetype:zip', and then ctrl+a and right-click copy danbooru file page urls, and then queue those up in a url download page with the 'file import options' set to 'do not skip downloading because of known urls'. Native ugoira support remains a long-term hydrus goal. If you are getting weird zips from somewhere else, please let me know. If the zip contains a list of jpegs or pngs, it is probably an ugoira.

>>12875 It only does 'creator' and 'title' tags. The first HF parser I made grabbed the artist tags, but they ultimately proved too low quality to keep. Too much #i_feel_good_today #lewdtuesday and #the #party #is #in #my #pants bullshit unfortunately. Alternately, are you not getting the creator and title tags? I just did a test here and it seemed fine, but let me know if you aren't getting anything at all.

>>12884 Yeah, I just wanted someone else besides me to try it out. I believe it works for me, and based off of function definitions and descriptions, I'm pretty sure it's acting as expected. However, it's not like I've done some absolute test like using Wireshark to see that no DNS lookups are being done locally, but, rather, are all being done through port my proxy lives on. >>12887 The website is ATFbooru. I created my own parser/gallary downloaders ect. because I didn't see, at a glance, an existing one that suited my needs. It works perfectly for images–including gifs–however, despite the data being in .webm format when I go to the page in my browser, the download is a .zip. As you said, the zip contains a list of of images, therefore, it's probably that thing you called an "ugoiras". I can look into the 'danbooru file page parser - get webm ugoira' downloader and add it to my own script. However, I do want to be certain, the webm conversion wont lose quality right? They're equivalent? I ask as a comparison between differing video compressions (e.g. Xvid yields a loss in quality when compared to H264).

>>12893 That's the tricky thing with ugoiras–they are a pretty weird format, a bunch of images rendered by javascript according to variable framerate. There is no excellent lossless format to convert them to. The sites you are getting webm conversions from may be using some clever encoding options to preserve lossless and frame timings, but they may not. Webm has a lossless mode, but this is not typical–the filesizes can get pretty gigantic. The problem is aggravated by most artists putting jpegs in their Ugoiras rather than nice pngs, which can look beautiful in an ugoira when presented right, so you are already taking the compression hit from jpeg. Ultimately, if I can write a native hydrus ugoira detector and renderer (although we'll have to figure out how to attach the JSON frame timings, which aren't guaranteed to be in the zip), then we can just keep the original files. Or figure out the right params for lossless variable frame rate conversion using ffmpeg and do that webm/whatever job 'properly' ourselves. Whole thing is a bit of a shitshow with no ideal solution at current. If you want to copy the danbooru parser, I think it basically detects the webm embed with a higher URL priority than the normal file download link. Then, when the page provides zip on the download link and webm embed, it goes for the embed, and when there is a jpeg or whatever in both, it won't find the video embed and falls back to the non-sample-guaranteed download link.

>>12886 Well it's not really that tedious….if I knew how to script. I thought once Hydrus imported the files, it gave them a new creation date matching their import time though

>>12894 >filesizes pretty gigantic not too important. >sites use clever encodes ect. >artists putting jps in their Ugoiras This is where my lack of knowledge of pixiv comes into play, but what is it that exists on the artist's computer that is uploaded? Does the artist upload a zip of images and then pixiv determines how to render it? If so, then the zip is the "loseless" original since that's exactly what was on the artist's PC. As a completely separate aside it's a tad sad when you notice something, but it doesn't work out. E.g. there was an attribute field which held all the tags for the image, delimited by spaces. Consequently, if you HTML parse your way into that object and get the attribute, proceed to split (via regex option in Hydrus) you can get a list of all the attributes. However, although one will have a list of tags, they wont be counted as distinct results. I guess it's where optimality and usability are traded.

>>12896 Ah shit, yeah, sorry: I had forgotten how the copystat worked. It copies modified date, but a new creation date is made. I use this, if you are further interested: https://docs.python.org/3/library/shutil.html#shutil.copy2 Some permission bit stuff is copied as well. This was not true in early versions of the program, I think maybe four or five years ago, so files imported before then will have modified dates the same as creation dates.

>>12897 I don't use Pixiv, so I don't know how you make an Ugoira. The uploader sets the framerate I think. I presume they have an upload form for all the images and you add ms timings beside each file, and then they create the zip and the accompanying JSON or whatever to deliver the timings to your browser. But maybe they have a program that generates that info clientside and you upload it all together. Yeah, I'm sorry about the inability to split strings at that last stage. When I next put a good amount of time into the downloader, untangling the rats' nest of the edit parser ui will be a top priority along with fleshing out some needed parsing functionality like 'split this one string by whitespace and return multiple strings'. iirc, the current pipeline can't deal with n strings appearing from that space atm, so I can't just add a checkbox or a 'split this by xxx'. That might be wrong though, so I'll make a note to look at it.

>>12885 Thanks for the detailed response. I don't understand what you meant by this because no matter what I do, I have to switch focus from the tag input window to the media viewer to go to the next image: >Also, by default, hitting page up/down on an empty tag autocomplete input should send a previous/next command to the media viewer parent, so you don't even have to change focus to tag and move on. Btw, would you be averse to fleshing out the hover tag pane on the left of the media viewer with more functionality? I picture a possible workflow inside the media viewer being: 1. text input gets captured automatically by a new search box added atop the hover tag pane when pane has focus. 2. The hover tag pane shows tags already on the image when the search box is blank. When search box contains text, it's populated with autocompleted tag suggestions, and any autocomplete suggested tags that are already on the image have a highlighted gold background. 3. If there's no autocomplete tag suggestions, then the tag doesn't exist and therefore the Enter key should create and append the typed tag to the image. Idk what the performance implications of this would be, so maybe namespacing can be used to cut down the tag pool so it's less laggy? 3. up/down arrows can navigate autocompleted tag suggestions without needing to use mouse input to change focus from the pane. Enter key appends tag to image, delete key removes tag from image. Shift/ctrl click to select multiple tags perhaps? 4. Left/right arrows can navigate to the next image without needing to change focus via mouse click. 5. right click>mange>file tags for more advanced tagging

>>12796 Still doesn't work. The rule34hentai downloader still ignores Explicit rated images on rule34hentai by saying it couldn't find file or post URL to download.

Is it possible to download a pixiv tag starting from the oldest images through the gallery > pixiv tag search function or do I need to go through page by page starting from the first/oldest page in url download instead?

>>12923 Another bug I've discovered: if I go via the individual page links, after page 10/11 it keeps giving me the same set of images no matter which page I go to afterwards. Very odd!

>>12897 To better explain, if I open up the media viewer and then hit F3 to open up a manage tags window on top of it, I start with the keyboard focus in the autocomplete tag input text box, so if I were to start typing a tag, results would load in the list below. If from there, instead of typing regular text, I instead press page up/down, the underlying media viewer will browse previous/next. Do you not get that? If you have multiple tag domains, like 'local tags' and 'public tag repository', then pressing up/down arrow in the same situation–focus on the tag input text box, and no other content in that box–should switch between them. Do you get that at all? I am open to allowing easier tag editing in the tag list on the left, but the simple view it presents makes it tricky. It presents the 'combined' tags from all tag services in the client, so if you add a new tag, should it go to 'local tags' or 'public tag repo' or something else? I suspect my next step there is to add simple right-click menu options for 'remove tag' and then refactor the tag processing code, which is currently all bundled up into the manage tags dialog, into a shared space, so that taglist can be clever enough to say either 'remove this tag from x yes/no?' to 'this tag is in two services, remove it from x or y?' and so on. Once I have that code refactored, I am open to playing with more quick workflow here.

>>12922 Hmm, I am not sure then. If you open network->downloader definitions->manage login scripts, go into the 'shimmie login' and run a test on the domain with your credentials, it should list the results of each page hit. You should be able to copy the result body the server returned. You could try to copy and paste those pages into some new .txt files and see if there is a 'bad login' or cloudflare-related error that the login system isn't picking up? I am not sure if I wrote the shimmie login script or someone else, so I do not feel very confident talking about it. I understand it does work for some users.

>>12924 >>12923 Yeah, the basic hydrus gallery downloader works on the same principle of starting at page 0/1 and walking through each 'next' subsequent page, adding results, just like you might browse. Since most boorus present images in descending upload order, you will be getting newest first. I know some users have played around with adding a booru-specific param to the url to change the sort order, but this has mixed success. Can you say more about the same results after page 10/11 issue? Is this for pixiv, and is it in your browser or hydrus? Can you give an example URL that does this? Several boorus cap their results to 500 or whatever, even if they suggest something big like 152 pages, and after page 25 or so they'll deliver a blank page, 404, 403, 'please login to see more', or the same last page results again. Does this sound like that?

>>12930 The pages on pixiv in the browser are fine, they continue as normal. But no matter how many times I refreshed the client, reloaded page 12 in the browser and copied it again, it gave me the previous pages' images over and over, which obviously didn't import because they're already in the db. Here's the URL I was copying: https://www.pixiv.net/search.php?word=%E3%81%A9%E3%81%86%E3%81%B6%E3%81%A4%E3%81%AE%E6%A3%AE&s_mode=s_tag_full&order=date&p=12 The previous URLs had exactly the same layout except the number was just lower. 11 is the last one to load properly, after that it just returns page 11 irrespective of the page number.

>>12929 Actually forget all that, I'm an idiot. I didn't set the login to shimmie, this probably should be a default setting for rule34hentai.net if it isn't already.

Why do I keep getting this? How do I stop getting this? I've only got three pages open.

>>12932 I'm dumb and I figured it out. It needed a login to complete more than 11 pages it seems.

Just started using Hydrus, and I'm trying to get it to autotag an existing collection, but I'm a bit confused. >Tags are rich, cpu-intensive metadata. My repository has hundreds of millions of mappings, and your client will eventually download and index them all. Seems way overkill that the user should have to download all that information if they only need tags for the specific files that are part of their own collection. Isn't there a more efficient method? I'm aware also of the tag archives, but it seems that's also a broad and inefficient approach containing data for the complete file set from a booru/gallery website as well. Is there no other way except to maintain a massive database of all the tag associations of all files ever seen? If downloaders could be used to query downloading tags only for the user's existing files (it may be possible, but I am ignorant), that would be very useful and eliminate the need to waste HDD space/ processing power.

>>12946 Only if the hash of your file exactly matches the hash used by the booru of choice. Protip: they usually don't unless you got it straight from that booru. Use >>4475. And read the thread. It'll take some light editing to get it working due to a recent hosting issue.

>>12946 As I understand it, main reason is privacy. Under this system, you have full access to all those tags without having to tell a tag server anything about which files you do or do not have. And you can just sync once in a blue moon, instead of having to phone home every time you add a new file.

>>12948 That's a very good point I hadn't considered. >>12947 Having a real bitch of a time messing with scripts. I got a few from that furfag dr guy. Got one of them to work for one image that I manually selected, and then everything took a shit and the scripts just hang at the fetching stage now. https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/wiki/2.-How-to-fetch-tags-in-bulk-with-md5-hash It seems this method would be the absolute easiest but there is no longer anyway to get the md5 file hash in the options menu, and the default file names are the same as the sha56 hash. I think I'm going to just throw in the towel and try the public tag repository.

Not sure has this been asked before: How to minimize the active subscription windows? They are blocking the thumbnails and are very annoying.

I ctrl+f'ed the threads, so hopefully this hasn't already been asked. I created a default sankaku channel tag subscription with a test query of "tifa lockhart" (without quotes), but I'm getting a network error. Specifically, bad gateway 502 along with some html shows up at the top of the edit screen for the subscription. When I first performed the search, it asked me for a user agent string I believe which contained firefox and some other stuff. I did a similar subscription for gelbooru which worked fine. I'm using tor but can access the site just fine in my browser.

>>12932 >>12935 Thank you for the update. Pixiv seem to change their access rules every few weeks, it is a constant battle.

>>12933 Ah, thank you for this update. I think it is supposed to be default for new users–I think the shimmie login script has an example URL that does the link-up automatically on client first-boot initialisation–but I wonder if you got caught up in some combination of version updates that meant this did not apply? I apologise if so. The login system, as I am sure you now know, it pretty a scrappy v1.0, barely out of prototype. I do remember other users telling me R34H had funny login rules anyway, so I recommend you keep an eye on this in case it starts giving you more trouble. A future version of this whole system will be able to detect if you get suddenly logged out for some reason and will try to recover and give human-friendly errors and so on.

>>12934 I apologise, I fucked up the clipboard watcher! It is spamming those messages false-positively on certain kinds of URL that are not matched. It is fixed today, please let me know if you have any more trouble with it.

>>12946 >>12947 >>12948 >>12950 Yeah, privacy and low server CPU are the top reasons. I prefer that your personal CPU cycles happen on computers you control, so the tag repo is really just a clearinghouse of data. The PTR is getting huge now though, 580 million tag mappings, so after this duplicates work I am going to put some time into figuring out better solutions here. One top priority is adding a tag filter to the incoming stream, so you'll be able to say "I don't want filename: or title: tags" and so on. This is very much on my mind. The PTR has been a great success, but it was always a crazy prototype and we are hitting some limits.

>>12952 No excellent way yet. I think each sub has an option in its edit panel to not show a popup, but I generally advise against this as you often then don't notice it at all, even when it is having problems. This is on my mind, and I have a fairly high priority job to figure out an up/down arrow button or something so you can collapse and expand them. For now, best solution is to hit network->pause->subs and then just unpause them when you are happy for them to run.

>>12953 502 is a server-side error. It could be that Sank was overloaded at that time and could not respond to your request, or it could be CloudFlare giving the 'we tried talking to the site but it timed out' similar message, or they may have blocked Tor. Is the browser Tor as well, or just hydrus? If you run a test download and check the gallery log button (the little icon button on the gallery search section of the downloader panel), it should bring up a list with the first URL it tried to hit with more error info in the 'notes' column. Do right-click->copy notes and put it in a text viewer–that should be the html page that was returned, which should, if you scroll down a bit, have some more info. It might be like 'Tor is blocked m8' or something else. I'd be interested in knowing what you find, but be careful just pastebinning that html and linking it here, as it may have personal info.

I just clicked here from /tech/, shithole that it is. I have no idea what this project is. Could we have a sticky?

>>12973 >New user? Start here -→ http://hydrusnetwork.github.io/hydrus/ lrn2read

>>12966 Thanks, seems to be working now. >>12966

What do you guys think a good max allowed bandwidth would be? The default 64mb for sankaku is too little.

>>12965 That'd be nice. Or make each of them a tab would be nice too.

I noticed some of the tags from the PTR are just flat out wrong, such as pictures in which a girl is standing and facing the viewer so that her ass isn't even visible being tagged ass, and so on. Is there a way to deem certain public tags as less important and hide them, or a way to ignore public tags and copy only a portion of them to local tags? I suppose I could always just add a local alias and search by the alias instead of the public tag.

>>12986 I added this after being contacted by a top person at Sank who was concerned about their bandwidth costs, which I sympathised with. After some discussion, we compromised that I would set a restrictive bandwidth limit by default. You can of course edit your client's rules to be whatever you want. If you want to do them a favour, you might like to try to find what content you can on other sites.

>>12995 There are not excellent ways yet. The PTR has plenty of weird and incorrect stuff, and there are so many new tags every day that it is completely beyond me to moderate effectively (I am also not a natural janitor or site-manager). I will be doing substantial work on the client-repository relationship after this current duplicates work. You will have more control over what tags you get and where they go. I expect to add multiple 'local tags' services and easier ways to migrate your tags around so you can better handle all this stuff.

Anyone else having problems with the deviant art downloader, first the login script stopped working, now it can't seem to download any page content to parse correctly.

>>11887 >>11791 here, sorry for the late reply I tried it again on v355, still happens I'm using the AIO reddit and imgur parsers (or the imgur one that comes with the hydrus client, not sure) from the "Hydrus Presets and Scripts" repo

I created a gallery download page and it stopped after 19 images with a message about 502 bad gateway. I figured the booru was having issues, but no, I can access it just fine. In fact, a new gallery with the same exact query is downloading right now just fine, making the message below confusing: expires in 50 minutes: 502: <html> <head><title>502 Bad gateway</title></head> <body bgcolor="white"> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx</center> </body> </html>

Is it possible to have two clients share a database? I keep my data on external drives and want to be able to work in both Windows and Linux.

>>13018 I think the answer is yes, they just can't be open at the same time.

>>11122 Sankaku gives you at most 1000 results from any query, but you can circumvent that to some extent if your initial query was at most 3 tags by looking up query + order:filesize and query + order:filesize_asc. Is there a better solution to this? Finding a custom query to get as close to 1000 images without going over and then having to do this every 1000 images is a pain. Even 2000-3000 images would be a massive improvement.

>>13029 Nm, I happened to be checking the bugs thread and noticed someone made a parser that solves this supposedly. Just out of idle curiosity, why are parsers shared as png files and not text files? Also could a parser be malicious?

Databases work cross-platform, right? I want to try out hydrus through wine because of all the linux version's issues. How do I go about making the switch? Do I simply copy and paste the linux hydrus network over the windows one like I was simply updating? Or do I restore the (linux) backup I have through windows?

just started using this a few days ago, really liking it for sorting my giant folders of booru rips, but one thing I can't find is parent child picture relations, like danbooru has. I want to be able to set one picture as a parent, and then another as a child picture, for alt versions and small doujins/image sets. Is there a way to do this or am i fucked?

>>13031 I wrote that parser. It's a png because Hydrus lets you export them as .png files for easy importing. If you're that worried just copy and paste this: [58, "sankaku gallery page parser fixed", 2, ["sankaku gallery page parser fixed", "b29f9b48f7c6374d12e7d4cbb527f16fd3c725946dd6479c0684e5e0e1dea15f", [55, 1, [[], "example string"]], [], [26, 1, [[30, 4, ["Next page from rel link", 7, [27, 6, [[26, 1, [[62, 2, [0, "div", {}, 22, null, false, [51, 1, [3, "", null, null, "example string"]]]]]], 0, "next-page-url", [51, 1, [3, "", null, null, ""]], [55, 1, [[[1, 7]], "parsed information"]]]], 0, false, [6, 50]]], [30, 4, ["Thumbnail URLs", 7, [27, 6, [[26, 1, [[62, 2, [0, null, {"class": "content"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]], [62, 2, [0, null, {"class": "thumb"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]], [62, 2, [0, "a", {}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]]]], 0, "href", [51, 1, [3, "", null, null, "example string"]], [55, 1, [[], "parsed information"]]]], 0, false, [7, 50]]]]], ["https://chan.sankakucomplex.com/?next=3794306&tags=krystal"], {"url": "https://chan.sankakucomplex.com/posts/index.php?id=123456"}]] And I suppose a parser could be designed to work in such a way it makes a shitload of network traffic to get you IP banned from a booru, honestly, mine might because sankakuchan hates people using their bandwidth yet has the most expansive collection of images imo. Also it's not too hard to make a image file into a malicious script, that used to/might still be an issue on Linux, it just requires you to execute the file which isn't really a concern with Hydrus.

>>13033 You can set images as alternates, and you can click an images and find all of its alts, but not much more. It is something the dev is working on currently.

Is there any way to print a list of tags and their occurances? It'd be really helpful to weed out the typos and stupid non-tags artists insist on using in their galleries

>>13033 This is a highly requested feature, which I suspect may win the 'next big job' poll that the dev does every few months. IIRC, the duplicate system overhaul he's doing right now is basically a precursor to being able to overhaul the 'alternates' system. For now, I'd suggest setting those images as alternates of each other.

I imported the latest Quicksync update files (not the full db), and it's taking forever to process now, like 1 entry per 10 minutes, then cancels out after around 15 minutes. Is there something i'm doing wrong or do I have to let this run for a year?

Is there a way to set multiple tags with one shortcut?

>>13034 But doesn't hydrus execute the code from the png? Also the downloader still finishes at only 1k even though I've set it to stop after 2000 and changed the bandwidth restrictions. Is sankaku channel tag search the right downloader to choose?

Is there a simple way to set up a blacklist for subscriptions? I have a subscription setup for sankakucomplex with a few hundred artist queries in it, and while I do want Hydrus to download anything tagged with those artists, I don't want it download anything tagged with, for example, "holding_hands". I know could add "-holding_hands" to each query, but I don't want to have to do that for every single query.

>>13014 It seems ok to me here. I just did a login test and download and things came through ok. Are you getting any errors, or are the URLs just not appearing when you do a gallery search? If they have any errors, can you look at the list of file import statuses (hit the little icon button on the downloader page) and right-click->copy notes on the bad results, so we can see more of what went wrong? I wonder if they turned on CloudFlare protection or something briefly.

>>13016 Thank you for this follow-up. After a bit of playing around, I was able to reproduce this problem. There is a certain unusual parsing type that was kicking in here that was not sending tags to the next stage down. This is fixed for 358. Please let me know if you have any more trouble here.

>>13017 If you turn on help->debug->report modes->network report mode while this works, does that give you any more information? Is there perhaps a url class that is redirecting the URLs to a different URL? I've had that sort of problem before, where I was accidentally hitting the wrong URL despite the URLs in the file list looking correct. It could also be some CloudFlare-style protection that is blocking anything that doesn't look like a browser with a BS 502 error. Usually you get more info than that brief html document though. If you have some example URLs, I am happy to try from my end in browser and hydrus.

>>13018 >>13027 Yep, the db is platform neutral so you can have two installs point at the same db and they should be fine. Only rule is only one client install can be open on a db at a time. Normally this is impossible to violate since the client will detect if it is already open on the same computer and complain, but if you have a funky network setup this check can be skirted. If you just want to have a db on a usb drive and move it back and forth between two machines or boots, Linux and Windows, you should be completely fine. I recommend you extract two portable installs and put the db proper in its own directory, something like this: E:\hydrus_db (db and client_files folders go here) E:\hydrus_linux (linux install extracted here) E:\hydrus_windows (window install extracted here) And then launch Windows/Linux with shortcuts with '-d="..\hydrus_db"' or '-d="../hydrus_db"' as appropriate. But you know your situation better. Here is some more help about all this: https://hydrusnetwork.github.io/hydrus/help/database_migration.html Let me know if you would like any more help.

>>13032 I'll just point you up to my post above: >>13065 Basically there is no trouble migrating the db around–it is completely portable and will work on any install from any location, so you are good to copy or move it around to test in other locations. As long as you can see the main db dir with your hydrus client exe or script, you should be good to go and if everything is working, it'll launch. I still recommend making backups and so on before you try anything too fun, just in case. The help file here talks about the different components of the hydrus install: https://hydrusnetwork.github.io/hydrus/help/database_migration.html I think in your case you'll be wanting to move around your original install_dir/db folder. Extract hydrus to your wine folder structure and then copy that db dir straight into the install. I am happy to help if you run into trouble.

>>13051 I am no cyber-hacker-3000, so I cannot promise anything iron-clad, but my serialised png system shouldn't execute any external code. The data embedded in the png is zipped up JSON, stored as grey bytes at the bottom of the png itself, which I read and load. It unzips to the same as in >>13034 's post. There's a couple of data bytes in the top-left of the png to say where the data starts and how long it is. If what is read and unzip parses as a hydrus parser object, it is good to import, otherwise it will throw an error. At no point are arbitrary lines of python code intentionally executed. If someone has an attack vector for https://docs.python.org/3/library/zlib.html#zlib.decompress or https://docs.python.org/3/library/json.html#json.loads, they could make up a malicious png. My Windows dev machine here says the current zlib version I am on is 1.2.11, which the docs suggest is currently OK. If you are very concerned about your privacy, I recommend you only import by copy/pasting the json text and giving it a sanity pass with your eyes just to make sure it isn't trying to api-url-redirect your gallery queries to "NSAHydrusUsers.ru.cx" or anything.

>>13037 Not at the moment. Many clients have way too many unique tags to show in one place in any kind of reasonable wait time, so figuring out show-all UI has been tricky. There's a 'big job' to figure out a tag cloud control which would likely have this tech. If you only have, say, 2,000 unique tags, you might have some luck messing around with wildcard searches in the autocomplete tag input. I have some blocks to stop simple '*' queries, but someone was telling me the other day that '*:*' works and will show basically everything you have. Be warned it may take some time for such a query to return all results and may be hyper laggy. Otherwise doing a search for a whole bunch of random files and then sorting the results' tags by incidence is a decent way to find (1) and done stupid tags.

>>13043 Try forcing an analyze under database->maintain->analyze->full. It will take a few minutes, but if your problem here is due to too much data at once, this should fix it.

>>13050 Not yet. Future shortcut system should be able to do this, but I will likely not work on that until I have caught all the old code up to what is current.

>>13058 Sure. You do this through tag import options of any downloader context. You don't want to set this for every individual subscription, so I recommend you go network->downloaders->manage default tag import options and then set them either for the global default or per domain. If you haven't overridden this info in your subs, your subs will inherit from these defaults at runtime. Under your tag import options edit panel, check the button for 'set file blacklist'. Add your scat/gore/whatever tags that you don't want, and any downloader using those tag import options that finds a page with one of those tags will skip the file download and give an 'ignored' status. It is worth pausing your subs while you do this and testing this with a manual download page, just to make sure you have it set up how you expect.

What was the procedure for manually copying login cookies into the client?

>>13067 Thanks for the breakdown, and I'm not too concerned about it. It was an idle thought that arose from thinking how this png system weirdly obfuscates the downloader code, so I wondered what the security implications were. >>13034 I looked and I didn't see any new downloaders added, so I assume this was added onto the sankaku channel tag search downloader. Is this intended for use with the gallery downloader? Maybe that's why it halts after 1k downloads.

>>13072 This should be explained in >>11696.

>>13072 >>13075 I have a lot going on at the moment, but in the 'to-do fairly soon' list is adding cookie access to the Client API, which will make this whole thing a lot simpler. The guy who writes the browser plugin is keen to add a button that'll just transfer login cookies to hydrus in one click.

>>13073 I don't know exactly how or what you imported in your situation here, but if you added this, you likely also need to update the 'links' to make sure it is applied to the right 'url class'. Check network->downloader definitions->manage url class links. Find the sank gallery url class and make sure it is matched up to the new parser you imported.

Anyone else having issues with pixiv? I keep getting an error that "The parser found nothing in the document!" Only started happening within the past 48 hours.

>>13083 see >>13057

>>13078 I imported the png version of >>13034 via network>downloaders>import downloaders. It's supposed to allow the sankaku gallery page to exceed 1k items. Interestingly, after I did network>downloader definitions>manage url class links and applied `sankaku gallery page parser fixed` (which I assume is from the png I imported) it consistently stops after only 24 items.

>>13085 Hmm. That is the correct procedure for switching over. I expect you'll want to switch back until this gets figured out. I am afraid I do not know anything about the 'fixed' parser other than it is supposed to pull a different gallery page with some 'startswith' id or something, so I can't talk cleverly about debugging it. I assume you are logged into sank if you were originally getting 1k results, otherwise I would say 'try logging in'.

Is there a way to use Hydrus with sites that make you click "Agree" or "Enter" before you can use them, e.g. giantessbooru.com?

Whenever i try to download tags from the repo i get this error. It's always the same keyerror. I tried looking in the DB files for they key in case it was conncected to a faulty path but i can not find it.

Hydrus doesn't download anything from DeviantArt, no matter if I log in or not. I understand that you're supposed to have the client download something without logging in before you can do it with an account, but how?

Gelbooru added some gay default tag blacklist you have to modify to access certain tags. Any way to remove them from hydrus? Here's an example. https://gelbooru.com/index.php?page=post&s=list&tags=loli

>>13114 Also, when you go into the options you can enable all site content and it gives you a special key you should be able to throw in with searches to get the full results like "&api_key=anonymous&user_id=9455", but I couldn't get it to work with anything.

Speaking of tagging systems, what are idealistic searches you wish could be usable on boorus? >eg. "on-model" >eg. "girly/femboys but no masculine males in the image" (possible solution would be having negating tags for like "non-femboy") >eg. "horsecock but only on horses" As a side note, I found an imageboard which used anime girls for an image captcha. I wonder if that could be utilized for tagging boorus, like how Google uses theirs for AI training.

is there a way to exclude a tag from a 'everything' search?

>>13111 Never mind, I figured it out. I needed to clear cookies first, through the login menu, then try again. Now that I have downloaded some pictures without logging in, I should be good to go.

>>13107 Yeah, should be able to. In the new downloader system, that process would count as a 'login'. You can create one of these yourself, although it is not trivial to do. You also need very basic web development knowledge, such as what a cookie is. The help for all this starts here: https://hydrusnetwork.github.io/hydrus/help/downloader_intro.html There is supposed to be some login help here: https://hydrusnetwork.github.io/hydrus/help/downloader_login.html But I just discovered that I never wrote any! If you would like to write your own login script, skim some of the downloader help to see what my approach is and then check out network->downloader definitions->manage login scripts. Your client should have 'hentai foundry guest login click-through' as one of its defaults, which basically does the same as what you want here: if it doesn't see a session cookie in the HF session, it simulates hitting the main page and clicking-through. I think if you duplicated the HF login and then edited it to your other site you might have some luck. That particular HF script also posts an HF content form to unhide spicy content, which you can remove. I am happy to help you if you run into trouble, and if you figure something out, please post it so other users can benefit!

>>13108 Thank you for this report. Please pause the repo if you have not already. Can you then go to your log (install_dir/db/client - 2019-7.log) and scroll down to this traceback and copy the last bit? (If you still have the popup, you can do the same thing with the 'copy traceback' button). I think I know what is causing that error, but I'd like to check the whole error be certain. It looks like your files manager is/was unable to find one of its subdirectories. Have you ever moved your database around with database->migrate db? If you have, please go to your moved 'client_files' dir. If you have not, please go to the default at install_dir/db/client_files. Are there 512 directories in there? There should be 256 in the format fxx and 256 with txx (files and thumbnails, respectively), with the xx counting in hex from 00 to ff. Do you have the 'f4c' directory? I suspect you do, because if it were missing, hydrus would complain on boot. If so, it sounds like your db has somehow lost its internal store of db locations. If you do have the f4c folder, please make sure your client is closed and then jump up to your base db dir, normally install_dir/db, and run the sqlite3 executable. Copy paste these lines into the terminal that pops up exactly: .open client.db
SELECT COUNT( * ) FROM client_files_locations;
SELECT COUNT( * ) FROM client_files_locations WHERE prefix LIKE "f%";
SELECT COUNT( * ) FROM client_files_locations WHERE prefix LIKE "t%";
.exit You should get 512, 256, 256, but I assume some will be less. Please let me know what you get. We can fix it manually if only f4c is missing (i.e. if you get 255), but if there are more missing I won't walk you through manually fixing this issue, since it is a pain in the ass going through 256 entries to figure out what is missing. If this is your issue, I will write a new repair routine on boot that will catch this unusual problem and try to fix it automatically. I am 95% certain I can have this in for 359. You should not import any files to your client in the meantime. That missing directory issue will affect any file imports. The cause for this issue is also a concern. I cannot think of any way that one of those directory entries would be removed. Anyway, let me know what numbers you get, and the full traceback, and we'll get further into this.

>>13111 >>13121 Thank you for this report. I have a report from another user that suggests the DA login script is broken, and I am due to give it a look this week. I am not sure what your cookie issue was, but please keep me updated if you learn anything new.

>>13114 >>13115 I don't have a gelb account so I can't talk too cleverly, but I think your best bet here is to have hydrus login to gelb for you. I recommend you use a new throwaway account. Am I right in saying that gelb has a blacklist tied to your user account? I assume you can set your throwaway account's blacklist how you want, and then hydrus will inherit that when it logs in. To add that api_key to searches in hydrus, you'd have to alter both the gelb GUG and URL Class in hydrus. It might be a bit of a pain if you aren't used to the system, so I recommend trying the login first. Let me know if that works for you. Related help at the bottom here: https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html

>>13121 >>13124 OK, so I can download stuff if I'm not logged into DeviantArt, but once I do log in (and it does say I'm logged in just fine), it doesn't download anything. The login says "valid", and I get no error messages; it just doesn't import any files.

>>13122 Thanks! I'll look into it and see if I can make it work. I actually sought help from that downloader_login page before posting, but it wasn't too helpful, as you can imagine. :p

>>13117 I don't tag on boorus, but for hydrus I am all for longer/combined tags to clarify these situations. I think tag parents in hydrus can help tag quicker too. For instance: horse penis on horse could have all these parents, some being grandparents: horse penis equine penis animal penis penis animal genitalia species:horse species:equine whereas 'horse penis on futanari' or '-on human' or whatever other pseudo-suffix would have a different set of nice parents. It is neat as your brain can work on this higher level of recognition and it lets you add a bunch of valid tags in one action. Same with something like "girly/femboys but no masculine males in the image" or the various "small female big male" stuff you come across sometimes. With these specific tags, you can define precisely the core of what you are interested in in the image without being awkward or overbroad in the search. Also it lets you quickly and precisely dismiss what you don't want with a single negation tag like ['horse penis on horse', '-horse penis on human' ], rather than ['horse penis on horse', '-species:human'] or ['horse penis', 'species:horse', '-species human'], which are not the same thing. I think you could easily go crazy trying to define these forever, however, so I will fall back to one of my main tagging rules: only tag what you are personally interested in. If you want to see horse penises only in their natural God-given state, then pursue this 'horsecock but only on horses' path, and perhaps the '-on intersex' companions you might want to exclude with a negation tag. But don't try to tag everything on everything on everything or you'll go nuts. I have spent thousands of hours creating an engine that burns trillions of cycles of CPU to precisely categorise and search different types of bepis. Didn't expect I would be here ten years ago, jej. I am certain that within ten years we will be feeding this info into AI tech of our own with some fun results. :^)

>>13118 Yeah, when you start typing a tag, put a hyphen "-" before the tag, so like "-footwear". This will discount any file with that tag from the search. I think all regular tags will work this way, including special tags like "creator:*anything*". If you have some tags in the 'selection tags' area on the left, you should also be able to right-click them and say something like 'exclude this tag from the current search' and some other options. Note that 'system:everything' is secretly implied to any search that has >0 tags, so if you just enter for '-footwear' or whatever, you don't need to add the 'everything' yourself. Also, when you do big searches like this, try hitting the 'searching immediately' button (ctrl+i shortcut as well, I think), which will pause the search-on-new-tag and lets you queue up a bunch of tags before actually firing off your big search.

>>13126 Thank you for this info. No idea what's going on here–maybe the Eternal Cloudflare has struck again, or maybe once you log in they now deliver differently 'shaped' gallery page data or something. I'll look into it.

>>13125 >Am I right in saying that gelb has a blacklist tied to your user account? Yes, but it also has a default blacklist which you need to disable in your "account". Your account being any account including temporary anonymous ones. Anyway, I tried it before but I forgot to modify the GUG before, so I tried it again and still nothing. The URL throws the extra information in properly, but still no results. (1-3) I also tried creating a temporary account, it logged in properly, but still no results despite the cookies looking like they're working. Throwaway account and literally billions of possibilities for the remainder of the hash, so there's no problem with the fifth pic. (4,5) I tried restarting hydrus just in case but still nothing. The API thing doesn't work on the browser, so I guess that's Gelbooru's fuck up, but logging in with an account with the content enable does, so that might be some fuckery with how hydrus downloads stuff I'm guessing.

>>13123 I have 250 'f' folders and 256 't' folders. I did migrate my db from one folder to another at one point and suspect that something got messed up in the process. I have also noted that when i run the 'file integrity' tool other folder names appear, which i presume are other missing ones.

>>13136 Another example.

>>13085 That parser was faulty, I believe. I made it work for me some time ago, should be at >>13138 – please tell me if it works.

To hydrus dev specifically: how do you manage all the tasks involved with a project of this scale? I find it staggering that you're able to keep on top of it all so well, presumably as one person. Apologies if there's already an answer around.

I asked some time ago about how to restore old downloader functionality(e.g. "get X more new files"), and I tried simply increasing the "stop after this many files" values(search and gallery parser) and resuming the search, but Hydrus doesn't do anything with it. Is there anything else I can do or must I switch to searching with a high value initially and pausing when I'm satisfied?

>>13125 I've been messing around with it a little bit. Even though I was able to log in to Gelbooru successfully in Hydrus, it didn't allow me to download any blacklisted tags, neither in a download page nor subscription. However, following the instructions from the help page you linked and exporting a cookies.txt from browser into Hydrus did allow it to work. I suppose that means the relevant info is not tied to your account but is in a cookie. I think this is the setting in Gelbooru's options that's causing the issue.

>>13168 I thought I had looked through the cookies before but I'm guessing I'm retarded. Pic related is enough to get it working.

I've been working on creating my own parser recently, and have been successful at the parser bit, but it only downloads the first page of a gallery link. How do I make it download the entire gallery? My gallery parser correctly returns a "gallery next page url", and I have a url class of type "gallery url", yet only the first page is being downloaded. Am I missing something stupid?

Whoahhhh so what the fuck am I looking at here? What exactly is hydrus? Can someone please give me a run down

>>13171 Try setting it to no limit and download a tag you know goes past one page, then look at the gallery log. The last URL should give you a note that'll explain what's fucking up.

Is it possible to have the tag autocomplete enter existing namespaces when tagging with a word that isn't a non-namespaced tag? e.g. entering "splatoon" automatically tags as "series:splatoon" since I don't have a "splatoon" tag in my tag services. I know it's possible with tag siblings but adding an additional pair for each tag is a lot of extra work.

>>13184 It already autocompletes in the searches if that's what you mean. If you're talking about tagging it as "splatoon" and having the tag translate to "series:splatoon" then yeah, you need sibblings.

>>13132 >>13170 Hey, I am sorry I was late to this. I think another user (sorry if it was you somewhere else, and I am forgetting) was telling me that this cookie is not tied to the account, but the session. I still have a job to add cookie export to the Client API. This will make this operation a one-click action for anyone with the browser add-on and clear the hassle from this shit.

>>13137 >>13136 Hey, I am sorry getting back to you late here as well. I believe this now auto-heals on boot in 359, so have you had luck with that? For some reason, six entries have been dropped here. You might like to check install_dir/db/help my db is broke.txt just as some background reading. It might be worth running a db integrity check on client.db at least, just to see if there are any other dropped rows in that file. If the new version did not fix this automatically for you, you can fix it manually by using this line: INSERT INTO client_files_locations ( ?, ? ) VALUES ( "fXX", "client_files" ); For each dropped fXX prefix. fb9, f4c, and four others that I imagine you can figure out by booting and running some more file integrity checks or similar. If your actual location for the files is not install_dir/db/client_files (as per the "client_files" parameter there), the client will ask you to relocate to the correct place on boot.

>>13146 I use ToDoList by AbstractSpoon to 'keep track' of everything, and I strongly recommend this software for any project tracking (I also use it for daily schedule management), but in truth I am always drowning and overwhelmed, ha ha. I am just one lad, and once hydrus picked up a little, my todolist queues quickly turned infinite, almost always growing faster than I can process them. You can see the current running todo count at the top of this page–1,783 items atm jej. I sometimes cut that number by a handful after a good week of small work, but it typically grows by 15 or so items a week. The ride never ends. I have found the trick, given the unrelenting queue, is to instead bottleneck my own schedule so I don't burn myself out. Marathon not a sprint, etc… Although I love working on hydrus, I get tired pretty easily. I usually do about 30-40 hours a week on hydrus, and while I could easily 'schedule' 100, I'd probably still not catch up on my to-do for another twenty years and I'd kill my soul pretty quick and quit the project. Strict schedules have been helpful. Although I am fairly clever, I am not conscientious. I am not normal or professional, so I have some different ways of working compared to typical devs on other projects. I don't do many widely considered good practises and the back of my mind generally recoils at them, viewing them as glow in the dark. Coming to know what I am weak at–and then not try to do that–has been difficult but important. I know that I burn out and end up in drama-bombs when I work with others, so I have a strict solo-dev policy for hydrus, and that has worked really well for me. Weekly updates is another–I am a 'do homework the night before' kind of person, so whenever I have tried monthly updates, I end up playing an MMO for 2.5 weeks and then scrambling to catch up. This schedule keeps me focused and allows for quick prototyping iteration and bugfixing turnaround. Focusing on larger jobs has been difficult, but the 'big job' poll, giving me a day or so a week to push on a longer-term thing, has been great. And having the new 4-weekly rotation of small jobs, cleanup, small jobs, ongoing polish has been good at ensuring I don't get lost in the muck of some bullshit for six weeks and forget about other important things. The most difficult thing that I still fail at is keeping up with messages. I am not social and don't enjoy stepping out of being Anon. It is real work for me to deal with posts and DMs and emails, and it isn't something I can avoid, so I mostly catch up when I have the motivation and guilt to get it done. It helps having an Anon-centered userbase. I love imageboards and enjoy having tech-fluent users whose hand I do not have to hold. It means I can put out scrappy tech and make autistic but functional UI rather than needing to make things nice and easy and pretty, which I am not good at, and still get great feedback on complicated topics.

>>13159 Yes, unfortunately there is no way at the moment to get that old queue-building method back.

>>13168 >>13170 >>13191 I guess since this is divorced from needing an account, the cookie is not actually remembered account-wide. Thank you for this information. If this is a tick-box with a typical html POST form to save it, I should think it would be possible to both write an account-less login script and extend the existing account-credentialled gelbooru login script to also do this step, if they user wanted. The Hentai Foundry scripts do something similar here, to do the initial home page 18+ screen click-through.

>>13171 >>13183 If you are testing the URL in the normal URL downloader, this page doesn't have the pipeline tech to do a gallery walk, so it will only ever parse the single gallery page you drop/paste into it. If you want to test a full gallery walk with 'next page' stuff, you will need a GUG for it and do it in the gallery downloader.

>>13184 >>13185 Yeah, my first siblings system proved only half the job here. I want to make sibs and parents much more portable so users can just grab like a 500KB text file from somewhere and just pull a whole bunch of decent 'series' siblings from the PTR to their local tags in one go. Doing it as one person by hand is a huge pain, and the current system does not resolve subjective disagreements well (e.g. is "bodypart:", though technically accurate, actually 'good'?). The next step of tag sharing will be better en masse definition curation, and making it more shareable.

>>13193 Thank you so much for this fascinating insight. I've been struggling with organisation a lot recently, so I thought I would ask you how you do it, as you're clearly able to continually deliver despite all the challenges you mention. That's where I'd like to be, and I have every respect for you. That software looks like just what I've been looking for (and not finding) - thank you again. I thought that keeping up with messages might be difficult, as you have so many scattered locations to check. I expect it to be a shortcoming that imageboard communication is naturally ephemeral, so you may feel pressured to follow up with people in a timely manner. Perhaps that helps you keep on track though. I expect I've taken quite a bit of your time at any rate, for which I apologise. Certainly don't feel like you should respond to me. I have held off on suggestions for the software in respect of the fact that the userbase is something worth preserving, as you say. Any improvements to the UI, for example, would probably lower the barrier of entry and result in an influx of lower-skilled users, causing more hassle for you. I had wondered, though, if it was possible to write hydrus in such a way that frees you up to work entirely on the backend, leaving frontends to interested users. But I'm sure it's a lot of work for not much payoff.

Am I required to run hydrus as an administrator? I don't recall needing to several versions ago.

>>13206 Thanks m8. I appreciate your comments. I like to insist I am just a lad though, no one special. I don't want to get a big head. It has taken years for me to figure out a good schedule and attitude to doing this work, and there is no magic trick I have found except keep working every week and recovering over and over from failure. If you want to figure out your own system, my best advice is to keep at it, and if you are young, don't expect it to come as fast as you want. Stress is part of the whole package, like any other work, and another resource to be managed. Stoicism works for me. It is a little cliche in some imageboard circles, but if you haven't read it, I recommend Marcus Aurelius: https://archive.org/details/meditationsofmar00marc Now that hydrus is functional and does a whole bunch of things, my overarching plan is to move myself and the hydrus code to be more platform-focused, providing a base that users can make things for, rather than pushing towards a complete package I have to continually tinker with. The downloader overhaul last year was the first really big stab at this–moving the whole thing to be completely user-configurable so regular users can populate and maintain it, rather my having to hardcode and update all download info myself. I now want to extend the Client API, maybe play with add-ons of some sort, and 'generalise' the code to allow more user-configurable behaviour. This lets me worry less about infinite specifics and spreads the workload without my sperging out at working in a team. I'll be doing some PTR/tag repository work after this current duplicates stuff. Main push will be not dissimilar: giving clients more control over what they pull from the PTR, and starting work on improved local tag siblings management. Completely decoupling the UI from the db is probably too big a job to consider at the moment, but I know some users are planning pseudo-booru wrappers using the Client API. We'll see how that goes and iterate on from there. I am all for new suggestions, btw, so please don't be afraid to say something. I just have trouble keeping up with them a lot of the time. I'm not about to add a Facebook plugin, ha ha, but thoughts on improving and cleaning up workflows is always helpful. As the guy who understands the cogs behind everything, it is often difficult to picture what is confusing or unhelpful to users.

>>13209 Shouldn't do, unless your db is in a protected location like 'C:\Program Files'. The client needs to write to its db dir, which is by default in install_dir/db, so if that is in a protected or otherwise read-only location, it might not work as a regular user. What error messages are you getting, if any, that drove you to try 'run as admin'? Here's some general info on the db and how to move it about, if that is helpful: https://hydrusnetwork.github.io/hydrus/help/database_migration.html

>>13212 Ah okay, the install directory got moved recently after discovering Chocolatey and installing hydrus through it. I'll see about moving it back to the root of C:\ where it was before.

>>13211 Yes, that makes a lot of sense. I agree about stoicism. I happened upon your recommendation of Marcus Aurelius to another guy, and it's good stuff. Some self-discipline is certainly in order, which cannot be provided by software. I certainly intend to give some feedback after I've had a chance to organise my thoughts. I'm sure most of it, if not all, will have been said already though so hopefully it's not too tiresome reading duplicate suggestions. Can't really see a public to-do list (or the like) around to help prevent that, though.

>>13192 Updating to the latest version seems to have solved the issue. When i started the client i got the prompt that the db entries were missing, and i could then point them to the folder where my files are stored. Will do some integrity checks now to see if anything else breaks. Thanks for the help!

Is there a way to force-download the whole tag set of the PTR? I increased the max bandwidth for Hydrus services but it's saying >service has exceeded bandwidth - next request in 5 days 17 hours I know there was a way to do it in a previous release but I can't find it now.

>>13221 Ah, never mind. I found the option to import.

I already have the PTR downloaded, is there any way I can export it and import it on a separated hydrus?

Did the dev ever add the ability to manage multiple databases? I know there was talk of it at one point, but I haven't kept up with development in a while. I like the idea of having separate databases for SFW and NSFW or some similar scheme.

Why is there still a search plugin for the Mishimmie? It's dead, it was never exported and Dreamhost has no interest in restoring it.

>>13217 Yeah, I'm afraid my to-do is too big to reasonably convert to a public system. It has personal info in it too, from user submissions and so on, so I'd have to sanitize rather than just copy/paste. I also haven't found an online issues system that doesn't make my head explode either–for a while, I had github issues activated, but for me it just resulted in dozens of long discussions about how potential new features might behave, and I lost more and more work time to it until I couldn't keep up and abandoned it. All that shit is driven towards boosting social media engagement, which is the very thing I hate, kek. I know some users on the discord tried to make some sort of wiki to act as a 'frequently asked issues' or something, but I don't know how it ended up.

>>13220 Great, thanks for letting me know. >>13226 Yeah: turn on help->advanced mode and then check the repo under services->review services. Hit 'export updates' to pull them out, and then on the new (also advanced mode) client hit services->import repo update files to push them in. Then do one sync with the PTR on that new machine so it knows the updated files it wants, and it should recognise it already has them and be able to start processing right away. Be very careful doing this. It is a lot of CPU work all at once, and I didn't design the processing pipeline to deal with it. I recommend you let it run for x minutes processing on a shutdown maintenance run (which will have strict timing rules and be able to cancel itself eventually even if it runs super slow), and then restart the client and run database->maintenance->analyze->full. This will make the db do some optimisation stuff on the new populated db tables so it can do the rest of the processing at decent speed. I overall do not recommend hitting 'process now', if any user has told you about it, on a fresh sync. Some users have tried to do a full processing run in one go from nothing, which usually locks up the client due to mis-optimisation and slow work. Let me know how you get on!

>>13238 Not yet. It'll be a pretty huge job, unfortunately. I'd like to have multiple local tag services coming in after this duplicates work as part of a significant en masse tag management overhaul.

>>13241 Thank you for this report. I'll take it out of the defaults.

Is there a way for me to filter by filetype? I know you can sort by filetype as an option, but say if I wanted to only see .jpg files of a certain tag, is there a way for me to do that besides manually adding filetype tags to every image?

Guys, how do you set up a Derpibooru login? I do not know what kind of login script it uses. My grabber just uses the API key.

>>13258 yes you can filter by filetype same way as you apply other filters to a search

what happened to the downloaders here https://hydrusnetwork.github.io/hydrus/help/adding_new_downloaders.html

is there a way to know the last time hydrus was open? I went to a repair center and I'm afraid the saw all the nasty shit I have.

>>13273 Maybe the log files? Check last time modified, or if there are any dates written to them.

>>13279 fuck, they did. fuck fuck fuck fuck

>>13280 >>13279 fuck I'm not going back there never again.

>>13281 Holy shit, there were looking at your stuff for like 7 hours? Maybe they want a copy

If my filenames already contain character delimited tags like "dog, corgi.jpg" how can I have Hydrus import those words individually when I select "add tags from filename", rather than creating a single "dog, corgi" tag? I assume there's a regex trick. Also any general tips for scrapping tags from an existing image library's filenames?

>>13281 F This is why we use passwords (And not repair centers)

Hopefully ther's an easy way to do this. I have two seperate instance of Hydrus with different files, tags etc. Is there a way for me to merge them into a singular Hydrus instance with all tags intact? Could I start the server on one port and connect to it from the other instance and grab everything?

The menu bar on Hydrus is missing. Not the sidebar, the one on top. Is there a way to get it back? I don't remember removing/disabling it. On a semi-related note, is there a list of built-in keyboard shortcuts?

I wanted to make a thread, but it would probably be useless if the BO/devs don't agree to the proposal, so I think I will just post here. Due to the recent take down of ExHentai, I'm aware some anons use Hydrus, either as a database for their own pics or not, myself included, since I use it to tag hentai doujins I store for easy access so there's a chance that they have doujins databased on hydrus of their own. What chance there is to convince the Hydrus dev of making something out of this? So anons makes a collective database of mangas and doujins so they can be easily shared there and to some other place? It could work like p2p, so no extreme bandwidth use in Hydrus servers, hopefully, and if it works good, it'd work like a torrent client of sorts, but to display these doujins directly. If it goes well, big circles can host their doujins themselves there. I might be extremely wrong, though. But just want to know if there's a chance.

>>13307 Hey, I am afraid I have a ton of IRL shit going on today, so I can't catch up with my messages properly until tomorrow. I will try and check in again later this evening. Please feel free to make a fresh thread. There is certainly a lot to talk about on this subject. If Anons can get use out of hydrus as an archive tool, I am happy. As I say a bit here >>13305 , to me this episode is a reminder that you can't rely on content, particularly spicy content, remaining in a location you do not control. Hydrus was in part designed to solve this problem by being a 'local' booru. That said, hydrus is not currently well geared towards paged content like comics. Increasing support is planned, and I am in no way opposed to adding cbz/cbr import support and improved paged browsing, but at the moment the overall hydrus workflow works best for single images like you would find on most boorus and imageboard posts. Any Anons who would like to know more, please feel free to ask other users here or check my 'getting started' help here: https://hydrusnetwork.github.io/hydrus/ In regards to IPFS or using Hydrus File Repositories or something to effect better sharing on large scale, I think this will be increasingly possible over the next few years, but hydrus is not currently there. The IPFS integration (which also lays the groundwork for other small-file p2p support) is under active development now and is still mostly for advanced users who are already familiar with how hydrus works. There isn't an easy switch to throw and have doujins flow nice and easy again. In future, I'd love to do eventually add IPFS/hash/tag cross-referencing to be able to search IPFS or other file sharing domains using tags and file identifiers harvested from other sources. I am open to more opinions and thoughts. I am not big into the doujin scene, so I am largely ignorant of the actual mechanics and the overall size and shape of the problem to solve. I saw some posts saying Sad Panda was about 50-200TB. If there is a db out there with all the file hashes and tags and so on, it wouldn't be super difficult to convert it to hydrus tags and import it to a client. Current PTR is 620 million tags covering 31 million files, so if those are the sorts of numbers we are talking about, I think the current hydrus db can handle some jobs related to this at least.

>>13265 >>13258 Yeah, try system:filetype in a regular search page. You can mix it with any other tags or whatever.

>>13263 I don't use derpibooru–I assume there is a derpi login script on the github that another user made? If you have imported it ok, go network->downloaders->manage logins and find the entry and click 'edit credentials'. Put in your username/password and hydrus will try and login as you the next time you make a request on the derpibooru domain. Most of the downloaders in hydrus download as if you were browsing. Again, I don't know if the derpi login script (and accompanying downloader you have) works on an api key, but I imagine it just uses user/pass and simple gallery/file post page requests like most of the others.

>>13269 What's the problem? Did some go missing?

>>13307 ipfs is the solution, and it's being developed, but the timeline is very long and not likely to come out any time soon.

>>13281 If you like, close the client, go to install_dir/db, and enter this: .open client.db
SELECT dump_name, datetime( timestamp, 'unixepoch' ) FROM json_dumps_named WHERE dump_type=13;
.exit That'll give the timestamps of the x most recent 'last session' and 'exit session' timestamps, just in case any are left over from that 07/23 period. It probably won't add anything more useful than that log, but you never know. It depends on your idle settings, but since tag repo processing kicked in two and half hours after boot there, my guess is they opened the client and then for a period they didn't move the mouse, probably starting about 1pm or so, then moved it again at 1:30 to cancel processing, then left it alone again. I assume your client had caught up with the PTR by the 3:41 processing job since it stops there, or maybe they were continually moving the mouse until 18:40 looking through all your stuff, maybe, and then closing the client safely. If your session does not seem to have changed in your boots since then, the worst the guy probably did is click through your open tabs. If the files (or empty pages) are the same as before, he didn't hit f5 on anything. In an unrelated note, hydrus runs well from an encrypted partition. Ciphershed is an easy-to-use solution: https://www.ciphershed.org/

>>13333 Shit, should have said, 'run the sqlite3 executable' in the db dir. That'll give you terminal that you can paste those lines into.

>>13299 It is generally a pain in the ass every time, since it depends on what the exact file format is, and regex is not the most pleasant tool to work with, even if it is excellent. For commas, something like this should work: (?<=parentfoldername\\).*?(?=,)
(?<=, ).*?(?=,)
(?<=, ).*?(?=\.) But I am a bit rusty.

>>13301 No simple merge system yet. If you only want to move files and tags, it isn't super difficult–basically, export all the files and reimport them to the new db, and hit help->advanced mode and use the advanced service-wide update button on both clients' appropriate local tags panels on review services to export and import a Hydrus Tag Archive of your tags. If you need to move ratings or complicated (i.e. other than 'everything is archived) archived status, there aren't easy solutions. For inbox/archived, you can split it into two jobs, but ratings have no import/export yet. Let me know how you get on. I can figure out a hacky fix to transfer the data or maybe some simple import/export finally. Subscriptions and several other objects should be exportable and importable from their dialogs, either to clipboard or png files.

>>13302 The menu that has file | pages | database and so on? That should not be removeable in hydrus, but perhaps your OS or Window Manager allows you to clear/hide it? What OS are you on? Can you take a screenshot of what you see? The shortcuts system is in a mess of several different systems. Some is hardcoded. The new system is available under file->shortcuts, which you will be able to check out once we get your menubar back!

is there a requirement needed for ubuntu lts minimal to get the server to run? the client appears to run fine but getting an unhandled error on the server when i close it from the command line. end goal is to move the ever growing db off my pc and onto this box that has way more storage. when its running i'm not able to connect from another computer and the firewall is disabled ubuntu. Unhandled Error Traceback (most recent call last): File "threading.py", line 914, in _bootstrap_inner File "threading.py", line 862, in run File "site-packages/twisted/internet/base.py", line 1267, in run File "site-packages/twisted/internet/base.py", line 1276, in mainLoop — <exception caught here> — File "site-packages/twisted/internet/base.py", line 902, in runUntilCurrent File "include/ServerController.py", line 407, in TWISTEDDoIt StartServices() File "include/ServerController.py", line 333, in StartServices HydrusData.Print( 'Starting services\u2026' ) File "include/HydrusData.py", line 970, in Print print( repr( text ) ) File "include/HydrusLogger.py", line 159, in write self._log_file.write( message ) builtins.ValueError: I/O operation on closed file.

Hi to all, There are any bad side effects by installing hydrus (and it's archive) into a share program folder like mega or dropbox? I wanna try to do this in order to access and update my archive from multiple computers

>>13312 Feeling sad about this, also https://8ch.net/animu/res/111544.html#q112288 (not using crosslinks to prevent bumplocks)

>>13347 Threw a bunch of python modules that it was missing. still not run able. Unhandled Error Traceback (most recent call last): File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/home//.local/lib/python3.6/site-packages/twisted/internet/base.py", line 1272, in run self.mainLoop() File "/home//.local/lib/python3.6/site-packages/twisted/internet/base.py", line 1281, in mainLoop self.runUntilCurrent() — <exception caught here> — File "/home//.local/lib/python3.6/site-packages/twisted/internet/base.py", line 902, in runUntilCurrent call.func(*call.args, **call.kw) File "/mnt/hydrus/hydrus network/include/ServerController.py", line 407, in TWISTEDDoIt StartServices() File "/mnt/hydrus/hydrus network/include/ServerController.py", line 380, in StartServices HydrusData.Print( 'Services started' ) File "/mnt/hydrus/hydrus network/include/HydrusData.py", line 970, in Print print( repr( text ) ) File "/mnt/hydrus/hydrus network/include/HydrusLogger.py", line 159, in write self._log_file.write( message ) builtins.ValueError: I/O operation on closed file.

>>13312 Alright, mate. I understand that Hydrus infrastructure is not quite ready for something like what I have in mind, but as I understand, Hydrus is being developed collectively, isn't it? So what I'm going to do is make a thread to rise awareness of this, so people can have it in mind and help with the development. All you said so far sounds correct, and I have the tags saved somewhere. Thanks for your reply.

>>13352 Nevermind i'm just going to move it to an external drive

>>13337 Worked like a charm, thank you! I only needed the tags and the file associations so that was very easy.>>13337

>>13338 I'm on Debian, using 2bwm and bspwm mostly. I wasn't able to find a way to restore the menubar, but I was able to export all my files with their associated tags and re-import them into the new version of Hydrus, so all's well that's ends well. When I did have the menubar missing, it looked like this. It wasn't an environment thing because other GTK programs had the menubar present and when I tried to copy db into the new Hydrus release the menubar disappeared there, but before I did so (and now, after exporting, deleting the db files, and re-importing) the menubar was present.

>>13352 >>13347 >>13357 Thank you for this report. Another user told me that 361 is having trouble starting services correctly–they only seem to start for him when he hits ctrl+c, which then quits the program. I did change this this past week, and it was all fine on my test machines, but something is messed up with the initial event queue for certain platforms. I am going to give it another go this week. Please let me know if 362 works for you, or, if it is convenient, see if 360 works ok.

>>13350 Yeah, I generally recommend not putting the actual db on a remote system. SQLite likes to have some clever hard drive locks, and these remote locations tend to offer 'fake' versions of the lock commands if they even offer them at all. It might eventually cause malformed errors, or it might just not work for you at all. It is also good to have the actual db on a local system for latency purposes. You can try it if you like, and I'd be interested to know how it goes, but make sure you have backups of everything, and don't try to run two separate clients from two separate computers on the same db at the same time. I know several users who enjoy putting just their media files on a NAS or even in the cloud. Keeping the db files and thumbnails locally but putting files on remote storage tends to work ok. Here's some related help talk about your options: https://hydrusnetwork.github.io/hydrus/help/database_migration.html

>>13353 I'm afraid it is just me, one lad. I have trouble working in teams, so I have a strict policy of working on my own for hydrus. I am happy to help other Anons with their various situations, but I am always drowning in work, ha ha. I can never get things done as fast or well as I'd like. That said, the overall push of the past couple of years has been to make hydrus more 'platform' based with the completely user-configurable downloader system and the new Client API. This lets other users fix their own problems without having to wait for me to hardcode-fix the danbooru downloader or whatever. I'd like to continue this trend going forward.

>>13360 Thanks. That's odd. On my end I just make a MenuBar object an attach it to the main frame, and then wxPython and your Window Manager are supposed to handle the rest according to native controls. Some Window Managers put it up on the main OS frame. My best guess is 2bwm was trying to put it into a location wxPython couldn't handle and it bombed out. Increasing compatibility for different Window Managers is a constant fight, so please let me know if you discover anything new here. >>13359 Great, thanks for letting me know.

If I were, for purely cosmetic reasons, to change the current media destination's location drive letter (it's the same drive, but now E instead of L) how would I go on about that? I can't really migrate the database as it's the same drive. Just tell Hydrus where to find it again via adding a "new" location?

I have a really old version of hydrus. If I update it will I lose all my tabs?

Hey, Thank you for your hard work. This is an amazing tool. My question: why not upload the tag database to something like megaupload/mediafire and lets us import it that way, instead of through chunks of 64mb/day? Is there something preventing you/anyone from doing this? Of course it wouldn't be up to date but that would eliminate a large portion of the waiting from our side /bandwidth from your side. (I also asked this through tumblr, feel free to ignore my question there)

>>13375 Already been done here: https://cuddlebear92.github.io/Quicksync/index.html#two In Hydrus use services -> import repository update files

Oh, got another one: Is there a faster way to get a single file out of hydrus than right click > share > copy > image (bitmap) if I want to post it somewhere?

>>13374 Ideally no. Theoretically, it should be fine™. A bunch of stuff will change, like all those thread watchers will become multiple watcher pages each with one page, but nothing should break, hopefully. There's a max page limit of about 145 or so, now, and I am not sure what yours currently is, so pare down your session if it is larger than that. The main thing is make a backup before you do it. If it all goes wrong, you can always roll back. I think we are talking more than 100 versions. The client won't let you update more than 50 versions at once due to bitrot (I can't guarantee the update will do anything). I recommend trying 10 versions at a time. If you are on 250, try 260 and boot it. If no errors, try 270, 280, and so on. If it breaks, try 275 or whatever, half-way between your jump. Let me know if you run into trouble. If you are a madman, you can try a couple of 50s or whatever, but whatever you do make sure you have a backup before you try. I know one user just did like 320->360 and he said it was ok, but this is a complicated subject and every client is different. >>13375 >>13376 Yep: https://hydrus.tumblr.com/post/186658823864/hey-there-thank-you-for-your-amazing-work-i-was Let me know if you have any problems. >>13377 I think you can set up a shortcut for that, but it might only work in the media viewer. Try file->shortcuts->media and add a new shortcut for 'copy_bmp'. However, if you are uploading to 8chan or any other normal site with an upload box, just drag and drop the thumbnail to your browser. It is a lot nicer to send the real file rather than a bmp/png copy. If you are trying to DnD to discord, try turning on the discord BUGFIX option under file->options->gui.

>>13376 >>13379 I see, thanks!

Trying to add the public tag repository and I don't see a place to put an access key. Any idea what I'm doing wrong?

>>13400 Scratch that last question I guess, figured it out, but now it doesn't want to connect. Turned off my firewall momentarily to see if that was the problem, but it's not.

For someone with a ton of images is there some way for me to limit my inbox search to, say 1000 images so everything doesn't slow way down trying load all the images in the inbox?

From /v/'s draw thread.

>>13403 someone had requested a mascot for the software.

>>13401 >>13400 just do the option on the help menu to set it up automatically for you

>>13405 I did for both before coming here, and it still wasn't connecting.

>>13402 System:limit is what you want. Once you click on it you can specify how many files to limit it to.

>>13407 I figured it out, I was using an older build. New version works just as advertised. Also seems to be downloading only 6000+ tags, will that number increase?

>>13419 Also, I have all of my images sorted into fairly general but still useful folders, is there a way to have Hydrus keep those images assigned to those folders while uploading?

I am very slowly bumbling my way through learning the program and figuring out how to use it properly, and only seven hundred or so images in did I find the automated tag lookup script in the options menu. I noticed, however, that the script does not also pull the urls from the pages of the art, as the more roundabout way I was using does. Is there any quick and easy way to make it so the urls get pulled and attached to the image along with the tags as well?

>>13420 Nevermind, figured that out too, turns out it's an option in file>import folders.

>>13403 Hey, this is great! Here's some other ones made by users from a while ago, mostly just having a giggle. When previously asked 'What should a hydrus-tan have?', I've generally said it should have an error somewhere, or long UI hangs, ha ha ha. Making her thick works with hydrus having a bloated install and db, I like it.

>>13421 That's a good idea. That tag lookup script stuff was the prototype for my later downloader engine overhaul, and the way it fetches and parses its results is not great. I'd really like to rework it to use the newer parsing system, at which point it could definitely associate URLs in a sane way. That whole system is also pending an eventual throttled 'auto-check' system that you'd be able to queue up and have work in the background in some sane way that didn't end up eating bandwidth and CPU. Best way to associate URLs right now is to run a gallery download. A bunch of the site parsers now pull md5 or similar hash and recognise when the client already has the file (and hence don't end up downloading it again, but will get tags etc…), so they work a bit like the lookup scripts anyway. You might like to do a test on a site you like for an artist you like and see if that solves your tag&url problem quicker. Help on this if you are new to it: https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html

>>13431 Thank you for your attention! That is what I have been doing, feeding the hashes of the existing files I'm importing from my own folders through the downloader to get their tags and urls, the only problem for me is that the process for that is being excruciatingly slow for me, taking upwards of two hours to get through about 150-ish files, meanwhile the tag lookup is nearly instant. I should assume that is purely a connection problem on my end, yes? And also thank you for all your documentation, it has been an immense help so far.

Is there a way to delete unused tag info? Hydrus's two .db files sum to 19.5gb filesize, while my actual images are only 4.4gb. It seems like it's storing a ton of info about tag associations for images I don't actually have. Is there a way to tell Hydrus to go through and prune all data that doesn't match to an image on disk?

Forgive me if I've missed any explicit support for it even, but is there a workable way to parse and download images from Pillowfort blogs, akin to Tumblr support? I ask because a few artists have moved there ever since Tumblr's porn crackdown.

>>13432 Ha, sorry for insanely late reply. Since you see normal tag searches go fast, I think your connection is fine. There is a built-in limit for 'gallery' URL hits in hydrus, which might be what you are running up against. Normally, I strongly recommend keeping this limit in, as it makes regular booru searches work at a sensible pace, but if you change the setting under options->downloading, under 'gallery downloader' section, the 'additional fixed time…' setting, which is likely 15 or 30 seconds, take that down to 3, and these url queries, if they are indeed waiting on that, should go through quicker. I wouldn't recommend taking it down to 1, but you can try if you want. Gallery hits tend to be CPU heavy for servers, so hydrus tries to also be polite by not hitting them too frequently. Hash lookups may be ultra-fast serverside, but if they have set up bandwidth banning based on frequent heavy requests to the gallery script, you might get hit without meaning to.

>>13445 If you are synced with the PTR, unfortunately you need to get all the PTR definitions and data as well, which is about 20-odd gigs now. As you guessed, it gets all the mappings (~700 million at this point!) the server has, so all computation happens clientside and with complete privacy. This tends to be less of an issue if you have a 1TB+ total collection, but for your situation, it is not great. If you wish, you can delete the PTR, which will clear most of that data after a vacuum call. It will not delete master records, which I have a planned solution for as per here, which I answered earlier today: https://hydrus.tumblr.com/post/189534434774/ive-been-using-hydrus-and-trying-out-the >Unfortunately, there is no way to clear out ‘orphan’ master definitions for hashes and tags yet. This is becoming a bigger problem with various larger dbs, so I expect to have it in in the next couple of months. I know what I need to do, I just have to write a decent maintenance routine that won’t gobble up too much CPU. The ideal is to have it work in the background, so you won’t have to do anything–it’ll just shrink your db naturally over time.

>>13446 I haven't got one in by default. The main repository for user-made download scripts is here: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System/All-in-Ones/Single-Sites Unfortunately, I do not see one for Pillowfort there. I just checked out the site and saw it is in beta and needs a registration fee. Let me know if I missed a way to view as a guest. If you have a login, do you know if they have an API to talk to? Is their browsing experience similar to tumblr? I do not make downloaders myself any more, but if you could provide some example html/json/xml data for the Pillowfort equivalent of a booru's gallery tag/username search and post pages, perhaps the guys who do make downloaders now can figure something out. If this is a bit technical for you, feel free to get into the discord and see if one of them will walk you through it.

So im getting disk I/O errors because my hdd is presumably dying on me and I neglected to make any backups of my database. I move the entirety of the db over to a new drive successfully except for client.mappings, I get errors whenever I try to move or copy it. Am I fucked? Also the captcha on this chan is fucking cancer holy shit

>>13450 Also I have tried cloning it in sqlite, seems like its getting stuck at current_mappings_10. The program also still works fine, no tags seem affected though I dont really know what client.mappings handles. The only reason I noticed something was wrong was because I suddenly couldn't process repository updates.

My yande.re tag parser has disappeared for some reason. Can anyone post an import file for me?

Is there a way to re-import files from my collection at mass in order to change where their tags are stored? I made the mistake of saving all import tags to just the public repository, now im noticing now that a huge amount of images are getting tagged completely incorrectly by it so now everything is jumbled. Therefore I just want to make a new local tag system for specifically the imported tags but re-importing everything manually would take weeks.

>>13450 I responded to you on Endchan. https://endchan.net/hydrus/res/10.html#432

>>13455 Try this: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/raw/master/Download%20System/All-in-Ones/Single-Sites/yande.re-tag-search-2018.09.22.png

>>13459 I am not totally sure about your problem here. When you say to re-import the files to get their tags again, did you add tags to your files when you imported them? Was that from regex filename parsing, or neighbouring .txt files, or something else? Or were the files downloaded, and these tags are parsed from boorus? If you were to do this the slow way, what would you be doing? services->tag migration may be of use here, but if things are too jumbled, maybe not.

>>13464 >>13463 >>13465 d8ca57 is me, fuck this multiple imageboards with different BO trip systems, ha ha ha.

>>13465 Parsed from boorus mostly. The slow way would be to manually download the images again using the urls that are saved with them, one at a time.

>>13464 Thanks! This works when I do a test parse, but when it comes to actually importing a new image from yande.re it's not pulling any tags for some reason. I double checked that the url class link was correct, but still no luck.

anyone recommend a good external viewer for mac? don't really know whats good

How could I go about pulling "date taken" data from my pictures and get that converted to a tag?

I'm going to merge two different databases together with one being on an old hydrus version. Is just exporting all the files them importing them with the tags enough or is there a better way to go about it?

>>13480 that's probably the most straightforward way of doing it

>>13471 I got it. It turns out that my default tag import options had been reset at some point.

Here's a bunch of questions since I just got a SSD and cleaned my HDD. First I was thinking of migrating everything(hydrus, thumbnails, DB files) but the actual files to the SSD, but I'd rather not bloat it straight away since it's a really small SSD. Is there any way to check the size of everything before moving? And about merging DBs, I remember reading something about having different sets of tags in the same DB, is that a thing I can do and I'm just stupid or has that not been implemented yet? Can I hide a set of tags by default? And is there a way to apply a certain tag to every file currently in the DB other than manually? Thanks in advance. 8ch is shitting the bed as usual, pls don't doublepost

I wonder if it would be difficult to create a thread image ripper like the ones for 4/8ch for other *chans running tinyboard + vichan or whatever 7chan is running? Could one just edit the 4chan one or is it a more involved process?

Anyone have a GUG for 4plebs? It seems similar to an imageboard setup but watcher, URL, and simple downloader all fail, yet the full images are there when I click manually and are just stored on its own CDN like on 4chan. Or is there some workaround?

Is there a way for: -automatically tagging like "page:1-page:n:" for downloading from booru images, not only imported from local files -transferring part of the files to another hdd (for example large films) -handling external subtitle for video

>>13467 Ah, yeah. Hydrus does not remember the origin of every tag, so you can't do a local-only retroactive mining of tags based on tag source. Best way is to set up a new url downloader page and feed the urls in again (doing a right-click on a big ctrl+a of the files you want should let you copy all urls that match a certain site, and then you can paste back into the url downloader). In order to get the tags again (normally hydrus will just skip urls it has seen before), you'll want to set the tag import options to 'get tags even if urls/hash already known'. Do a test of a dozen files first here, to make sure you have it all set up how you want, and then do a bigger batch.

>>13479 Hydrus does not read EXIF data at the moment. I plan to add this sometime and add search predicates and tag conversion for it. I do not know when I will add this, so if you want to do it now, you will have to use an external program to pull it before import and then convert the tags you want to a newline-separated 'sidecar' 'filename.txt' file to import beside the file. I know some users who have done this, although I am afraid I have forgotten the name of the EXIF extracting software they used. I think they had a system where the software converted to xml files, and they used a script to convert that to newline-separated as hydrus understands for import. If you are not very technical, it may be a pain in the ass, but I am happy to help if you want to give it a go.

>>13480 >>13481 Yeah. You might want to do it in two batches: one for inbox files, one for archive, so you can quickly regenerate that information in the destination. If both clients have services->tag migration, that's the best way to import/export tags atm, using Hydrus Tag Archive files.

>>13484 database->migrate database has good estimates of db size, ok estimates of media size, and ok estimates of total thumb size. If the SSD is very tight, you might not want to move the db files–they need a drive with free space to work well. I recommend 1-2 times as much free space as the size of the largest of your four db files. This is no big deal if you don't sync with the PTR, but if you do, your client.mappings.db is probably 18GB or so. You can now create multiple local 'my tags' services. Hit services->manage services to add more. The new services will appear in any place you can add/edit tags as separate panels or tabs. services->manage tag display is a new-ish panel that lets you hide tags for single services or all services. There is no way atm to apply a tag to all files in a file domain. I recommend you set up a page with: system:limit=1024 -the_tag And then do several rounds of f5-ctrl+a-f3-ctrl+v-enter-enter. You can push the system:limit a bit higher to do it in larger batches, but you'll take an inefficient CPU hit dealing with too many at once. If you have 100k+ files, this may be too much work.

>>13485 I keep meaning to knock up some quick parsers and url classes for the bunkers, and then I do not find time for it. It would not be super difficult for anyone familiar with the download system. Most imageboards offer good APIs or nice html, and as you say, if you have a parser for one vichan board, you probably have support for 95+% of them running the same software. 8kun is difficult because of the DDoS stuff. >>13486 I don't know anything about it, I am afraid. How do they fail, is it a 503? If they have aggressive CloudFlare protection, this is a common cause of hydrus not being about to get files when a browser can.

>>13488 - Not yet. I would like to add this some time. - Try database->migrate database to set up multiple storage locations. It allocates files across multiple locations randomly according to set location 'weights', so if you just have a couple of multi-GB files, you can't guarantee it will be smart about putting them in one place, but as your collection increases in size, it will be more accurate. - No support for this, unfortunately. I am not sure there will be a neat solution in the near future. You are free to do what you want, but I recommend you not use hydrus for movies. I designed it for imageboard-style files, where you need to keep track of hundreds of thousands of smaller files, so most of the workflows and maintenance timers are built around that. Files that have sidecars just don't work well with the hydrus storage system.

is it possible to delete the deleted files record for specific files, rather than the entire database? I know I can do the latter via `services > review services > local > files > all local files`, but I want to keep the whole deleted files history, except for maybe ~120 files worth.

Alright I think I'm an idiot. I'm trying to download the images from an artist on pixiv, and despite having other pixiv downloads that work this one just doesn't and I can't figure out why.

I want to use Hydrus to manage an existing collection of pictures downloaded from boorus, and nothing else. If I only want to get tags from those boorus instead of importing the entire PTR, is this https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/wiki/2.-How-to-fetch-tags-in-bulk-with-md5-hash the best way to go about it? There was also something about Hydrus tag archives in the manual, but the archives linked there are several years out of date.

I would really like to use hydrus, but I feel I'm losing the browseability of folders. How do you "browse" certain core tags easily? Like, right now, one folder I browse frequently is based on specific characters where the directory structure is: "my-images/series-name/character-name/". I've imported a bunch of these folders and given them tags, but now I miss the ability to see, at a glance, which series I might want to browse. I figured out that in hydrus I could mark certain tags as "favorites", which is a start. With my old folders I could quickly navigate the hierarchy using the enter+backspace keys to go up/down and arrow keys for moving around, and I can't do that with hydrus tags. It's also clear that the list of favorites would be unwieldly at some point. Let's forget the hierarchy for a moment. Each character is a nice conceptual "album" and it'd be great if I could just browse my "character" tag and easily switch from one character to another. I can double click on a favorite tag to bring up that tag search, but if I double click another tag it does an "AND" search…I first have to navigate to another part of the window to remove the first character's tag. So I'm stuck. I really want to move to tags though so I can easily browse by artist or other tags which cuts across my "/series-name/character-name/" folder structure. But I want those tags to provide the same browseability had I used a "/artist/character-name/" folder structure.

>>13504 I personally have a bunch of tabs set up with tags to organize some my db without having to search every time. For example, I have a blanket tab for all 2d art which then has other tabs in it that are tagged for whatever else, including an Artist tab that then has a bunch of tabs that are organized by artist. It wouldnt be hard to do the same for what you want, though it is maybe a bit less convenient than a simple folder structure though you gain way more than enough to make up for it. Speaking of, a feature i'd really like to see is the ability to add a "master tag" to a tab so that that tag is required for all searches in that tab and all others that are in that branch. Would make it a bit less tedious to organize that way.

>>13507 And though im not sure its within the scope of the program, actual folder-like functionality would be great to see. The way I would imagine it would essentially just be creating tabs and giving them a presence in the actual gallery screen instead of just the tab bar, while being able to navigate through them like an actual file explorer. Organizing everything through tabs is kind of hard on the eyes and awkward to use if you have a bunch of them, so this would be a godsend. I'd probably lose some time going full autist and setting up my own folder structure to organize absolutely everything, though.

>>13510 The way I would imagine it is that you can specify a folder structure in a separate input box, below the input box for the search string. So you might say: folders: <namespace1> / <namespace2> / <namespace3> which would automatically populate a virtual folder structure with the results of your search. Items that were returned by the search but don't have a tag for namespace1 would be put inside a 'misc' folder at the bottom. (and recursively, there would be such 'misc' folders inside each level of the hierarchy)

Is there any way to make the duplicate filter value pngs over jaypegs? I tried to set it to negative but it only goes as far as 0. Also, since when was this implemented? I just found it now when searching for backup options.

I accidentally deleted a tag, my intention being to replace it. Using the show deleted option in the tags menu, I can still see the tag, but it isn't applied to the files it once was. Is there any way to re-apply a deleted tag to files?

new to hydrus, is there a way to change a namespace's color? I added two new namespaces recently and they both were the color of one I already had.

I am unsure on whether to post this here or in the bugs thread, but after updating the program to the latest release, I noticed that I no longer can press Ctrl+C to easily copy the file's path. However, the hardcoded shortcut section still references that as the shortcut to copy files into the clipboard. Is this change by design, or is it truly a bug? >>13552 If I'm not mistaken, you can do that by going through File > Options > Tag Presentation

Where is the place with the most current info on getting downloaders to work? I am searching but I keep finding outdated info. I am not good with computer. I just want to download every single picture of my waifu on the entire internet.

Hey lads, apologies again for lateness. New schedule calls for me to check this four times a week, let's see if I can stick to it this time. >>13496 Under options->files and trash, turn on the advanced file deletion dialog. It provides an option to delete a file without leaving the deletion record. If the files are already deleted, there isn't a way to do this I don't think. Maybe you can do something clever like searching for the files in 'all known files' domain and then re-deleting them. You know what, I'll make a job to add a 'clear deletion record' thing to the right-click menu for these files.

>>13497 As I understand it, Pixiv now prohibit nsfw results from non-logged-in searches, and the hydrus pixiv login script silently broke some weeks ago, I think due to them adding captcha. Please use Hydrus Companion to log into pixiv now: https://gitgud.io/prkc/hydrus-companion

>>13499 Yeah, that md5 lookup process is decent. The other solution is the Public Tag Repository, which I specifically designed to solve the problem, but if you are new to hydrus, you might want to hold off on that for a bit. The PTR is now a sledgehammer, and if you only ever want to tap out tags for a couple thousand files, it might not be right for you. >>13531 I folded that into a changelog a little while ago, I think. I basically just exposed the internal values I was using for the score to users. I will make it so you can value pngs over jpegs. Out of interest, can you talk a bit about your situation where that is useful? Is this for, say, a set of patreon rewards, like 'raw' ultra-high-quality png versions? In general, most users I have talked to about this tend to have many more situations where the png is some clipboard.png imageboard copypaste post of the original jpeg, so they generally prefer jpeg to be the higher weight be default.

>>13546 Some deleted tag storage and workflow is borked, so let me know if this doesn't work: Open up the file(s) in manage tags dialog, set to 'show deleted', and then double-click the 'deleted' tags that have '(X)' after them. The double-click should apply a re-add action. I think you can also just try to add the tag as normal, and the delete will be overwritten. I am assuming this is for 'my tags' tags, and not something like the PTR. Undeleting tags for a tag repository is more complicated and really needs a janitor.

>>13553 Ah, thank you for this report. I am pretty sure that is old–I think all the copy stuff is now under file->shortcuts->media, with action names 'copy_(bmp/file/path/sha256_hash)'. I'll clear out that old text. >>13555 Here's the main repository: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System/All-in-Ones/Single-Sites Open up Lain in network->downloaders->import downloaders and drag those pngs on her. It isn't always perfect when there are object conflicts, but it generally will import everything automatically.

So I have a couple questions here, first I was wondering if its possible to automatically add a specific tag to every imported file that DOESN'T have another specific tag thats parsed from wherever im importing the file. For example, I'm trying to categorize my collection by 2d and 3d files, 3d is usually tagged on boorus but 2d usually isnt so its kind of a pain doing it manually. Second, how do I completely delete the ptr so that I can re-download it? Im fairly sure my ptr is corrupted because a lot of tags are mixed around where they shouldn't be. I've tried deleting it from the manage services menu and making a new one but that just processed the repository updates that ive already download again and doesn't fix the jumbled up tags. Dunno if re-downloading it would even fix the problem but I figured i'd try.

Is there a way to clear all previous deletions without reason? I've done some fuck ups in the past and now that there's deletions explanations it's a good time to unfuck my hydrus.

>>13569 I don't think you can do this 2d/3d tag logic atm. My best suggestion for now is to have a search page set up with: -2d -3d (possibly system:known url=somebooru.org) i.e. 'all the files that are from site x that don't have 2d or 3d) And then once a week or so go: F5 Ctrl+A F3 Type '2d', hit enter Ctrl+R On it, to do this manual job in a single batch. For the PTR, I am generally confident that your PTR is not corrupt, but the tags were applied to the file incorrectly due to human error or a mis-parse. Sometimes, for instance, a website changes its layout and a bunch of 'favourite' or 'users also liked these tags' get sweeped into a bunch of files, which slowly gets corrected on the hydrus side from user petitions and janitor approval. Simple mistakes are fairly common, and the chance of a PTR file being corrupt and that not being noticed by the client during processing is infinitesimal, so my bet is on the former. The client knows exactly what the update files should be, and if there is a single byte difference from what the server has, it throws all sorts of errors. If you notice completely the wrong tags, with no correct tags, on many many files, particularly if it is the same kind of swap (like everywhere you expect to see 'series:touhou', you actually see 'character:samus aran'), it is rarely possible that your db has switched up some master tag definitions, possibly due to a hard drive fault. Can you provide an example hash of a file that has bad tags, and what tags you see on it? Right-click it and hit share->copy->hash->sha256. I'll check it my end and let you know if I see radically different tags.

>>13570 I do not think so. If you are feeling brave, and you are not connected to any file repositories, and you absolutely have a good backup in case this goes wrong, try running the sqlite3 executable in your install_dir/db directory and copy/pasting each line here in turn: .open client.db
delete from deleted_files where hash_id not in ( select hash_id from local_file_deletion_reasons );
.exit Then boot the client. Do a sanity check, a test import on a previously deleted file or something, just to make sure everything seems good. Hang on to the client.db backup just in case this screwed something up and we need to reimport its old data. Oh, and run help->debug->data->clear db service info cache, which will force your client to recount deleted file numbers next time you look at them.

>>13563 >I have talked to about this tend to have many more situations where the png is some clipboard.png imageboard copypaste post of the original jpeg I scrap boorus and yiff more than image boards, so pngs are usually actual pngs instead of Trojan jaypegs. I like pngs better since they're usually cleaner even when scaled down, I only opt for jpgs instead if the resolution difference is big enough to make it worth it.

Do parent tags do anything if you have "auto-add entered tags' parents on add pend action" disabled? I disabled it because I don't like how it will add them if you enter the child tag, but it won't remove them if you undo it before clicking apply (for example, if you entered the child tag by mistake) and that's caused me to apply non-applicable tags to files on a few occasions without me noticing. I figured that turning the option off would instead add the parent tags once you actually click apply, but instead, it seems to just ignore parent/child relationships entirely now.

>>13576 Thanks, this is interesting. >>13597 Yeah, at the moment it only applies on add/pend actions, which means 'add a tag to my tags' or 'pend a tag to a tag repository'. 'delete a tag from my tags' or 'petition a tag from a tag repo' actions, which remove tags, do not apply parent associations at any time. The reason for this is hydrus generally doesn't know why a tag was added. So if you had the situation of: add character:asuka (series:evangelion added as a parent) add character:rei (series:evangelion would be added, but it is already there, so no action) 'oh, actually rei isn't in there, I'll remove it' remove character:rei series:evangelion is not touched You don't want to remove 'series:evangelion' because it could be there for legit reasons, and the code isn't clever enough to work through the human reasons for it being there. Potentially, I could improve this logic by, say, only removing a parent if there are no valid children, or adding a history to manage tags dialog so it remembers that a tag was added by parent implication and not direct human action, but this is a complication on already quite complicated behind-the-scenes logic. I leave it to the human for now and will revisit it when the siblings/parents system is more intelligent.

Is there any way to export .jpgs files with the tags embedded in them so you could view said tags with windows explorer or android or other programs? If anyone would know a way to add .jpg tags from .txt files that would also be grand. Is there a way to disable certain tags/namespaces from being written to the .txt on export?

I'm trying to import https://www.deviantart.com/karbo/art/A-snack-in-a-trap-828646610 via hydrus companion but it's not working, any idea why? All other deviantart posts import just fine

Just added Hydrus Companion add-on to my Firefox, and I can't seem to figure out how to make the add-on extract tags from, say, Pixiv when sending the page to Hydrus. From what I'm seeing in the settings and advanced features it requires putting in your own regex or something, and I have no idea where I'd begin with that. Is there an easy way to just allow me to directly pull tags from pixiv when I send a page to Hydrus?

My Windows updated and I can't run hydrus anymore, what could it be? This appears on the logs. The second one is the error if I try running client.py through cmd

>>13623 I know there are programs that can do auto population and scraping of NTFS metadata, but I do not have any personal experience so I do not know any application names. Other lads who have done it needed to write scripts to convert .txt to/from xml files. Yes, unfortunately you cannot filter the tags exported to .txt yet. It is on my list.

>>13626 I am afraid it works for me, which is actually a surprise because I understood the hydrus DA parser/login stuff was having trouble recently, particularly with nsfw–can you say more about the error? What happens when you try to import, does it give 'error' or 'ignored'? If you check the import list on the downloader, does it have an error in the 'note' column? Can you right-click->copy and paste that here?

>>13629 Did you upgrade from a very old version? I forget which one it was, maybe around 330, I jumped from 32 bit to 64 bit, and those errors look a bit like that. The instruction back then was to do a 'clean install', which sounds simple but you absolutely have to do it correct: 1) Make a backup. 2) Delete everything in your install directory except the 'db' directory and its contents. 3) Install as normal. This clears out the old 32-bit dlls from your install directory while leaving your database intact. If you are updating from a very old version, more than 50 versions at once, you may have trouble even if you can get it to boot. If you know what version you were before, I recommend only updating about ten versions at a time. So if you were 325 before, try doing 335 and then booting the client. If that works, then try 345. And make sure you have a backup, just in case something goes wrong. Let me know how you get on, I am happy to help.

>>13629 Shit, I just read your post again–you updated Windows, not Hydrus. Sorry, can you explain a bit more about which version of Windows this was? I am not sure what is going on here at all. A clean install may help nonetheless. Have you ever set up hydrus to run in compatibility mode anywhere? Was it set that way maybe by the Windows update?

>>13639 I updated an older version of windows 10 to a newer one, so I don't really know, it doesn't seem to be running in compatibility mode. I tried doing a clean install and the extract-only zip but both give me the same problem. I have python installed and tried uninstalling it from my computer but that didn't really seem to fix it. All of the other software I had on PC works fine after the upgrade, the only one that gave me problems was hydrus.

>>13637 I attached image with that info to post >>13626 https://media.8kun.top/file_store/2bfe1c61cf7f184e676f43e5a082cbca9d9452d242558c3faf97f55bb1536527.png

>>13644 >>13637 oh right, actually all NSFW posts on deviantart get ignored like this, I have cookies though

>>13629 Okay, so I managed to fix it. I had to install the windows media feature pack ( https://www.microsoft.com/en-us/software-download/mediafeaturepack ) because my new windows is this N version. Just leaving this here in case someone else has the same problem.

>>13645 Ah shit, I just logged into DA in my hydev browser and got a 'we just updated our layout, here's a video to explain the changes' thing. I wonder if they have done what pixiv did a while ago, which is to roll their new layout out to usernames at different times, perhaps as part of a kind of A/B test. If the new layout breaks the hydrus DA parser, then it will start failing for users at different times as they get switched over. I am not sure if this is true though, since it only seems to be hitting you for nsfw, and I am still ok, I just tested it again. I am not a DA user IRL. I recommend you let these downloaders/subs get their 'ignored' status, to keep them in memory. Once we have this figured out, all downloaders should have a button or menu option somewhere to 'retry ignored', at which point you will be able to retroactively get the missing files here. It may also just be some weird cookie sync thing that'll fix itself in a couple weeks. Please let me know if you discover more about this, and I will keep an eye for more user reports to see if this is a more widespread issue. If I didn't say, have you tried clearing your hydrus cookies for DA under network->data->review session cookies and then resyncing with Hydrus Companion? Maybe there is an old 'hasn't clicked nsfw age-gate' cookie still in your hydrus that is tripping things up. >>13655 Thank you for this follow-up. I had never heard of 'N' edition before. I will write this down for future lads who run into the same thing.

>>13674 might be something with my firefox, looks like all cookies, not just from DA stopped being imported correctly (they're empty) but hydrus companion dev tried with fresh profile and it worked so something on my end probably

>>13684 nevermind, managed to import cookies correctly but still get the ignored status

>>12641 is there a way to query deleted files from the client API?

Hi, I have a few neewbie questions: 1. Just installed Hydrus, downloaded and imported 2 GiB of updates. Processing is almost finished by now, pic related shows the directory sizes in …/conf/Hydrus/. So, my virgin Hydrus with no media imported consumes 30+ GiB of disk space. Is it really that big from the beginning? Should I maybe perform some cleaning? 2. …/conf/Hydrus/client_files contains 4 GiB of files with the names like "da2e0e38098fc73502574fbba8832ce7a6f6a88c475ff33e53c152246b0e2afa". What's that? 3. Archive with the repository updates is called 1PTR_UPDATE_FILES-20200216T131032Z-001.zip, so apparently it's from February, 16. I expected that it should cover all the data I need to download, but after the import was done services→review services→repository sync→downloaded was about 70%. So I still had to spent few days waiting while the rest will be downloaded from the repository in 512MiB chunks. Is that a normal situation? Did I do something wrong?

Some faggot has spammed the "female" tag on thousands of unrelated images to shit up the PTR, any way to deal with him besides manually petitioning tags? At this point I'm pretty sure it's intentional sabotage and not just a broken parser script.

>>13690 Only what is available here: https://hydrusnetwork.github.io/hydrus/help/client_api.html I don't think you can do deleted files yet, and now I look, I don't think the file metadata call does file service status (like 'is in my files') yet. I expect this to be added as I work on the api.

>>13693 1) To keep the client fast, network resources low, and to improve user privacy, the Public Tag Repository shares all tags to all users. Essentially your client is mirroring all its data so all work happens clientside, on your CPU. The PTR has been running 8 years or so and now has about 750 million tag-file pairs, which is about 28GB. It will grow a little bit every day. Adding file import data to the client does not add much space. My IRL 1.7 million file client is 30.7GB total. Future improvements of this system will allow you to better filter what tags you get from the PTR, I estimate reducing total db size for users who want fewer tags about 67-90%. 2) The extensionless files are the update files your client downloaded from the PTR. They are kept in case the data needs to be reprocessed. If you are interested, they are zipped json, mostly very long lists of numbers. 3) Is this from QuickSync? I do not know enough about the QuickSync archives to talk cleverly about them. It may be missing some. I do not think you did anything wrong. The client will figure out what it needs and fix you up automatically by downloading anything missing and doing more processing when ready. As you did a very large processing sync all in one go, I strongly recommend you do database->maintain->analyze->full. It will take a bit of time but will ensure that your db is optimised and running fast when you start importing files. Normally, this optimisation happens naturally over time as a client processes the PTR over weeks, but the QuickSync archive tends to happen a bit faster than it kicks in, and now you have hundreds of millions of rows here, that can accidentally lead to extremely slow searches for a couple weeks unless you run that maintenance routine. Let me know if you have any more trouble.

>>13695 The main janitorial team for the PTR are on discord, if you would like to talk to them directly: https://discord.gg/3H8UTpb Best solution now is to petition the tags as you see them. If do not want to go to discord, I recommend you set a 'reason' for the petition with the basic problem here, and the janitorial team will get on it if they do not already know. There are not excellent janitorial tools for hydrus servers yet. As the PTR grows and more users use it, I expect to improve how all this works. At some point, users may well be getting their own access keys, so bad uploaders can be cut off. These tags could be malicious, but when I ran the PTR, accidents due to mis-parses or ESL (English as Second Language) confusion vastly overwhelmed any other problem.

Is there any way to force redownloading files? There's a few pictures that are broken due to HDD failure, but the only way to redownload them is to permanently delete the file then either import it or redownload from an URL ignoring deleted files.

How does Hydrus handle continuous downloading of resources with anti-chronological post order? Such as the boorus where the recent posts appear on page 0. I mean, suppose Hydrus has downloaded X pages and was stopped, in the mean time Y new images appeared, so now when it tries to resume from page X it won't be at the same spot where it left off. Is this all accounted for and I shouldn't worry when making my own parser?

>>13707 It checks the pages in order and saves the URL it finds to download. Accessing a page is pretty quick so it isn't any problem unless you stop at X page while there were X new pages of content, it'll ignore the old files but download new ones. Say there's 5 files per page, you download two pages and stop, while you're not downloading 15 new files get uploaded, it'll continue from that second page next time, meaning it'll skip the first 5 recently uploaded and download the Y files from then on, including the ones you already downloaded.

Is it possible to delete all PTR data from my local database? I just use hydrus to organize my reaction images and realized I don't really want 25+gb of tag data from the PTR on my hard disk

>>13706 Thank you for this report. For hard drive failure, there is: hit database->maintain->file maintenance->review scheduled file jobs, then go to the 'add new work' tab. You add jobs to files as selected by a normal search, (so sometimes you might want to say 'all jpegs' or something) but since your problem probably hit all files randomly, you probably want to queue up 'system:everything' (or if you do not see that, do this for inbox, then archive) and then hit 'run this search' to load those files to get the job. In the 'add job' section, make sure 'if file is missing/incorrect and has url, move file out and try to redownload' and then click 'add job'. Then, in the background, your client will check every file in your db that it is what it thinks, and if it is not, it will queue it up in a new download page. Depending on the size of your collection, this may take a very long time and take a whole heap of HDD and CPU since it needs to load and think about every file you have. You might want to break it into batches or pause/play it under that same 'file maintenance' submenu only when you are happy for it to go.

>>13707 >>13708 Yeah, if you are making your own parser, as long as the site you are parsing provides files in 'newest first' order, you are good. You only need to concentrate on pulling urls from each page and then any available 'next page' url. Subscriptions have a whole bunch of logic to think about this stuff and decide whether they need to get the next gallery page or if they have caught up to where they were on a previous sync. Then only requirement is that the site serves files newest->oldest like a typical booru. Note that regular download pages are one-shot and do not do this sort of syncing logic. You can have any kind of 'highest rated first' or whatever sort order and it doesn't matter–these just can't be turned into reliable functioning subscriptions.

>>13719 You can delete most of it. Hit services->manage services and find the 'hydrus tag repository' entry for the ptr and delete it. It may take the dialog a minute or two to close as it finishes the big delete job, so give it time if it hangs. Then, when that is done, it is usually a good idea to restart the client. Then hit database->maintain->vacuum->full, which again will take a few minutes, but will recover your hard drive space. Unfortunately, the 'master' definitions are not currently removed, so your db will still have a ~7GB client.master.db. I have a plan this year to write maintenance code to recycle this info and recover the space, but it is not in yet.

How do I set hydrus to aline my thumbnails again? Can't remember what update changed this but I remember hydrus was able to aline thumbnails perfectly whenever I clicked on or highlighted an image in the top row. I could never describe it good enough to ask but it became a little annoyance to me. Another annoyance, whenever I switch between pages/tabs. Say I selected a certain number of images that I want opened to another page/tab. If I delete any images for the new 2nd tab, the 1st tab will reset the scroll bar back to the top. If that makes sense. Before it didn't used to do that and just kept its scroll position like when switching between any other tab and now it doesn't. So basically I want it to keep my scroll position like before. Both these little annoyances started happening after the last big hydrus project/update so I'm not sure if their was a setting or something I missed.

I updated by about 20 versions to version 386. I remember there being some changes associated with the public tag repository around the time that I started my old version of Hydrus. I had been able to sync with it, then it stopped working and Hydrus notified me that I had to update to a new version in order for it to keep it synced. I have only updated two weeks ago so I have a lot of processing of the PTR to do, apparently. My questions are: What is the nature of this processing? What does a single update correspond to, and what does a single definition correspond to? Why does it only do a double digit number of rows per second?

Sage for doublepost. I looked into the help files, especially help/database_migration.html. Fucking hell, it was worth taking the effort to move Hydrus to my SSD and keep the media on my HDD. It's now processing 100x faster. I have some feedback for that help page and the process in general. It seems to me that moving your media files and their thumbnails away from the Hydrus install directory is a more common use-case than moving the SQLite database away from the Hydrus install directory, but that page places the 'informing the software that the database is not in the default location' heading first. I was getting confused while trying to figure out what to do, thinking that I'd have to pass it a command-line argument at some point. But if you set the non-portable locations of your media files and your media thumbnails and move those files, you can afterwards just close Hydrus, move the entire Hydrus directory from your HDD to your SSD, open it again, and it'll just work. Shouldn't that be the thing that is emphasised on that page, and the detail about moving the database to a non-default location should come last? I also found the thumbnail location override setting confusing, because the thumbnail directory path is listed in two places thanks to that setting, and because I don't see why specifying the thumbnail directory's path should work any differently to specifying the media file directory's path(s).

How do I parse this kind of JSON in my custom parser? The server returns either a photo object with type=photo and more data in a subtree by the "photo" key with URLs to download images, I can handle that fine. (first image) Or it can return a "doc" type object with "doc" subtree with an URL to download it. The doc can be any file, I only want to download it if has "ext":"gif". However the URL doesn't have the extension inside of it and I don't know how to do a check by a different key before passing URL to the downloader. (second image) I tried to make a Veto rule which looks up .doc.ext=='gif' and vetoes if it doesn't match, but then it won't download the regular photos, since they don't even have the .doc subtree.

>>12641 what's the best way to subscribe to an entire booru site? I'm training a resnet to score files based on my preferences so I'll need to do hardly any work. If I try to do this in a subscription will my client catch syphilis and die? I was going to just do sort:id:asc as the query in the gallery downloader and slowly raise the file limit, occasionally deleting the query when it becomes too big and incrementing the start id like "sort:id:asc id:>10000". Also, since I'd be downloading *everything* I wouldn't want to grab the latest stuff first and accidentally download something weird the moderators haven't purged yet.

>>12641 >>13755 also, regardless of how I setup syncing, I'm probably going to download deepdanbooru2019 first to speed things along and use the client api to import the urls + tags -> hashes

>>13752 >Fucking hell, it was worth taking the effort to move Hydrus to my SSD and keep the media on my HDD. It's now processing 100x faster. intel ssd optane 900p is literally 10x faster than an nvme SSD so if you gotta go fast you could throw your db on a $300 optane

easily the most 'tism pleasing program i've ever encountered. magnificent software is magnificent. also at the risk of sounding gay as shit the "Act from need, not toward purpose" line in the help pages was profoundly impactful. super nice little unexpected bonus, that. all my gratitude are belong to hydev on both counts. regarding questions, am i ok updating version 370 -> 380 -> 387 or should i hit 373 first and go from there? does the move to qt mean anything for the 10max update interval? next, i can't seem to get a downloader for 4plebs working. test parse all checks out. keep getting file ignored error 403 no matter what i try or how i tweak things. halp? and lastly, could you take a look at the code which manages tag parents/siblings via the context menu next time you're poking around in there? specifically the part that preloads whichever tag was right clicked on so that it shows up as already entered in the left-hand column? dunno if it's the preloading that does it or the fact that it's present on every tab/service but those mappings set up via the context menu have a tendency to end up getting cross applied to other services. kinda seems like maybe stuff like "both fields required for new pairing" and/or "don't auto apply uncommitted changes" is getting bypassed somehow.

I may be the world's worst googler, I apologize in advance. Has the nijie login script been updated lately? Since the recent changes to it none of the 3 I have are doing the job.

Is there any way to, when importing from a Twitter URL, to generate a "creator:" tag with the username?

>>13778 add this to the twitter parser
[30, 4, ["creator tag", 0, [27, 6, [[26, 1, [[62, 2, [0, "div", {"class": "user-actions"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]]]], 0, "data-screen-name", [51, 1, [3, "", null, null, "example string"]], [55, 1, [[], "parsed information"]]]], 0, false, "creator"]]


Has the OR and AND dialogue been replaced completely? If so, how would I go about using OR in a search? My old tabs using it are still around, but I cannot make a new search using it.

>>13779 Works perfect, thank you friend!

>>13781 hold shift while pressing enter on a search tag to add it to an OR search, or you can click the OR button on the right of the autocomplete popup for a interface for an advanced OR search.

Does anybody know how I can add the login details for e621? I've tried some Q&A threads but I can't find the details on how to do this. I tried the danbooru login script but it doesn't work, I get: The login for "e621.net" failed! It will not be reattempted until the problem is fixed. The failure reason was: Missing cookie "_danbooru2_session"!

What's with these files in my database folders that have no extensions? Adding the usual image file extensions didn't make my image viewer recognise the file as an image, and searching the hash directly didn't find anything either. Are they repository updates from the public tag repository? I did notice that I have 31034 files under 'all local files', 23018 under 'my files', 7979 under 'repository updates', and 37 under 'trash'. The first number is the sum of the rest. Speaking of which, how exactly do these repository updates work, and why do I have 7979 of them?

>>13786 >Are they repository updates from the public tag repository? Yes >Speaking of which, how exactly do these repository updates work, and why do I have 7979 of them? As new tag mappings are added to the repo, they server creates update files and they are downloaded and processed into your DB.

>>13784 Yes seconding this e621 is useless without login! HOW DO YOU LOGIN?!>>13784

>>13746 Should be options->thumbnails, "EXPERIMENAL: Scroll thumbnails at this rate…". It'll tick down one thumbnail height at 1.0. Also there is a thumbnail % visible scroll check, which might actually be what you want here–set that to 0% and it'll always scroll. EDIT: Seems like that isn't working right? I'll look into this in case it is a bug. Might have been Qt that changed that, or some other thumbnail scroll-to stuff I did recently. I have heard about the 'scrolling to top' issue. I did do some work on this,but I will check this as well. If you are on an older version than current, updating may fix this.

>>13751 tl;dr: Let it work, it'll take about a month of background work in idle/shutdown time to catch up. Read EDIT below. I don't have time to do full technical explanation, but: The tag repo sends all tags to clients, basically mirroring the server db to the client, so that all work happens on your hard drive and your CPU. Every day or so, the server generates compressed update files of all the changes in the past cycle. These are zipped up json, mostly lists of numbers. Then this portable data needs to be integrated into your client, lining up serverside file and tag definitions with what your client knows, and then adding the actual file-tag pairs so you can see PTR tags for your files. There are also auxiliary operations like maintaining the autocomplete count cache and other metadata. EDIT: Since you are getting double-digit rows per second, are you on an HDD instead of an SSD? The modern PTR has 700 million mappings, and does not work well on a high latency drive like a hard drive. If you cannot move your hydrus install to an SSD, you should pause it now as you will never be able to sync up to date and keep up to date. Info on migrating a db here: https://hydrusnetwork.github.io/hydrus/help/database_migration.html

>>13752 Fugg, should have read more. Great, I am glad you got yourself sorted. Thank you for the feedback on the help. That makes sense. I will make a job to give that help page and the UI a pass.

>>13753 I think you want to make a 'subsidiary page parser'. This is advanced and not well documented, but basically it lets you split a post up into sub-posts. This part isn't important for your case, but what is important is a veto on a subsidiary page parser only cancels the content of that subsidiary parser. So, something like: MAIN PARSER
photo content parsers
content parsers anything photo and doc have the same structure for, maybe tags
SUBSIDIARY PAGE PARSER
VETO if no gif
doc specific content parsers like url In this case: If you have a photo, it pulls the photo content, then runs the subsidiary parser, which terminates because no gif. = photo url If you have a gif doc, it pulls no photo content, then runs the subsidiary parser, which pulls gif doc content. = gif url If you have a blah doc, it pulls no photo content, then runs the subsidiary parser, which terminates because no gif. = ignored import status

>>13755 >>13756 I very strongly advise against downloading an entire booru. If you have not seen this relatively new document, I have written several of my thoughts, which are based on experience with many users, here: https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html That said, some users want to mirror for archival reasons, or because they just want a specific small niche booru, or they are madlads, if this applies to you: Most boorus will accept '*' as an input query, which will just return everything. You would do as you are thinking here, setting up multiple download page queries in batches, variations on 'id>x', to fill in all the gaps, and then when you are caught up, you would set up a sub for '*' to catch all new content thereafter. If the booru is big like danbooru, it probably runs too fast for a hydrus subscription to to keep up with reliably. My guess for a technical limit, for a client that runs 24/7, would be a booru that spikes during a busy period at more than 100 files an hour. I will again say getting everything is almost certainly impossible for any human to sift through properly. If you haven't done much archive/delete processing, I recommend you try doing 1,000 files. Time how long it takes to do properly, then multiply that by how many files you are expecting to get. If you want to chase spicy content before it is deleted, I would recommend you try to filter your query to catch that specifically, or find a different site to download from that simply is ok with hosting it. Maybe search for filetypes, if the booru accepts it, or 'does not have a creator tag yet' if the booru supports it, anything to help you separate the wheat from the chaff. The most productive way to set up subscriptions is for artists you like. You will soon be overwhelmed with that like everyone else, and all the happier that you don't have a 30:1 trash-to-good ratio. Let me know how you get on!

>>13759 Thank you, I am really glad you like it. I made it for other Anons like me, so we are brothers. You should be good on the Qt update step, if you haven't tried it yet. There were no special install instructions, thankfully. As always though, the best thing to do before a big update is just to make a complete backup beforehand. You can then go nuts and even try it all in one step, since if it all breaks, you can just roll back, no worries. For 4plebs, if you open the 'file/watcher import status' window on the downloader (hit the 'table' icon button), whichever is breaking, and find the url that is failing in the list, is there extra info in the 'notes' column? Right-click->copy notes on that to get it in your clipboard, then paste it into a document editor. It should give more info, and hopefully a full html readout of the error page, which should give more info. 503 is often CloudFlare problems, but 403 is Forbidden, which usually means you are hitting a malformed url. Did you make the downloader yourself? Does it work in the parser testing window? Is there maybe a api url converter in the url class that is hitting the wrong thing? help->debug->report modes->network report mode can help here. You can also pastebin/email/whatever it to me, and I'll have a look. Thank you for the report about siblings and parents. I can see how something like that would happen, I'll check it out.

>>13768 I am afraid I do not know much about it. The one here: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/blob/master/Download%20System/All-in-Ones/Single-Sites/nijie-artist-2018.12.14.png Looks to be more than a year old, so if the site has just changed, I would not be surprised if it has broken. If so, I will remove it. For now, for any non-simple-booru, I am recommending people use Hydrus Companion to log in. It is a browser add-on that lets you log hydrus in by copying your browser cookies across. The page with instructions is here: https://gitgud.io/prkc/hydrus-companion However, if the layout has changed and login script is broken, the downloader itself may also not work. Let me know if this is true, I'll give the guys on discord a buzz and see if anyone wants to update it.

>>13784 >>13790 For now, I recommend you use Hydrus Companion, a browser add-on that lets you copy your browser's logged-in cookies straight to hydrus: https://gitgud.io/prkc/hydrus-companion This is a good blanket fix for sites that do not have native hydrus login script support.

>>13798 >If you want to chase spicy content before it is deleted, nono, I specifically would want the opposite of that >I will again say getting everything is almost certainly impossible for any human to sift through properly. that's why I'm training a resnet to do it for me :^)

Is there a way to have the duplicate filter only show the pixel by pixel duplicates?

>>13799 cheers lad welp. said fuck it and went for it. i updated backup then updated hydrus clear through to 388 in one go. nothing 'sploded; db booted; all seems well. happy days. new qt version looks and feels sharp af too. i liek. regarding 4plebs downloader, still no dice there. i was half hoping the update might just end up sorting it out but no such luck. anyhow. the list below isn't exhaustive but hits the main bulletpoints of what all i tried: -new custom downloader made from scratch → test parse are successful; download is fail -edit of first downloader stripped down to bare-bones → test parse are successful; download is fail -duplicate (structurally speaking) of first downloader dialed in for a different but similar website → test parse are successful; download is successful -import of cuddlebear downloader with smol modification → no test parse because am jsonlet; download is fail -reimport of cuddlebear downloader left alone this time → plug-n-play mode so no test parse; download is fail -simple downloader on default setting after nuking all traces of 4plebs from url classes and parsers → plug-n-play mode/no test parse; download is fail >"help->debug->report modes->network report mode" can help here. neat, that's handy af. picrelated. left side is using bog standard unaltered cuddlebear (json parser & 3 url classes from github repo). right side is the one i made (parser and 2 url classes, link here https://pastebin.com/F4kA9kxw ). the 4plebs urls don't "look" malformed and browser will navigate to them just fine but i can't seem to get anywhere with it when i try going through hydrus. all paths lead to 403.

>>13793 >Should be options->thumbnails, "EXPERIMENAL: Scroll thumbnails at this rate…". It'll tick down one thumbnail height at 1.0. Also there is a thumbnail % visible scroll check, which might actually be what you want here–set that to 0% and it'll always scroll. Yeah this is what its always set on for me. I tried changing the "Do not scroll on key navigation" percentage but I haven't really notice anything or at least relating to my issue and setting the "EXPERIMENAL: Scroll thumbnails at this rate" to 0% gave me errors, if that's the one you mentioned to set. Also I updated to version 389.

>>13801 Friendly neighborhood anon here, I whipped up a login script for e621 based off of the danbooru login. Not sure if it's 100% efficient(never made a login script before) - but it works, at least.
[73, "e621.net login", 2, ["e9438b0ebc12025f09faf1366734b927fef9832ce6d7d37d68f52da37512d0ea", [21, 1, [[], [], [], [[[51, 1, [0, "remember", null, null, "remember"]], [51, 1, [3, "", null, null, "Ph2fFwrvF7603984"]]]]]], [26, 1, [[72, "password", 1, [1, [51, 1, [3, "", null, null, "example string"]]]], [72, "username", 1, [0, [51, 1, [3, "", null, null, "example string"]]]]]], [26, 1, [[74, "hit home page to establish session, ignore passthrough", 2, ["https", "GET", null, "/", {}, {}, {}, [21, 1, [[], [], [], []]], [26, 1, []]]], [74, "hit login page to parse auth token", 2, ["https", "GET", null, "/session/new", {}, {}, {}, [21, 1, [[], [], [], []]], [26, 1, [[30, 4, ["get authenticity token", 14, [27, 6, [[26, 1, [[62, 2, [0, "input", {"name": "authenticity_token"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]]]], 0, "value", [51, 1, [3, "", null, null, "example string"]], [55, 1, [[], "parsed information"]]]], 0, false, "authenticity_token"]]]]]], [74, "post login form", 2, ["https", "POST", null, "/session", {"password": "password", "username": "name"}, {"commit": "Submit", "remember": "1", "utf8": "\u2713"}, {"authenticity_token": "authenticity_token"}, [21, 1, [[], [], [], [[[51, 1, [0, "remember", null, null, "remember"]], [51, 1, [3, "", null, null, "$5betre%0"]]]]]], [26, 1, []]]]]], [["e621.net", 2, "Login required to access special content (clear your blacklist)."]]]]


>>13810 Not yet. Pixel dupes are recognised in realtime at the moment, but I have a plan to make a cache for this data and let you search for them specifically. I do not know when I will get to this. I expect pixel dupes to be the first scenario you will be able to set up auto-duplicate resolution rules for. (e.g. 'if potential pair is pixel dupe and one is jpeg and one is png, auto-select the jpeg') With luck, we'll have auto-resolution of many imageboard clipboard.png files.

>>13812 Thank you–I think I figured this out. There is a little-used feature under network->data->manage network headers that lets you set the User-Agent (or any other http header) of hydrus for different network contexts. There should be a couple of examples in there you can basically copy. If I add a new User-Agent entry for archive.4plebs.org set to my web browser's string, the download works. Please forgive the bad UI–this is even more hacky code than usual. My guess is they have set up something that says 'if the User-Agent is anything other than these known browser strings, return 403 forbidden'. When you export a complete downloader package under network->downloader definitions->export downloaders, there is a button to include these custom http headers, which other users will be able to import too under network->downloaders->import downloaders. The files under the domain i.4pcdn.org do not seem to need the User-Agent.

>>13822 Thank you for this work! I will roll this into 390.

Is there a way to give two IMAGES (not tags) a parent-child relationship or similar? Similar to the "this image has x child images [link]" feature on borrus. Basically I want some way to link e.g. a two-to-four image set together

Does anyone have (or is the Dev planning) a gallery downloader for AGN.PH? Nintend0h has been getting uppity with takedowns, and I wanna rip pokemon shit before they get shitcanned… just in case.

>>13826 This is a slightly better version, now that I messed around with the login script settings more. I can't improve much else until the new version where the wx->qt bugs are fixed.
[73, "e621.net login", 2, ["e9438b0ebc12025f09faf1366734b927fef9832ce6d7d37d68f52da37512d0ea", [21, 1, [[], [], [], [[[51, 1, [0, "remember", null, null, "remember"]], [51, 1, [3, "", null, null, "Ph2fFwrvF7603984"]]]]]], [26, 1, [[72, "password", 1, [1, [51, 1, [3, "", null, null, "example string"]]]], [72, "username", 1, [0, [51, 1, [3, "", null, null, "example string"]]]]]], [26, 1, [[74, "hit home page to establish session, ignore passthrough", 2, ["https", "GET", null, "/", {}, {}, {}, [21, 1, [[], [], [], []]], [26, 1, []]]], [74, "hit login page to parse auth token", 2, ["https", "GET", null, "/session/new", {}, {}, {}, [21, 1, [[], [], [], []]], [26, 1, [[30, 4, ["get authenticity token", 14, [27, 6, [[26, 1, [[62, 2, [0, "input", {"name": "authenticity_token"}, null, null, false, [51, 1, [3, "", null, null, "example string"]]]]]], 0, "value", [51, 1, [3, "", null, null, "example string"]], [55, 1, [[], "parsed information"]]]], 0, false, "authenticity_token"]]]]]], [74, "post login form", 2, ["https", "POST", null, "/session", {"password": "password", "username": "name"}, {"commit": "Submit", "remember": "1"}, {"authenticity_token": "authenticity_token"}, [21, 1, [[], [], [], [[[51, 1, [0, "remember", null, null, "remember"]], [51, 1, [3, "", null, null, "$5betre%0"]]]]]], [26, 1, []]]]]], [["e621.net", 2, "Login required to access special content (clear your blacklist)."]]]]


>>13826 that did it! thanks so much again man

>>13830 Not yet. The next big push on the duplicates system will expand the 'alternates' file relationship into a full-featured file-association tracking system, as you say exactly like boorus do. Ideally you'll have multiple new relationships to set like 'recolour', 'costume change', and hopefully some basic indexing as well so you'll be able to order several WIP (work in progress) versions of a file. It'll also have ui support, so if you are looking at a file in the media viewer that has a parent/master, there will be a quick way to navigate to that without losing your place in the navigation carousel. >>13831 In general, I do not write downloaders any more. There are too many sites for me to keep up with and get my regular work done. If you have not seen it yet, the main repository for user-made downloaders is here: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System/All-in-Ones/Single-Sites There are several users on the discord working together to make downloaders as well, here: https://discord.gg/3H8UTpb

>>13833 Thank you, I have copied it. As I said in the bugs thread, I have fixed those two things (I assume you) highlighted, and I gave a quick click through all the other edit login script UI. I think it is all good now. I'll put off rolling this in for tomorrow's release, just in case you would like to do more. Please let me know once you are up to 390 if you are happy with this as-is or if you have a new one.

>>13836 I've updated to 390 and everything looks like it is working as it should - the second script is good to go as is.

I started getting an error when downloading from Danbooru: "Could not find a file or post URL to download!" This just started happening today, and I didn't change anything on my end other than updating to v390. Any ideas?

Is there a way to move a selection of files from one database to another while retaining the tags? I could do it one tag at a time, but it'd be great to have an easier way of doing it.

Hi, I'm looking at some programs to see what fits my needs and didnt see this in the docs. How does hydrus handle not downloading the same image twice? If I delete a crap image, will I have to worry about it being redownloaded again from the same place? (as an extra detail, I'm guessing you can't help it if it's a different site with the same image though) Lastly, as a test, I trying to import 25+ mp4 and webm videos that all fail with "stream missing" errors. Is this a known thing? Thx and sorry if it's a silly questin.

>>13852 Hydrus will recognize the hash of files that you already downloaded and skip them. You'll see "already in db". Files that you previously deleted will be tracked the same way. Files that are similar but not identical will show up in duplicates processing, and you can choose how to handle them yourself. The help section on the Hydrus website has some good info on the basics like this. I'm not sure about the .mp4 and .webm problem. Stream missing makes it sound like the codec isn't parsing the files properly, but that's just a guess.

>>13847 Fixed: I changed the danbooru file page parser to look for "class=image-container" instead of "id=image-container".

First time user here. Already add my login credential for both gelbooru and sankaku sites. Why the gallery download doesn't works with loli tag for gelbooru? it said "The parser found nothing in the document!" It works normally with other tags for gelbooru. Gallery download works normally with sankaku for loli tag, but for some reason the tag imported doesn't add loli tag into the downloaded images, everything else got added, I even got loli_face tag but not loli tag. Also can I changed the position of the windows? I want to switch the place of preview and thumbnail windows.

Also is there a way to make Hydrus preview the thumbnails first before downloading the original image just like Grabber and then I can pick which image to download later one by one using right click or shortcut? Because downloading each image and then picking which image to saved and delete take too much time and bandwidths. Grabber can do this but this app doesn't support local database and tags.

Is there a way to autotag the images I import without using the bloated public tag repository? I'd only like for the artist to be added, everything else is unnecessary for my purposes.

>>13854 Oh, the hash makes more sense. Yeah I have no clue about the stream missing, I have the newest ffmpeg codecs I could get from the ubuntu 19.04. Just let me know if there's something I can try.

>>13861 Responding to myself: looks like the hydrus client didn't detect my ffmpeg codec in the "about" page. I tried a separate client download and it recognized and work.

After a recent client update a tiny portion of my files have begun showing up glitchy in the client (low to massive distortion). So far I've found 5 out of 200, all of them are webm or mp4. They ONLY glitch when viewed through the hydrus client, opening them in any other program has them showing up fine. Is there a way to fix this? If not, is it possible to edit a file's metadata (mainly import date) so I can import them again and put them back where they belong?

Regarding the URL classes that redirect from a web page to an API call, which checkbox settings take effect? The page class or the API class ones?

>>13857 Gelbooru needs the cookie fringeBenefits=yup to view that content, make sure you have it in hydrus. >>13858 No >>13859 Parser's can get tags from the site they are downloading from. Most parser's should already get some sort of tags, you can choose which one's it gets in tag import options.

>>13868 The cookie.txt fixed the problem. Since there are no way to changed how the GUI looks, is there a way to make Hydrus act like a normal booru so I navigate them from a web browser? Hyrdus is a great tool for scrapping image from boorus, but the UI navigation is really bad.

If a hydrus database is corrupted, but in a subtle way that isn't noticed immediately by the client on boot or during automatic maintenance procedures, if that can even happen, then is it possible for the backups that you make to become corrupted too, or is there some kinda check in place to make sure that the backup is safe before writing it to the backup location?

>>13869 There are some third-party tools that make a web UI for Hydrus. This one is probably the best at the moment. https://github.com/mserajnik/hyve

Is there any way to change the font size of the GUI? I can't find it within the options. I'm on a 4k monitor and I am going to go blind

>>13875 The last time I tried this it said, 404 when I tried to connect the API. And there are no tutorial for non tech savvy people. What is this even mean? HYVE_API_BASE=/api: the base path of all the API routes. No trailing slashes. HYVE_MEDIA_BASE=/media: the base path of all the media routes. No trailing slashes. VUE_APP_IPFS_GATEWAY_BASE_URL=https://ipfs.io/ipfs: the URL to the IPFS gateway you want to use for IPFS links (only available when connected to hydrus client). No trailing slashes. VUE_APP_HYVE_API_URL=http://localhost:8000/api: the URL to your hyve HTTP API. No trailing slashes.

Is there anyway to change where the temp dir is? I'm trying to free up some space on my ssd and there are ~2600 Hydrus files and 14 empty folders in the AppData\Local\Temp folder. Would like to move the temp dir to another drive or reduce the size if possible

How do I run Hydrus over TOR? Assuming I don't want my ISP snooping on my DNS queries to figure out what weird sites I scrape from.

>>13876 You could make a qss and change the font size in there, there is a readme in hydrus root dir/static/qss. Something like this.
QWidget {
font-size: 100px #whatever font size
}
>>13879 You can change it with a launch flag. client –temp_dir="temp/dir/path" >>13880 You should be able to add the TOR proxy into hydrus proxy settings in options>connection

>>13877 >HYVE_API_BASE=/api: the base path of all the API routes. No trailing slashes. Thats the path for the API. Say you wanted the API at localhost:8000/ass, you would set it to "/ass" >HYVE_MEDIA_BASE=/media: the base path of all the media routes. No trailing slashes. Same thing but for the media is servers. e.g. localhost:8000/media/ass.png >VUE_APP_IPFS_GATEWAY_BASE_URL=https://ipfs.io/ipfs: the URL to the IPFS gateway you want to use for IPFS links (only available when connected to hydrus client). No trailing slashes. This shouldn't be important if you don't use IPFS. >VUE_APP_HYVE_API_URL=http://localhost:8000/api: the URL to your hyve HTTP API. No trailing slashes. The URL to the API you set in HYVE_API_BASE for the web front end to access. I agree that it's not the easiest for non tech savvy people but then again neither is Hydrus.

>>13881 The client won't launch if I use this as a launch flag

>>13881 >>13884 I got it to work with two dashes instead. Am I free to delete the old temp files now?

Is there a way to control, how many images are downloaded per second in a gallery download? Seems like now it does 1 image per second, and even though I'm in the advanced mode, I can't find a way to change that.

>>13883 >but then again neither is Hydrus. I got no problem in installing hydrus and it got a wiki site explaining on how to use it. Hyve only got a readme full of tech jargon.

>>13868 Loli tag still not imported from sankaku even with cookies.txt added.

>>13890 you could try this one then, I thought it was pretty easy to setup, though no documentation. All you have to do though is run npm install then ng serve in its folder after installing npm and angular-cli. https://github.com/floogulinc/hydrus-web

I tried Suika https://github.com/Suika/hydrus-docker Because it got build in Hydrus client in it, no problem during installation with docker compose. I can download images from boorus and browser them with Suika's built-in Hydrus using VNC remote, but when I open the Suika's Webui I got 404 in the API page and no image in the gallery page. My Suika compose setup https://pastebin.com/wthuSjYE Since I already got Suika's Hydrus installed and some database and images in my NAS I tried Hyve next. I added Suika's db folder and client_files folder into Hyve's docker-compose.yml.client.example then ran it with my Portainer stack. Open webui and got this >ApiUnavailableError >Could not connect to hyve API. Please check your .env configuration. My Hyve client compose https://pastebin.com/3Vfj4TuC I changed hyve-sync ports: from '8000:8000' to '7000:8000' because my Portainer webui also used the same ports and I can't changed them. Next I tried to import Hydrus images and tags into Szurubooru, with Szuruporter, dunno if i'm just a brainlet or was it because Szuruporter never got update anymore because nothing works.

>>13849 https://hydrusnetwork.github.io/hydrus/help/advanced.html Hydrus Tag Archives under the heading for tag migration sounds like it'll accomplish what you're after. >>13859 check the link above, read the section called importing and adding tags at the same time. if that's N/A to however you've got your files named or whatever your folder structure is, then probably the public tag repo would be your best bet. can't say for sure as i don't use it myself but there might be a way to whitelist the artist tags and thereby reduce the bloat.

When I set a custom path for an external program, Hydrus does open it, but it doesn't open the file in it. Do you know why this is?

>>12641 Why do most of my downloads from danbooru get ignored?

Is there a way to count the total number of tags, or similar statistics other than the "how boned am I" window?

Is there anyway using a parser to add to an images notes. For example, some images have a story in the description and so I currently have to manually copy it into the images notes. Thanks

There is a typo in the character type dialog, "minUmum".

>>13902 Are you on the latest version? Danbooru changed some stuff so the parser had to get updated. >>13905 Unfortunately not yet.

>>13905 You can use tags for that. You'll lose line breaks, but at least it'll be searchable.

Just wondering, is there a guide in video form for info about Hydrus Network? The main guide is helping me a bit but i thought a video of footage of the software being used could help too

>>13877 Till today, i still don't know how to make this run. Still got "Could not connect to hyve API. Please check your .env configuration." I don't even know where do I get my API url from. Here my compose setup https://pastebin.com/jdZK0ite What I want to do is setup a Hyve on my NAS server using docker, let all my booru images and hydrus database saved inside my NAS. While using Hydrus windows client to download my images and send them to Hyve server folder on my NAS.

Hiya. Recently got into hydrus and am loving how easy it is to download and sort/process the thousands of images/manga etc that I usually download manually, so thanks for that! Just looking for a solution to the following issue: Version - 392 Situation - Downloading an artist's work via "pixiv artist lookup" gallery downloader Problem - Some posts are being ignored with comment "Ugoira veto". Is there some way to disable this veto and have it pull the ugoira files too? I can't find anything in the changelog about ugoira since version 329, and can't find any settings that would relate to it. Ideally I don't want to have to download and tag them manually.

>>13919 Wrong quote. >>13883 Fucking captcha.

>>13921 >Is there some way to disable this veto and have it pull the ugoira files too? Not with just disabling the veto, It would require an additional parser but even then ugoria isn't a video file, you could only get it as a zip of images.

Hello, is there a way to automatically hide the cursor after opening the media viewer instead of having to move the cursor around?

Is there a shortcut for undoing a deletion and adding it back to the current view? Neither Ctrl+Z or setting a hotkey for "undo" undeletes the file, I have to manually right click to undelete. If not, am I using the wrong workflow? Thanks

The cycle of me being unable to check and respond to my messages every day continues, worsened by current events. I apologise. I'll keep trying new ideas here on schedule and stress budget, the ideal is 24-hour turnaround. >>13865 Are you viewing these files through the hydrus native viewer, or the new mpv window, do you know? The new mpv player has audio support. You can also see what you have set to display files under options->media. Can you post one of the files, or otherwise link me to it, so I can test my end? There is no way to easily edit import time right now.

>>13867 Good point. I think those mostly apply just to the URL in the import queue, the actual URL being actioned, which will typically be the Post Page URL, not the API. Usually, the API conversion happens very late in the download process and is temporary, it only changes what URL is actually hit and what parser is used. I think only the raw URL stuff, the top three checkboxes in your image, would apply to an API URL, since that stuff applies to the URL itself and recognition and parser match-up.

>>13873 Unfortunately, a full integrity check is fairly slow, usually 1-10MB/second, so this is not done automatically. You can force a check under database->check->db integrity, and if you would like to set up a routine to regularly do so, it would increase your safety. Otherwise, your scenario can unfortunately occur. The good news is that if the subtle way isn't something you or the client immediately notices, it is usually something like 'a power cut that turned off your computer while you were committing PTR data malformed a 4KB database page of 'stained brown shirt' tags on a selection of obscure.booru.org files', and since you likely don't have those files, you probably never load them and never run into the problem, and in any case PTR data, which is about 98% of all your db data, can be recovered with reprocessing, so it isn't so big a deal. Check out 'help my db is broke.txt' in install_dir/db if you would like more background reading on this.

>>13876 I am working on nicer solutions here that you can select in the client. It'll involve changing the qss as >>13881 suggests, but with 'pt' instead of 'px', and I need to fix some custom text I draw in the program that doesn't yet follow qss rules. >>13885 Yes, you can delete anything in a temp folder if there isn't a hydrus client currently running and looking at it. Nothing in there is persistent across boots.

>>13888 All of that is controlled under network->data->review bandwidth usage. It is advanced, so be careful. I strongly strongly strongly recommend against just deleting all bandwidth rules, as you'll likely catch some IP bans and also possibly overwhelm your client DB and UI. Most of the rules that are in there by default–the 'edit default bandwidth rules' button–are set to ensure new users don't make a mistake and download 5GB of trash by accident. I recommend you keep the 1 file per second global domain rule, but it is up to you. If you are interested, I have longer thoughts on the philosophy of not overwhelming your inbox and sanity here: https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html

>>13849 Yeah, I need to write a proper 'how to migrate files with tags' guide, as this keeps coming up, but the main solution here is as >>13895 says: use services->tag migration with Hydrus Tag Archives. Check it out, be careful because it is advanced, and let me know if you run into trouble.

>>13901 It probably all depends on how the program wants to accept that launch path. They aren't all in the format: program_exe "path" But sometimes: program_exe –open="path" Or it might sperg out about paths with spaces in–don't forget the " marks. If you open up a cmd/terminal and do a test, does the same format you have hydrus set to do work, or does it similarly just open the program? If it works in the command line, then try turning on help->debug->report modes->subprocess mode and then launching the program. This will popup a bunch of debug messages about what is about to do, and the error may be apparent. If it isn't, let me know what you'll get and I'll have a look.

>>13904 Not yet. I'd like to expand this stuff in future and have some graphs and things. >>13905 I hope to have an expanded notes system with note parsing in downloaders in the next couple of months. It will happen in a 'medium sized job' week.

>>13906 Thanks, fixed. >>13910 Not that I know of. I agree though, although I am sorry to say I don't have the time or expertise to do it myself. >>13921 Unfortunately, there is no Ugoira support in hydrus yet, so the veto is the current 'graceful' way to error without an error. I have plans to add it, but it is a tricky format, so I can't confidently say when.

>>13926 Thanks, I didn't realise that hide tech didn't kick in until an initial cursor move. I assume you are navigating by keyboard? I'll check that out and see if I can fix it. >>13927 Shift-delete I think is the default. Please forgive this shortcut being a bit hidden, delete keys are funny, so these actions are still hardcoded and not yet moved to the full customisable shortcut system.

>>13947 Thanks regarding shift-delete. This still requires switching pages, tracking down the deleted item which is troublesome especially with no "order by time deleted" filter, clicking the image, and then shift-delete on it. My ideal inbox workflow would be (text after * isn't possible currently): A (delete bad img and auto move to next image) L (archive image and *auto move to next image) * Y (undelete image and auto move media browser back to undeleted image rather than tabbing to trash where the images are out of order by deletion time)

>>13949 Thank you. There is a two part solution here, I think: 1) (plan to work on better trash timestamp stuff deleted) EDIT: I was talking to another user about sorting by trash time not working, but I just tested it now and 'sort by time imported' works ok for trash, as long as you are looking at the 'trash' file domain, rather than 'my files' . I can see how this doesn't work when you are looking at a 'my files' page that happens to have some trash in it, though, so I'll think about this. 2) I have a barebones content undo/redo system, but it only works on a couple of things like archive/inbox, and that not well. The main thing I have to do there is keep pushing on shortcuts, which involves converting program actions from hardcoded calls to a generalised 'command' action system that all works through the same pipeline. As I move more of the program to that, I'll be able to hang those undo-able commands on an undo system plugged into that same single pipeline. Just have to keep working.

Hello all, I stopped updating hydrus after the 8chan debacle and never really went to search on how to continue getting news about it. I noticed the current version is 398, mines is 318. Could you let me know a list of major things that were added or changed since then? Also given this situation, I was thinking of deleting the whole database, settings, etc (except the pictures) and start over. If I copy the client_files folder under Hydrus/db/ would that be enough to start over and keep the pictures or I need to do something else? I used the Windows extract only version. Thanks for all the help. PS. Not required but is there a copy of tags of videogames/cartoons/anime I can borrow from anyone? Current mine it has a lot of junk tags. Just looking for characters and series names.

>>13956 Hello and thank you for everything that you have done for us, It is my first time running Hydrus on this computer and it's not starting up, it's just taking up CPU power in the background. I don't recall it taking this long to load. I have all the prerequisites checked. Is this normal? And if so,hould how long sit take depending on whatever pc specs you imagine? I apologize for the stupid question, it's just a little frustrating.

>>13960 I think you meant to write to Board Owner/Dev. I'm not him, i'm a normal user only.

I'm trying to make a parser for nyanpals.com/pettankontent but the site needs javascript to show the posts. Is there any way to load the page with JS or am I fucked in that regard?

>>13956 >Could you let me know a list of major things that were added or changed since then? If you want to waste 20 minutes reading, you could just go to the github and read the first few lines of each release. >If I copy the client_files folder under Hydrus/db/ would that be enough to start over and keep the pictures or I need to do something else? No, hydrus would consider them all junk files without the database file. If you don't want to update you can either export all files with tags(if you want to keep the tags) or just open the new hydrus and drag all the files in to import(you can just drag the db folder for that). >PS. Not required but is there a copy of tags of videogames/cartoons/anime I can borrow from anyone? If you don't mind bloat you could just download the PTR and either censor all tags except the ones you want or move all the tags you want to a new tag repository then deactivate the PTR.

>>13962 > Is there any way to load the page with JS No, use your browsers development tools network tab to find out where they are loading the content from with JS and parse that instead.

I'm having the same issues as >>13960 It's not starting up. I'm running the latest version, fresh install.

>>13969 I managed to track it back to a JSON page and convert it to {url:"/img/pettankontent/50a00b3bb9934e314dcbd1e6c614f2ab.png",tags:"miqo'te,keiko"}
{url:"/img/pettankontent/535c19b6fe565f1b90262cbf978285ad.png",tags:"miqo'te,keiko"} but now I'm fucked. I don't think hydrus has any way of converting this. It's not HTML so that doesn't work, but it's not JSON either so that breaks.

So I have an issue, I use a lot of personal tags and have them set up with parent tags exclusively using My Tags but they're automatically being applied to the ptr tags as well. Its pretty irritating because I specifically did it to more easily tag things for myself because the ptr has a lot of mistagged images but theres not much point of the tags are automatically being applied to the ptr as well because of the parent tags I set up in My Tags. For example, I set up a tag called #:3d so that it automatically sorts to the top of the tag list and so I can easily organize 3d images/videos. I also set up parent tags for it (in my tags) so that anything tagged with 3d rendering and the like will get it, now like a week later #:3d is getting applied to everything in the ptr too, among others. Im not sure if thats just on my end but its pretty damn annoying either way seeing as I have to commit ptr tag changes every time I want to get rid of them. Anyway I can stop this without getting rid of the parent tag entirely? Also, is there an option to make tags from the tag suggestion box in the manage tags window not disappear after you apply them? Before you could scroll through images and tag things without closing the manage tags window, but now the tags disappear from tag suggestions after you apply them until you re-open it. Makes tagging things a bit more inconvenient to do. I feel like a dumbass typing tag 5 times per sentence by the way

I think i messed up my installation/db by just copy pasting the install folder to my HDD with more space. I believe I have it all recovered and have access to the db again, but I can't drag and drop images into hydrus anymore, neither from Chrome or desktop/explorer. I think UAC now complains every time i run the exe. I don't remember if last time it just worked or had to change something to enable DnD.

>>13952 I recommend you update rather than starting over. There is lots of data in your existing db that won't carry over, and you can't really just switch out your client_files and continue. I recommend you update 10 versions at a time, as described here: https://hydrusnetwork.github.io/hydrus/help/getting_started_installing.html#big_updates Let me know if you run into any trouble.

>>13960 >>13971 I am very sorry to hear this. I updated from python 3.6 to 3.7 in 393, so that is my biggest guess here. Can you try a couple of things for me? If you go to install_dir/db/client - 2020-4.log and scroll to the bottom, how far does hydrus think it got in booting? What was the last step? Are there any errors there? If you download the extract release, if you don't already have it: https://github.com/hydrusnetwork/hydrus/releases/download/v393/Hydrus.Network.393.-.Windows.-.Extract.only.zip And extract it to your desktop, does that completely bare and new client run? It looks like >>13960 is Win 10. What OSes are you on, and is there anything special about your OS? Is it behind on many updates, or have an unusual .NET or Visual Studio Redist situation, or is it Windows 10 'N'? Could you have an anti-virus that quarantined one of the hydrus dll files? I recommend you roll back to 392 for now.

>>13962 Yeah, this >>13969 is the trick here. See what json they are pulling with their js–it usually is really nicely formatted, and actually easier than parsing html. On Firefox, right-click and hit Inspect Element, then hit the network tab and hit f5 or scroll down to load the next gallery page or whatever. You'll see it come in as it happens. There are a few 'api' parsers in a default client install that work like this. Check them out, and their GUGs and URL Classes (which use API transformations) to see how it all works. Let me know if you run into trouble.

>>13972 Ah, can you post an example URL, so I can have a look? EDIT: I looked it up: When I fetch: https://nyanpals.com/kontent?include=pokemon&exclude=nsfw&allow=&limit=10&method=uploaded&offset=0&order=DESC&token=null I get some nice JSON like what you have here, but it seems to parse ok. The only problem is that comma-separated tag string. There isn't a way atm to get hydrus to parse that single item into multiple strings, afaik, although I do hope to add this. An easier first solution is to just get the URLs. If you are unsure how to split that into rows in a hydrus parser, you want to change to a JSON formula and get the 'rows' value, then every item of the subsequent list, and then every 'url' entry of those. It should give you a list of URLs.

>>13975 I am sorry for this trouble. There aren't excellent tools yet to change what parents and siblings apply where. I want to change this in the near future, allowing you to set per-service rules and sibling/parent application. For now, the only option is in options->tags, uncheck 'apply all parents for all services'. This should keep your parents separate, but I do not think it is perfect. Let me know where it isn't perfect, and I will fix it. I accidentally made the new tag-removing suggestion boxes not refresh correctly on new media. I will fix this. Once it is fixed and you give it a go, please let me know if you would still like the option to have the tags no longer remove. The ideal here is that all the lists will constantly synchronise to the current media so they never show a tag that is already 'current'. Tell me about it. Sometimes doing this I have to say the weirdest shit over and over.

>>13976 Windows doesn't like it when you DnD between programs of different privilege levels. The most common example is one program running as admin and another not. I understand there are more subtle levels that can apply as well, but I do not know the details. Could that be the problem? It sounds like UAC is asking about admin rights there–is hydrus now trying to write its db to Program Files or a similar protected location?

>>13983 >An easier first solution is to just get the URLs. That works, but it means I can't get the tags since the URLs link directly to the images, which is the entire reason why I wanted to write a parser.

>>13985 That must have been it. I did have it installed in program files x86 just now. Reinstalled somewhere else and now DnD works. I guess i forgot where I had it installed last time. Thanks for the assist.

>>13988 Right. Unfortunately I don't have a 'split one text into multiple' lego block ready to slot into the pipeline yet. The current string transformer is strictly 1-to-1. I've just had a look at the code to check. I currently have: for every text parsed by the formula: test against string match convert using string transform I could try to slot in a new 1-to-n 'string splitter' object before the transform, but perhaps the real answer here is to also make a 1-to-n 'string processing' container object that can hold m match, converter, and splitter objects in any order you want. Then if you want to test a match after conversion you can, or after splitting, whatever you want. I could spam that processing object in several other places, along with the splitter. I will have to think about this. This specific problem of string splitting keeps coming up. I think it might be worth the extra investment to make a flexible, pipeline-simple system that will solve more problems than try to jam in more hardcoded stuff.

Are known urls for files supposed to be urls that point to the same exact file (hashes match) or just files that look the same? Like, is it something I would add myself when a come across a file on the web that looks identical to one in my database, or do I just leave adding known urls to the downloaders and such, who can check that the url points to that exact file, and not just one that looks like it? I wondered this when I saw the known url merging options on the duplicates processing page. I thought that it didn't make sense to ever merge known urls because they're different files, but then that got me wondering if known urls just need to point to files that are "close enough" to be okay. Of course, if that's the case, then that might cause trouble with downloaders skipping downloads of files from urls that are already known urls, especially if the file that would've been downloaded was a higher quality version.

Is there a GUI setting in 393 i need to tick to fix this error? Having the error on tbib and danbooru so far. Also I migrated my image db and upgraded like 50 versions, and I notice one of my older parsers transferred to. Maybe my parsers are outdated? (srry if that doesn't make sense, not 100% on this stuff yet). Here's the error anywho: Looks like an unsolvable CloudFlare issue: Cloudflare reCaptcha detected, unfortunately you haven't loaded an anti reCaptcha provider correctly via the 'recaptcha' parameter.… (Copy note to see full error) Traceback (most recent call last): cloudscraper.exceptions.CloudflareReCaptchaProvider: Cloudflare reCaptcha detected, unfortunately you haven't loaded an anti reCaptcha provider correctly via the 'recaptcha' parameter. During handling of the above exception, another exception occurred: Traceback (most recent call last): hydrus.HydrusExceptions.CloudFlareException: Looks like an unsolvable CloudFlare issue: Cloudflare reCaptcha detected, unfortunately you haven't loaded an anti reCaptcha provider correctly via the 'recaptcha' parameter.

>>13993 I should mention my whole comp routes over tor and that hydrus is running via torify

Hello guys ! Newbie here, I'm reading the help but I'd like confirmation if I have understood everything correctly: everytime you move an item to the db (drop&load or manual import) hydrus is in fact creating a copy of your item and a thumbnail. Those newly creating files are stored somewhere in the program both in form of hash, thus allowing it to quickly access the data and the thumbnail but not really to access it easily by the user using your regular framework (like going into windows file explorer) ? If this is correct, can someone tells me how much datas he generated for one item ? I would say it roughly doubles it but I'm not sure. I'm asking because I have a fairly large amount of files that I would like to process but not being able to allowed that much disk space I'd like to estimate the total amount it would required. I'm of course thinking of investing in some new storages and I'd like to know if someone ever tried to store their datas onto usb drives like it is mentionned in the help. I always have my doubts about that method because of how "temporary" is the nature of those but this seems like a good way to stock new items. Thanks everyones and special thanks to the dev, this stuff is a dream comes true.

Is it possible to assign extra mouse buttons as shortcut triggers somehow? The GUI doesn't seem to recognize them.

>>14000 The files are just copied so yeah double, plus a little bit extra for info about them stored in the database but this is minuscule.