/hydrus/ - Hydrus Network

Archive for bug reports, feature requests, and other discussion for the hydrus network.

Index Catalog Archive Bottom Refresh
Name
Options
Subject
Message

Max message length: 12000

files

Max file size: 32.00 MB

Total max file size: 50.00 MB

Max files: 5

Supported file types: GIF, JPG, PNG, WebM, OGG, and more

E-mail
Password

(used to delete files and posts)

Misc

Remember to follow the Rules

The backup domains are located at 8chan.se and 8chan.cc. TOR access can be found here, or you can access the TOR portal from the clearnet at Redchannit 3.0.

Uncommon Time Winter Stream

Interboard /christmas/ Event has Begun!
Come celebrate Christmas with us here


8chan.moe is a hobby project with no affiliation whatsoever to the administration of any other "8chan" site, past or present.

(94.10 KB 1024x768 1601788051705.jpg)

/big data general/ Anonymous 10/05/2020 (Mon) 05:17:17 Id: ecf9e3 No. 14802
or, "i've got xxxxxxx thousand images in my inbox, what the fuck do i do?" i'm currently having this problem myself, so i thought it would be nice to have a thread to collect information about various ways to automate sorting/tagging/etc, such as: * image deduplication scripts: >https://github.com/knjcode/imgdupes i'm presently also writing my own as well which should work somewhat better for large datasets, i'll post it here when it's usable * datasets, useful for running against aforementioned dedupe scripts >danbooru2019, contains all danbooru pictures + metadata up to early 2019: >https://www.gwern.net/Danbooru2019 i remember there being large dumps of other booru metadata on here years ago, but i can't find them anymore * AI/neural network software for automated tagging, classifiers and etc: https://github.com/KichangKim/DeepDanbooru https://github.com/imamar94/ramrem-classifier things i couldn't find but would find extremely useful: >AI anime/photograph classifier this would be very helpful, especially with deduplicating from danbooru2019 >subject classifier if there was a NN that could tell me if the subject of a picture was a person or something else, this would be insanely useful, especially for the next item >SFW/NSFW classifier about half of my collection is porn. i'm planning on deleting basically all of it, so just being able to get that out of the way quickly would cut my workload in half
Looks kind of like a ML-general so I'll treat it as such. Got a bunch of stuff an Anon asked me to dump somewhere. No general order, some of it will be more useful than others. I'll try put the actually useful stuff first. It sounded like the OP requested an image comparison tool based on heat-mapping. I forward that request, having forgotten the name of the tool(s). EDIT: Forgot some stuff, formatting. >http://www.visipics.info/index.php?title=Main_Page Still probably the best and easiest GUI deduper. Works fine even with Hydrus folders, Hydrus might not like externally deleting pics. >https://gitgud.io/koto/hydrus-dd/ see https://github.com/KichangKim/DeepDanbooru/ If you want automatic tagging, get it. Latest model is greatly improved. DD site seems even more accurate. Might have to do with model conversion. Run hydrus-dd evaluate-api-search --service myTagService in CLI to automatically tag everything in that tag service (default is my tags). >https://github.com/imbaguanxin/Recognize-Your-Waifu & https://github.com/swrd1337/waifu-recognition & https://github.com/bakwc/PornDetector & https://github.com/hhatto/nude.py & https://github.com/notAI-tech/NudeNet Classifiers. I personally don't care about these. Nudenet is the easiest to include in a project. >https://thisxdoesnotexist.com & https://github.com/justinpinkney/awesome-pretrained-stylegan Masterlist for NVIDIA stylegan projects you can test online. >https://www.gwern.net/About Gwern's site is cool if you're into this shit. >https://github.com/keptan/superCuteGrab Looks cool, but you need a saucenao account and it's generally a pain to build. >https://github.com/soruly/awesome-acg Anime&manga big data. >https://github.com/josephmisiti/awesome-machine-learning Bigass list of doing ML in a lot of programming languages. >https://github.com/BlueAmulet/ESRGAN & https://github.com/ptrsuder/IEU.Winforms/releases & https://upscale.wiki/wiki/Model_Database Enhanced Super Resolution GAN, GUI for effectively using it and scaling models for everything. What waifu2x wishes it was. >https://gitgud.io/prkc/hydrus-companion Obligatory shill for prkc if you're still using hydrus without the companion you're wrong.
>>14802 >i'm currently having this problem myself, so i thought it would be nice to have a thread to collect information about various ways to automate sorting/tagging/etc It would be cool for hydrus to automatically tag anything imported, or downloaded with tags like hasaudio if it has audio. Or even just all files, seeing I see no reason for it to fill this info in for missing files. It can't really get it wrong. Same with medium:video, etc. These are stuff it can easily recognize from the file alone. Should help the inbox sorting a little.
>>15077 It already does tag stuff based on the file, see the system tags. There is 'system:has audio' and 'system:no audio', and then there's system:filetype which lets you choose either the type (image, video, etc) or exact MIME (i.e. mp4) of files to seach. There are plenty of others too, like dimensions, time imported, hashes, filesize, etc.
>>15082 >It already does tag stuff based on the file, see the system tags. I was suggesting it automatically applies these tags when it detects them. For example if you import a file that has audio, it knows that so it just applies the tag. Unless I'm missing something, it does not seem to apply automatically.
>>15084 Is that not what it does? Unless you mean applying a non-system tag like "video with audio" or something, it does do that automatically.
>>15099 Just tried adding a new url with a bunch of videos to the watch, some have audio some don't. None of them have been autotagged with any tags, except for the few that have a hash matched. So it does not seem as if it is auto-applying tags for stuff that are videos, and stuff that has audio.
>>15101 do a search for "system:has audio" you absolute retard


Forms
Delete
Report
Quick Reply