>>12428 Strange, now it's working fine. I must've screwed something up during creation. When I last tried it, it insta-404'd

KILL URSELF NIGGER BABY

HYRDUR SSUCKS BAN ME

Imagine being a folderlet

Subfolders are ok, I use them too since hydrus is (currently) shit and crashes for no reason, but tags are objectively superior.

>>12316 sadly the only program that you would be up against is comic rack and acdsee in terms of full featured program and in terms of comic rack, nearly every feature they have, you also have, and in terms of acdsee you either use crash prone versions from 10+ years ago (version 8/first pro to version 9) or you use the recent versions which are pure bloat for unicode support, there is very little middle ground with acdsee due to so many of the in between versions fucking with features. >>12319 I honestly never want an automatic sort, as much as it would be good for going though my cluster fuck, I am still getting images that are full featured images going against 23 byte black boxes. running into this makes a pure auto system unacceptable to me, I would accept an auto system that looks at two images, and then has me spot check lets say it finds a better worse pair, it presents me the better and it presents me the wrose green border is better, red border is worse I scroll though like normal, but left click confirms green, right click confirms red. Confirmed green acts like better worse does currently, confirmed red takes it out of the auto figure it out area and into a manual pick.

Are there any plans at all to (at least have the option to) keep metadata of individual tag mappings, like a file's tag history or keeping track of which process added this tag to this file? Like was it typed manually, was it imported while scraping, etc. I assume this would bloat the DB by a big factor but at least for my personal use I think that might be worth it down the line

Poll >>12358 !

>>12339 tried it and what happens is it doesn't fetched the edited parsers at all until I change it back to normal

>>12340 here I changed the namespace from "creator" to artist. When I enter the URL, Hydrus no longer fetches the artist at all.

>>12341 Hmm, I'm not sure why that's happening since it should be the same part of the html getting parsed.

Just did another clean up as hydrus was eating 6gb on launch and dups were starting to lag the program to much, its now starting sub 1gb with next to nothing changed since the last time I did this which got it down to 2.3~gb still honestly see no reason for the watchers once they 404 to eat that much space, but good to see the program is getting even better with ram management. >>12205 as far as the images themselves go, yea, they are likely changing 1 pixel and re saving it, my shotgun approach to downloading has managed to seemingly sweep up everyone ever made but there are a few times where I knew the image was never getting posted again I just straight deleted the shitposts, but this time… I thought I got rid of all of them, but apparently it's somehow finding more. >>12208 after a while I thought this too as I was getting somewhere of 455k images, it was really odd that there would be 3 or 4 images that were dups of the same image that would come in.

>>12204 I have to ask, is it possible to have a custom sup finding thing? sadly I went though these a number of weeks ago, but I was checking an import archive, and apparently what I used to download corrupted images, it's more or less the standard at some point the image below this line is no longer loaded and its solid grey lines. with images like this, would it be possible to have a 'corrupt dup' setting where you either point out where the image is good, or where the image is bad (a fill tool may deal with bad better then good making it easier to do) and have it discount that section from a dup search? my logic here is if I have a large enough archive, or the interests see me downloading similar images regularly, there is a good chance I have the un corrupt version somewhere else already and just to head this off, these are images that have gone from a 120gb maxtor, to a 1.5tb seagate and survived 3 hdd failures there, to be moved to a 4tb and finally to am 8tb hdd, and finally to a currently 3tb image archive drive. odds are one of the drives fucked with the files in some way, or it was my horrendous method of acquisition all those years ago so the issue isnt my hdd corrupting images, its more I have corrupt images that may have to much corruption to be dup found

>>12239 No, I am afraid sectioning off good and bad areas of images is way too complicated for my current system to deal with. I encourage you to delete all corrupt files.

Here are a list of "expert system" Video Quality Quantifier https://github.com/Netflix/vmaf (C/C++/Python) https://github.com/aizvorski/video-quality (Python) https://github.com/bavc/qctools (C++) https://github.com/Rolinh/VQMT (C++) https://github.com/google/rtc-video-quality (Python) https://github.com/kahkeng/vqats (C/C++) https://github.com/slhck/ffmpeg-quality-metrics (Python) https://github.com/honzabilek4/VideoCodecs (C++) https://github.com/jsyzgaochao/iqat (C++) Here are a list of "expert system" Image Quality Quantifier https://github.com/andrewekhalel/sewar (Python) https://github.com/jeffh/CV-Image-Quality-Analysis (Python) https://github.com/VIQET/VIQET-Desktop (C++/C#) https://github.com/arcaduf/image_quality_assessment (Python) https://github.com/bukalapak/pybrisque (Python)

>>12295 Why don't you actually develop something on your own instead of endlessly shitting out github links

>>12302 Nah that is for >>12277

>>12136 I would like to have that, particularly for my own use, but unfortunately this functionality does not seem to be built in to my ui library, and furthermore my thumbnail panel is all my own custom code. I will have to write both ends of a system like this, so I have it as a 'big job' for now. It is currently "Write some ui to allow selecting thumbnails with a dragged bounding box" here: >>12152

>>12145 Thank you, the 'delete after export run' bug should be fixed in 347 for export folders that make filenames that include subdirectories. Please let me know if you still get it in the new version.

>>12146 Yeah, now here as a new 'big job' to work on: >>12152 . However, I am not totally sure what to do with the current booru. It was always a prototype that then never got revisited. I think next step is probably to convert it to the nicer new Client API code just so it is all operating on the same system, and then work on templating so users have more CSS/HTML customisation options. Then again, I do know of some users working on booru wrappers for the Client API as-is, so if they can do a better job than I can fit in, it may be better ultimately for me to abandon it. If the booru does not prove all that popular in the poll and some other users can fill in the gap, perhaps we can revisit the whole idea and instead I give more power to them via the Client API. If you have particular ideas on integrating the booru into the actual client ui, I am interested to know what you are thinking of. Please post in that big poll thread if you would like.

What 'bout wildcards in parent tags?

>>12148 That is an interesting idea. It might be dangerous if I were not careful, but I like it. Unfortunately, the tag siblings and parents system needs significant overhaul at the database level before I can do anything clever and new with it. If you are interested in new sibling and parent tools, please vote for "Improve tag siblings/parents and tag 'censorship'" here >>12152 .

>>11759 sauce for the first image?

>>12060 >>11759 able to post all images visible? I like what im seeing

>>12127 https://www.pixiv.net/member.php?id=16040178

is it possible to add in a option for color managment just to point to an ICC Profile? Would help greatly for people using an ICC profile on Windows color managment or or programs like displaycal.

>>12076 >>12077 >>12081 Hey, I apologise, this ICC profile tech is completely new to me. I will make a job to look into this, specifically how much control I have over it with wx or OpenCV/PIL, for next week.

>>12079 >>12080 I took some extra time yesterday evening (and just now) to finish some bugs and cleaning up thumbnail code. Going to test it a bit more, so release might be delayed a little bit. I feel good about it though, so no chance of anything going wrong™.

>>12014 Yeah, svgs are basically XMLs with a specially defined namespace at the beginning. If you ever get to document/xml handling, namespaces are probably the way to go to differentiate them from say, html files. The mozilla developer network has a surprisingly good primer on the stuff imo: https://developer.mozilla.org/en-US/docs/Web/SVG/Namespaces_Crash_Course I'm not versed enough in python to recommend a library, but considering how often python is used in math and how often svg is used to make charts, I'd wager there's a great one you can probably use.

>>12014 For initial parsing, have you considered libmagic from the "file" unix utility? It doesn't do much to extract metadata, but if it's just a matter of finding the proper MIME type it should do a good job. https://github.com/file/file/tree/master/python

>>12013 >have some sort of 'trust' network so users have more control over who they are talking to Wouldn't it be easier to start with some "all of this is shared" type bittorrent or such integration? There are good libs for that that are easy to use. You can define what is shared, but you'd omit the "with who" part for now. Sharing in a web of trust or with friends requires a good bit of UI work to make any sense, and security flaws will also upset people a lot more.

>>11962 ahhh… https://docs.krita.org/en/general_concepts/file_formats/file_ora.html

There may not be a formal specification for the .kra format it looks like, so source code might be your best bet, or just reversing it yourself as it's just zipped up junk. Here's the Krita stuff if it's useful: https://github.com/KDE/krita/blob/v4.1.8/plugins/impex/kra/kra_converter.cpp#L56 https://github.com/KDE/krita/blob/v4.1.8/plugins/impex/libkra/kis_kra_loader.cpp#L196 And here's GIMP's XCF loader code which may be of use if the specification is ever unclear: https://github.com/GNOME/gimp/blob/GIMP_2_10_8/app/xcf/xcf-load.c#L152

>>11969 >>11970 >>11972 Thanks lads. I'll queue these up and I'll see what I can figure out. Adding 'I can recognise it at least' .psd support wasn't a massive hassle this week, so I expect to do more of this sort of work. But I suspect I'll have to refigure some UI like the system:mime panel checkbox hell if we go too nuts here.

>>11955 The numbers there are the 'status' of that list of tags. In hydrus, a file's tags can have four different statuses: 0 - current - the file has the tag for this service 1 - pending - this tag is pending to the service 2 - deleted - this tag was current but has now been deleted 3 - petitioned - this tag is current and is petitioned for deletion Pending/petitioned relates to the 'pending' menu in the main hydrus client gui. Repository tags are not fixed as current/deleted until pending/petitioned are commited through that menu. In writing this out, I realise I fucked up here because a 'local tags' service won't ever have 'pending' tags. I will rewrite this help section to have it refer to a hypothetical tag repository.

>>11964 To follow up here, if you wish to edit a file's tags with /add_tags/add_tags, you will need to use this more detailed tag status information to figure out if you are 'petitioning' or 'rescind pending' or whatever. For practical purposes, if you want to calculate what tags the file currently 'has' for easy phone display, you can go: tags = ( current ∪ pending ) - petitioned Petitioned tags are pretty rare, so you can probably just go ( current ∪ pending ). You'll then be unioning those sets across all given service_names to get a file's 'all known tags' as you would see in the client.

>>11964 >>11965 Ah. I forgot tags can have statuses. Thanks for clarifying

>>11842 Yeah, for some complicated websites, an import 'item' produces more than one file or a new page to pursue. It is a technical pain in the ass, but for now, the x/y progress on an importer refers to 'import items processed' not 'potential files imported'. If you would like to check those 'already in db' files, you should be able to right-click on them and say something like 'open these in a new page'. Since in that note it says '10 seconds ago', they are almost certainly duplicates from above. (I don't know anything about nijie.info, but yeah, it looks like the _0 url without diff/main is a 'cover image' for the subsequent mini-gallery?) Again, some galleries give the same file twice, even on their nice APIs. I don't know why this parser pulls that 'cover', but I did not write it, so I can't confidently say this strategy isn't needed to catch edge cases. The nuts and bolts of this stuff are frequently fuckery duckery doo, particularly on Japanese sites.

>>11844 Thanks, that is interesting. I assume the lossless compression mode is functionally 4:4:4, right? But any lossy encoding is coerced to 4:2:0? My hesitance for webp in the past is that it still is sRGB, so I don't know how it is useful as we move to HDR in the coming years. Maybe they will do like webm and add new accepted encoders, but I dunno. It doesn't seem to be taking the world of internet browsers and phones by storm as it is. HEIF and FLIF seem to have their good points, but they are still similarly meme status for now. I'll play with animated webps a bit when I find time, as I'd really prefer not to use shitty gifs for animated thumbs, and I don't want to go completely nuts with apngs either.

>>11846 Thanks, I am glad you like it. When you get an app ready and want to share it about, please send me a link to it and I'll put it up in my release posts and help! Current plan for Client API is to get a simple 1.0 out the door. This should be done tomorrow for 243, where basic file search and file/thumbnail retrieval are finished. You'll be able to find all the files in a client with the tags 'blue eyes' and 'blonde hair' and then shows thumbs and files and tags. It should be possible to replicate very basic booru functionality. After the release tomorrow, please check the updated help here: https://hydrusnetwork.github.io/hydrus/help/client_api.html Which will have the last '/get_files/…' queries all filled out. With the 1.0 done, Client API work will then be fit into the regular weekly 'small jobs' schedule. If someone wants the ability to archive files or search by duration, I'll see if I can squeeze it into a small job. If you have ideas for your app, please submit them any way that is convenient–these release threads are fine, and you can also email me or even DM me on discord on Saturday afternoons US time if you want to talk live.

>>11789 Thank you for this report. My dark mode is a bit of a fudge–I only have control over the colour of my custom objects, whereas everything else (like window borders and backgrounds and button colours) is supposed to be inherited from your OS's current system colours. If you have a window manager that changes your OS colours, hydrus's dark mode switching can be helpful to make sure that my custom objects also change with it. There's a very speculative long-term plan to move from wxPython to Qt. If this ever happens, theming support will get significantly better, but until then it is all duct-tape, I am afraid.

>>11793 Thank you for this report. Are the updates processing during idle time or shutdown? If you have problems during idle time, can you change your settings under options->maintenance and processing to be shutdown-only, and, say, only 5 minutes? If that relatively small amount of work can go through, can a bit more, like 15 mins? Is there a significantly 'cleanup' lag, like 5 mins of work takes 3 mins to commit?

>>11810 Thank you. That is very strange. Here's SQLite's and Python's info on this: https://sqlite.org/tempfiles.html#temporary_file_storage_locations https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir So, for some reason, a different folder was not permitting a program launched from it to access any system temp location. Windows has simpler temp structure than Linux, so perhaps this really just meant stopping you from accessing the one under AppData/Local/Temp. Could somehow that directory (or say the client.exe) have got a compatibility mode or some other protected state applied to it? I am afraid this is way beyond my expertise.

>(including I think the Linux >0 distance similar files search crash, finally!) Was that at all related to any of my posts: >>11736 >>11758 >>11797 Because I was having an issue on linux crashing. In the process for searching for duplicates an SQL statement similar to SELECT phash_id, phash, radius, inner_id, outer_id FROM shape_perceptual_hashes NATURAL JOIN shape_vptree WHERE phash_id IN (1, 2, 3); would be executed and cause a segfault, not just for hydrus but anything that could execute an sqlite statement. Apparently Sqlite 3.27.1 (and possibly some earlier versions) had a bug where if the size of a list exceeded two items it just crashed. This appears to be fixed in sqlite 3.27.2. I imagine based on the code I looked at having a search distance greater than 0 would make it far less likely to happen if not impossible.

>>11804 There are quite a few places where the client fetches things with lists up to 256 items long, so if you found that just loading some files and tags was ok, I _think_ you weren't getting hit by that exact issue. But yeah, your posts are the bug I think I fixed. Only affected some Linux users, and I could not reproduce it. I worked one on one in the week with a user who had the crash and we ultimately figured it was a problem with SQLite hanging on to the phash while I did a Hamming Distance search on it (which requires some low level C++ struct calls and then some mickey-mouse CPU-level binary counting). Something to do with memory access violation is my best guess. Python usually can't crash by itself. Previously, I was doing hamming searches interleaved with the sqlite call iterating each row, but when I separated it into completed batches, so sqlite was 'done' with the rows, no crash happened. The 0 search distance could do a direct lookup without the hamming check, and that was running fine. It just seemed to be the combination of iterator on sql results and then low-level access on 'bytes' type response from that that caused the crash. Core dumps suggested the crash was in sqlite code, so I guess it was still hanging on to a bytes buffer or something that wasn't cleaned up. Could indeed still be a (different) bug in sqlite, or python's wrapper for it, but I dunno for real. Let me know how 342 works for you!

>>11801 Owarimonogatari >>11803 wholesome