>>12951 Based and redpilled.

wow very cool

>>12677 >>12842 https://standardebooks.org/ebooks/marcus-aurelius/meditations/george-long

>>12678 Stop lying about being the dev. Stop lying to yourself to achieve social results you don't actually want. Stop lying in saying that you didn't mean to say anything. Stop lying because if someone gave you what you asked for, you'd push it away because you didn't trust it. Don't lie. Ever.

>>12841 Yeah, I do plan to add videos to the duplicate system, and I originally designed it to eventually support them. The recent file maintenance system was a step forward in prepping for the CPU work we'll need to do to retroactively crunch the data on this in a reasonable way. I plan to do something like what you propose. The duplicates system currently works on comparing still images' shapes with each other, and it allows for multiple still image 'phashes' per file, so my task is selecting a good number of useful frames from videos that will match with others. If it is reasonably possible, I would like do something more clever than just picking one frame per x time units or frames. This would line up right if our two vids were very exact conversions or resizes, but some of the codec changes drop a frame at the start or do 29.97fps vs 30fps bullshit that would desync our comparison. My original duplicates system did add vids by using the first frame, but so many have a black/flat colour first frame that it lead to a billion false positive dupes. Vids are no longer included, and I also drop anything that looks too much like a flat colour image from the system entirely. If I could instead find the x most 'interesting' frames of a video, then 2-second gif clips of 20-second webms would have a higher chance of being matched, and 30/60fps conversions would too. I don't know, though. That is probably beyond me to do well, or maybe I can hack something that is good enough. I could do something like generating a phash for every frame in the vid and then have them compete with each other to remove similar-looking frames/phashes until the 20 most unique were left. It might pick up a bunch of false positives again with, say, a black screen with a bit of white text on (like an 'intro' title card) though. Still, I am almost ready to do this now, and dupe work is proceeding, including more efficient storage of potential dupes, so maybe the answer here is to get a simple system in and then iterate on it.

>>12846 when I say literal garbage, I mean the image is drastically different, to the point I cant even see how it thought they were dups, but those are from the asshole who specifically fucked with dup detection when creating trash images, the dup detector is able to have enough wiggle room that even if the images aren't lining up perfectly, it may spit out something useable, and I used 1 second just because no mater what I watch 1 second isnt enough time for a 100% scene change, it should pick up some duplicates from that. on the title card, you could make a generic here is a black frame with text, and have a few variations of it, this could be used as a compare to X image and trace it if it does, so it would automatically know that everything with it will bee seen as a duplicate. If you are able to, try to get in contact with the people from what anime is this, and see how they did theirs, it may give some ideas.

>>12846 Could always check how Video Comparer works. It's the best video dupe finder software I've used.

>>12713 Move left, move right, move first, move last on the video tab I have stupidly large remove convert down not hydrus view again and decide with check before remove music video in for good measure

Ill post this again in the new thread too, I decided that the converted down images will get will get a rating of Converted Down/Keep Large with 2 stars and not selectable as an option it allows files to either be converted down and hidden from a search, or specifically keep the large file and hide it, also removed all files from the archive, everything needs a filter pass. these images that were converted down were interesting enough for me to keep/convert down but not really sure if I will long term storage them or not. when I en mass rated them to converted down I got that error InterfaceError Error binding parameter 0 - probably unsupported type. Traceback (most recent call last): File "include\HydrusDB.py", line 561, in _ProcessJob result = self._Write( action, *args, **kwargs ) File "include\ClientDB.py", line 12905, in _Write elif action == 'save_options': self._SaveOptions( *args, **kwargs ) File "include\ClientDB.py", line 10112, in _SaveOptions self._c.execute( 'UPDATE options SET options = ?;', ( options, ) )

>>12753 Thank you for this report. This is an odd error–the problem here is that the options object is not being serialised into the db correctly. This is the kind of error I see when someone has like a 2GB subscription, and SQLite falls over trying to make a buffer big enough for it. The options object there is a small thing, automatically saved at various points. Perhaps it was being nullified in some way, or some invalid data was being added to it. I can't think immediately why it would be affected by a rating set event. Can you say more about this rating conversion? You had a 2-star rating service as set up as under manage services, and then when you did a giant ctrl+a->f4->set rating->ok, it took a moment to write that and then popped up this error right at that time? Had you set any other options recently, either in file->options or via one of the 'cog' menu buttons? Anything related to default sort based on that new(?) rating service?

>>12633 I suggested 30 minutes as the obviously safe option, where no one could possibly watch a preview for that long, but the majority of AFK sessions would be longer than that. I wouldn't expect to see much reduction in sessions pruned by going with 10 minutes, but who the hell is legitimately watching the preview for 12 minutes and wants that time recorded? Honestly, I'm not convinced this needs to be a setting. 10 minutes is way more than almost anyone will ever watch in a thumb. 60 minutes would cut out the vast majority of AFK sessions. It's such a broad range to work in that 30 minutes would do the right thing in 95%+ of cases (and I'm being cautious with my numbers), so why add more clutter to the settings for something almost no one would need to change?

>>12633 >>12654 Could any potential solution for this be extended to the media viewer? This seems like the same problem of getting distracted during, say, an archive/delete session, and leaving one image open (but minimized) for a few hours. I've had this, but also a similar problem with the media viewer (pic related). I basically had a bunch of 'alternate/duplicate' files, and I was flipping between all of them hundreds of times trying to decide which to keep, tags etc. Is it possible to detect if the client doesn't have focus, and stop tracking immediately? I feel like that would solve 90% of issues right off the bat. Otherwise, tying it in with the client's inbuilt 'idle' state could also work well (and is already user configurable in options -> maintenance and processing).

>>12656 >>12654 Thanks lads, only catching up now. This isn't in yet, but I'll keep this in mind as I do this system. I'll do min/max for media viewer as well. Options are easy to add, and I know someone will want to say 'no minimum time' or whatever, so I'll throw them in for anyone who is interested. 5s/10m sounds like an ok default min/max. I think I can catch a focus lost event, although some of that stuff is a little unreliable. I'll play around with it and we'll try iterating on this a bit.

>>12758 I do both. >you might have a bunch of images you dont care about That's what the archive/delete filter is for.

I have 1TB of inbox to go through while 800GB are now properly archived, as in, I have personally checked them and stuff. I've stopped working on dupes till the current dupe rework is done, since alternate not copying tags nukes my workflow, sitting at 80k dupes.

OP here, down under 60k dupes finally. >>12758 I save those to Downloads with the most important tags in their title for later import.

>>12751 I love hydrus man and hope he is doing well in all aspects of his life.

>>12752 >>12751 I am overall healthy and in a stable money situation. I hope you are as well. I get blackpilled sometimes, but then I remember my true troubles are all internal and that I have a lot to be grateful for compared to my ancestors. I still want to keep pushing on Hydrus every week for the foreseeable future. I love imageboards and all other Anons, including you–the cheeky fun we get up to keeps my soul going.

>>12754 Based and Bloomerpilled

>>9960 https://github.com/mserajnik/hydrusrv-docker - Hydrus docker https://github.com/mserajnik/hydrusrvue - Vue Frontend https://github.com/mserajnik/hydrusrv - Media serving https://github.com/mserajnik/quicktagr - media tagger with Szurubooru https://github.com/mserajnik/szuruporter - Hydrus to Szurubooru https://github.com/bakape/hydron - Hydrus rewritten in Go

Faptchas (classic) https://github.com/314parley/Hydrogen https://github.com/Laurelai/tsukiboards MotionCaptcha https://github.com/wjcrowcroft/MotionCAPTCHA Audio Captchas https://github.com/dchest/captcha https://github.com/lepture/captcha https://github.com/mojocn/base64Captcha Captcha cracking https://github.com/debasishm89/hack_audio_captcha https://github.com/mikeyy/nonoCAPTCHA

>>9963 These are also good JQuery alternatives https://github.com/madrobby/zepto https://github.com/dojo/dojo >>10008 https://gitgud.io/Dizmal/borehole A fellow PHP-fren made this

>>12593 >>12594 Yeah, try tag siblings: https://hydrusnetwork.github.io/hydrus/help/advanced_siblings.html The system is very imperfect, but it generally works. More time will be put into this this year, including better clientside preference management for tags synced over the PTR.

>>12596 There is no solution for this yet. I expect a future version of the tag siblings system to allow some sort of regex-based replacement (think a global rule for 'replace all "\s" with "_"') that will allow this. Since desiring underscores to either go away or be mandated is common, I am likely to hardcode a solution to manage this earlier than a generalised regex solution, but the tag siblings system just isn't clever enough to handle it yet.

>>12629 >>12632 I have yet to come down to a good description for what namespaces are myself. Here is my current first draft: Namespaces are good when: They are a higher category (evangelion is a series) They are useful to highlight (characters are important) They are useful to search for (creators are often searched for, and 'creator:*anything*' has value) They are useful to group together (multiple creators is useful to know, when listed next to each other) I've been approving 'clothing:' siblings on the PTR for a little while now and I overall like it. I'm mixed on some others though. My ideal solution here is to extend tag siblings to allow clientside preferences and then for you to say "If a group of siblings includes one with 'hair:' namespace, prefer that". I have found that it is easy for users to objectively agree that 'hair:long hair' and 'long hair' have the same semantic meaning, but the big subjective disagreement is over which is better. An actually bad namespace, imo, is one that breaks the first rule above. 'male:erection' is an artifact of how some gallery sites do various female/male focus on tags, but I would rather 'male erection' (which could nicely have parents 'erection' and 'gender:male'). 'erection' is not a 'male'. Also 'erection:male'. 'general:coffee cup' and 'object:pencil' are technically correct but overspecific for most users, but again I think the specificity of namespaces is highly subjective, so the true answer is to let users define what namespaces tags could have and let them then customise what they prefer to show.

>>12520 Yeah, I don't like the sort of the sort. I've found it tricky to figure out a concise way of saying some of this stuff, like age vs time imported. Yeah, perhaps something like 'sort by dimensions: width' would help here to group them better. I'll make a job and think about this a bit.

>>12525 Thank you for this report! Sorry, I fucked up the new focus logic when adding the always-on-top ability. I'll 99.7% have this fixed for 352.

>>12554 personally, just sorting them between file facts and database entries would be enough. database would be things like tags, rateings, so on so forth that is really only applicable in program file facts would be things like pixel amount, file size, hight, width so on. this would at least clearly separate the two. >>12551 now that you say putting it on another monitor, a floating window would be perfect. Currently I have a 4k monitor, I have the files open as the right half of the monitor, a magnifying glass as the second monitor, and if this stuff was floating, I could easily possession it wherever the hell I please and not care at all. not sure how easy that would be but that would be a solution that works as a fits most use cases.

>>12506 Check the big list of repos in >>12295 especially https://github.com/andrewekhalel/sewar (it will come in handy)

>>12511 >>12506 1a)I am chiefly limited in how much time I have. I'd love to do more complicated autodecision workflows and backup ui, and I think that sort of thing would help a lot, but I don't have time in this rewrite. I also read up on some jpeg quality estimation here https://www.politesi.polimi.it/bitstream/10589/132721/1/2017_04_Chen.pdf , but it is too complicated for me to implement in the time I have. I would also like to focus on the db side more in this cycle. 1b)I am moving to a simpler comparison system in this rewrite. You'll always be seeing files compared to the 'best' of a group once it is done, which should exaggerate filesize and resolution differences. I am squeamish about autodecision on filesize or resolution alone as there are plenty of stupid bloated pngs of jpegs out there, but I think that bias could be part of a larger system that takes multiple variables to auto-decide. 1c)Sorry, I just don't have time to write clever ui like this atm. 1d)Yeah, I am afraid of the edge cases here. My thrust will always be default to off and lots of user customisation. I'll prep any auto-system with rules like "if exact same pixels and one is jpeg one is png, the jpeg is better", but I'd like for you eventually to be able to write your own rules for what you want out of it. 1e)Yeah, single pixel edits are a problem here. In a future iteration of any autodecision system that took multiple rules to make decisions, I think a blanket "if one pixel different, the older file is better" could be the ticket. There's also issues with file metadata being stripped or altered by CDNs. 2a)Yeah, that's the difficult stuff. Any dupe filter can't be simple, or any simple rules should be able to gauge certainty and pass the decision up to human eyes when something smells fishy. This clashes with situations like waifu2x, where it is a blow-up of the original, but a clever and presumably desireable one. 2b&c)My thoughts on the autodecision system would be to have it run in the background for very easy decisions, but confirming decisions (or maybe just some decisions with low confidence) with the user could be another way to go. 3)Duplicate metadata is not deleted when files are, so the delete/reimport cycle doesn't affect it. Saying 'remove this from all dupe pairs and requeue in the system' is not easy at the moment (I think you'll have to do a bunch of right-clicking on the thumbnail in advanced mode to show the exact pairs and then sever the relationships and requeue), but this will be easier with the new data structure I am designing. In the new system, I'd also like thumbs to load with their dupes, whereas at the moment that info is only fetched on right-click.

>>12542 yea all of 1 more or less required the image difference comparison to be a thing. as for 1e its not even single pixels, its resaveing the exact same image over and over again as jpeg slowly corrupting it over the course of 100-150 images. that's where I see a slow better worse auto filter jumping from a good image and quickly ending with the most corrupted one is no fail safe was implemented.

>>12488 I dove into the code here as part of my research into the dupe db overhaul yesterday and discovered some problems in my original 1.0. I feel less confident about it as I did for certain edge cases where 4+ files want to figure out what worst->best order they should go in. What happens to tags and whether the 'worse' of a pair is deleted is up to the duplicate merge options (check the cog icon on the dupe filter to edit these). I believe the defaults are to delete the worse of a pair, but if for some reason yours is not set this way, I think this will help your processing. Clearing 'bad' files out of the local file domain clears out some unusual AB, BC, AC pair comparisons the current system wants to do. My plans for the new storage system will do away with the over-autistic comparisons here and push for a simpler model with groups of dupe files with a defined 'best quality' King.

>>12498 The archive/delete filter still uses the old delete system atm, just sending files to trash with a default reason. I would have added the new delete dialog to it, and likely will in future, but the batched way archive/delete filter stores up and commits its actions make it more complicated to integrate, so I put it off.

>>12499 I will not hardcode number shortcuts for certain controls for now. I think macros to tab&space navigate the dialog are your best bet atm. That's interesting about SFM, I didn't know about the image-based rendering. I had heard the software tends to crash a lot, especially as artists are often working on older laptops etc… Sounds like a nightmare. I assume there is no easy one-click open source workflow–that for instance just neatly eats up their pngs and throws them at ffmpeg–for these users, unless Handbrake can do something like that?

>>12504 Yeah, we might be fucked. 560M mappings already. I'll probably reserve the next 'big job' after this dupe work to trying to sort it out. A more temporary fix I may apply is a daily 10GB limit, which will stop uploaders from sending so much and throttle the incoming tags more smoothly over the month.

>>12509 *sigh If only we have a more web-friendly way of dealing with tags… Not saying social media works, but it is the first thing that came to my mind e.g. voting, crowd moderation etc. using psuedonym? Sorry dev

>>12513 No worries. Having too much is a nice problem to have, and I have many ideas on how to mitigate our problems. Clearing out the (1) and done tags like 'title:tuhentu honeuhoen tuhoe nthn', or giving users options on how much of that stuff they want to sync with, will be a likely next step. There is plenty more we can do here, and I am keen to keep on pushing. I am committed to keeping person-to-person interactions Anonymous in my code–boorus already do username-based curation much better than I have time to do–but I know I can give users more power to share and sync just what is useful to them.

This information is useful and interesting, but I unfortunately do not have time to go through it in any clever way. I can't work on big things like new duplicate finding algorithms in normal weekly work–it'll have to wait for the next iteration of the duplicate system. I'd also like to prioritise the duplicate processing workflow, which is the larger problem with the system right now. I am most interested in face and other feature detection in cv. I expect to use some of this stuff when we push towards machine learning auto-tagging.

>>10348 For the dev: https://medium.com/machine-learning-world/feature-extraction-and-similar-image-search-with-opencv-for-newbies-3c59796bf774 https://github.com/creotiv/computer_vision

>>10338 In case of ghostbin closing down https://pastebin.com/KPaYiXNM

>>12467 >>12468 >>12469 Really? https://pastebin.com/vzQTRFaa since ghostbin is shutting down

Thank you for this report. This actually isn't my code failing here, but the image library I use, Pillow. Just guessing, I suspect these files are slightly malformed (broken) and Pillow is having trouble dealing with them. As it happens, I plan to do some cleanup work next week to have a different library, OpenCV, do the initial metadata parsing work here and improve reliability of metadata parsing overall. I will test these URLs once that is done and see if they work. Thank you for the examples!