/hydrus/ - Hydrus Network

Archive for bug reports, feature requests, and other discussion for the hydrus network.

Index Catalog Archive Bottom Refresh
Name
Options
Subject
Message

Max message length: 12000

files

Max file size: 32.00 MB

Total max file size: 50.00 MB

Max files: 5

Supported file types: GIF, JPG, PNG, WebM, OGG, and more

E-mail
Password

(used to delete files and posts)
Misc

Remember to follow the Rules

The backup domains are located at 8chan.se and 8chan.cc. TOR access can be found here, or you can access the TOR portal from the clearnet at Redchannit 3.0.

Uncommon Time Winter Stream

Interboard /christmas/ Event has Begun!
Come celebrate Christmas with us here

8chan.moe is a hobby project with no affiliation whatsoever to the administration of any other "8chan" site, past or present.
(4.03 KB 480x360 JEXW8reB57A.jpg)

Version 350 hydrus_dev 05/01/2019 (Wed) 22:09:28 Id: d3155b No. 12458
https://www.youtube.com/watch?v=JEXW8reB57A windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v350/Hydrus.Network.350.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v350/Hydrus.Network.350.-.Windows.-.Installer.exe os x app: https://github.com/hydrusnetwork/hydrus/releases/download/v350/Hydrus.Network.350.-.OS.X.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v350/Hydrus.Network.350.-.Linux.-.Executable.tar.gz source tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v350.tar.gz I had an ok week. Some IRL things cut into my hydrus time, but I got some good work done. Some bugs in the new duplicate search system are fixed, and I improved advanced file delete and export handling. The poll for the next 'big job' is up here: >>12358 duplicate filter The search addition to the duplicate filter went fairly well, but there were a couple of significant bugs. The 'ghost pair' issue–where a queue would sometimes have a final pair that would never display and lead to high CPU until the filter was closed–is fixed, and safeguards added to catch similar issues in future. The issue with undercounting on large search domains (typically where the dupe page's file query was non-system:everything and covered >10,000 files) is also fixed, but giving the filter 500,000 custom files to work with can be really quite slow. I will keep working here to see if I can speed up big searches like this without compromising accuracy, but if you find your dupe searches are working too slow, try adding a creator: tag to bring the search size down–it works well. The duplicate filtering workflow itself is still a v1.0 pain. I expect to put some work in here in the coming weeks (and doubly so if dupe work wins the big poll), likely highlighting the differences between the two images with an always-on-top panel and better at-a-glance decision-making for easy comparisons. advanced delete This week brings an optional advanced file delete dialog. You can turn it on under options->files and trash and also set custom file delete 'reasons' to assign. Once it is switched on, any file delete under the thumbnail grid, media viewer, or duplicate filter manual file delete will instead give a richer dialog that lets you physically delete files immediately (i.e. skipping the trash) and deleting files without leaving a delete 'record' (which is useful if you want to easily reimport those files later on). You can also choose from your set reasons or type a completely custom one. These saved reasons reappear when that deleted file fails to import in future (due to being recognised as previously deleted), so if you are interested in tracking why you have previously deleted files, please check this out. the rest Export folders now have more timing options, basically the same as import folders. You can pause them individually, tell them not to run periodically, and force them to run manually under a new submenu under file->import and export folders->run export folder now. The Artstation downloader seems to be not working due to a Cloudflare issue. I am leaving the downloader in for now, in case it starts working again, but if you have a subscription for it, I recommend you pause it for now. The default downloader for new users (and you, if you have it still set as the default) is now safebooru. If you use the Dolphin file manager, check out this add-on that uses the Client API: https://gitgud.io/prkc/dolphin-hydrus-actions Advanced users only: Missing file folder recovery is better–if a file folder (one of the subfolders, like 'f39' or 'ta2') is missing on boot but found in another 'known' location, the client will detect the moved folder and propose an auto-update of its records rather than forcing you to go through the complicated manual recovery dialog. If for some reason it is complicated or slow for you to move your folders from A to B through the migrate database dialog, you can now shut down your client, move the folders manually in explorer or wherever, and then reboot the client and be up and working in one click! Just make sure that hydrus knows about B in its locations under migrate database beforehand, and it will do the rest. full list - the duplicate filter no longer applies the implicit system limit (which defaults to 10,000 files) on its search domains, which solves the undercounting issue on large search domains. duplicate operations will be appropriately slower, so narrow your duplicate file queries (adding a creator: tag works great) if they take too long - fixed the duplicate pairs filter's 'ghost pair' issue. it was failing, when 'both files' was unchecked, to remove pairs that included one file that was non-local. this accidental inclusion resulted in incorrect non-zero count and filter/random pairs that could not display correctly - insulated against potential future iterations of this problem (likely that one of the pair was deleted by another process while a filter is ongoing), with a notification and graceful exiting of the duplicate filter while saving progress - the 'show random duplicates' button now puts the 'base' of the group (to which all the others are potentially matched) as the first thumbnail - added a new 'advanced file deletion' section to 'files and trash' options page to turn on a new advanced dialog and set custom file deletion reasons - if this new dialog is turned on, a delete event from thumbnail grid, regular media viewer, or the duplicate filter's manual delete will launch it. it permits you to delete physically (skipping trash) in one step or delete physically without leaving a deletion record (for easier later re-import) and choose one of the deletion reasons in the new options panel or set a one-time custom reason - export folders now have more run-controls: 'run regularly', 'paused', and 'run now' - the file menu now has a 'run export folder now' submenu just like for import folders– it is simple now to set up an export folder that only runs when you tell it to - updated the on-boot missing file folder recovery process to automatically 'heal' file location mappings when a missing folder is actually in a location that is known (essentially, you can now manually move a bunch of folders from one known location to another while the client is off and it will recover automatically now). error dialogs will appear in this case summarising the problem and proposed fixes with a chance to bail out if you do not want it to happen - added a new frame type to 'gui' options page called 'regular_center_dialog' for yes/no style dialogs that are better in the center of the parent window - the custom web browser launch path and file type 'open externally' paths are moved from 'files and trash' to a new 'external programs' options page
[Expand Post]- as the superior '–temp_path' program launch parameter now exists for both client and server, I have removed the limited 'BUGFIX: temp folder override' option from the client's 'files and trash' page and use in the actual code. if this option was important to you, please migrate to the –temp_path launch parameter, which covers temp usage more comprehensively - as the artstation downloader is now non-functional, apparently by a cloudflare issue, the default gug for new users (and anyone with artstation set atm) is now safebooru - added dolphin file manager add-on link to the client api help - some misc file metadata fetching cleanup next week Next week is a cleanup week. Beyond boring ongoing code cleanup and rewrites, I would like to move some 'default' image parsing to preference OpenCV over PIL uniformly, including for the server (which currently does not need OpenCV). Having image work preference one library over the other at different times causes problems when they disagree about some image metadata (e.g. whether a file is rotated). OpenCV is fairly easy to get for all platforms now, and it generally runs faster and better, so I feel good about adding it to the server and making it the primary choice in all situations. We breezed past 500 million mappings on the PTR in the past couple of weeks! This is great, and I am really thankful for everyone's contributions, but this growth is putting pressure on several different areas. You may have noticed we finally hit my 256GB/month bandwidth limit on the shared PTR account at the end of April. Regular PTR-syncing client databases are also getting huge, about 18GB. I have multiple plans for how to deal with the various issues here and expect to chip away at them throughout 2019. I may reserve a month or two of future 'big job' time to handle this work.
Anonymous 05/02/2019 (Thu) 00:09:28 Id: 93a027 No. 12459
>>12458 This is a temporary issue, as its likely going to be completely sorted out by the next update. I have 2000~ files in the trash at the moment and because they are in the trash, they can't get a custom reason despite none being there. I don't think this will be an issue that manifests itself going forward but if there was an override for the reason they were sent to trash to get a new one that would likely solve all future instances of this too. other then that, I am loving the system
hydrus_dev. Board Owner 05/02/2019 (Thu) 01:09:40 Id: d3155b No. 12460
>>12459 Hey, I am glad it is working for you. The reason is applied as the file is added to the trash, so if you would like to set a new one for those files, I think untrashing them (shift+delete) and then advanced retrashing will sort you out.
Anonymous 05/02/2019 (Thu) 10:26:31 Id: 93a027 No. 12464
(6.56 KB 363x211 client_2019-05-02_05-18-33.png)

>>12460 that's what I have been doing and why I said its a temporary issue. I do have a thought, and if this can be implemented would be appreciated. would it be possible to give delete presets a hotkey? Here, let me try to explain. lets say I have a random set, and i'm going through them just looking at them, and up pops something shit quality. Currently I have to use the mouse to select the reason not sure about you but when I use the media viewer I have a macro pad in hand or I have my hands on the keyboard if I could press delete and then, on my current set, press 1-8 to quick select which one I want that would be a step forward. this, at least for me, isn't something im going to be using yet, as I have a copious amount of images inside thread watchers and galleries that are en mass cullable, as currently i'm moving the images to their own windows and going through them in one fell swoop per reason, but once I get done going through everything in broad strokes, needing to open them 1 at a time and deciding will be the way forward. there is no hurry on this as its going to be months till I get to this point even if I sorted like its a 9-5 ————– with that said, I have a question I would like some input on as im not 100% on how to go forward with sorting these so opinions are appreciated. a lot of threads have manga pages in them, and I have downloaded a copious amount of them. A good portion of manga/doujins, I already have the books for, I currently have 500~ files left in trash adding up to around 500mb, If I had infinite space, I would keep these pages, however, I don't think hydrus shy of 10-20 more years of development will be able to sort these into what manga they come from automatically, so they will get lost over time in the system, and do be honest when they are taken 1 image at a time, the set up pages are fairly worthless on their own but money shot (be it porn or a non porn pay off) may be worthwhile. im torn between 1)getting rid of all them 2)leaving them alone for now and deciding later 3)only keeping money shot/worthwhile pages. the primary thought being I likely have these elsewhere, but I cant say for 100% certain, but If I have the money shot pages, if I don't have them, I can find them, and if I cant find them, I have the most 'valuable' parts for future searches. now that I can delete, I am not as pressed for space as before, but till I cull and till I go through everything, my lack of space concerns comes at a cost.
Anonymous 05/03/2019 (Fri) 06:43:48 Id: 93a027 No. 12476
(1.28 KB 178x43 client_2019-05-03_01-38-04.png)

(3.54 KB 1093x36 client_2019-05-03_01-38-24.png)

(2.28 MB 1280x720 test.mp4)

Ok, ran into a secondary issue that needs to be solved, not with the program, but with the files. so I did a search, one of them is everything over 10mb sans gif, and one is everything over 10mb gif in total, over 100gb of files are here. sorting thought these I found a very egregious gif, that was 112mb big, I just ran the damn thing though handbrake, and it went from 112mb to 2mb, I mean holy fucking shit, are you god damn serious with this? however I ran into 1 problem that doesn't allow me to go forward. I wanted them to be webm with user input cable quality, this would see things that benefit from higher bitrates/quality benefit while things like video related would get a more lossy version. i'm ok with the mp4, but if there is a drag and droppable way to do this, that is not painful for webm, I would rather do it that way. as it stands now, im going to hold off on converting and instead just sort things that I absolutely dont want and things I would keep if the file size was smaller
Anonymous 05/03/2019 (Fri) 23:50:25 Id: 33e4a2 No. 12480
>>12458 Am i missing something or is the duplicate filter not doing anything? I ran the filter for a day and clicked through a bunch of different pictures but then the next time I launch the duplicate filter it shows me the same ones I already sorted as filtered and "comitted 150+" changes.
hydrus_dev Board Owner 05/04/2019 (Sat) 19:27:11 Id: d3155b No. 12481
>>12464 The shortcuts system is another one in mid re-write atm. There is a still lot to do just in getting all the simple commands working on the new system, so I'm afraid I don't think I can get advanced commands like 'delete with this reason' going soon. Can you maybe navigate the new delete dialog with tab and arrow keys? This is probably inconvenient, but could you maybe macro 'tab tab down tab down down tab enter' to your keys for the different reasons, just for your delete sessions? Yeah, the single manga pages is a tricky issue. My belief is that files' value is not in their existence, but in what you get out of them. If you will never look at them or want share them again, just delete them. If you want to keep some single good pages, keep those. Otherwise delete. Don't spend today worrying about integrating them into comic pages 15 years from now as we'll have a whole bunch of better tools then that'll solve all these problems better. Focus on what you want to look at in the hypothetical today. I'd say do 3. >>12476 The SFM Community™️ has a long history of these 1080p gifs and other silliness. Same as artists putting up 'original' quality 60MB 14,000x22,000px pngs on their patreon. Artists tend to be excellent at aesthetics and less so at practical file formats. There is no solution in the client atm, but I can definitely see some optional auto-conversion in the future. I taught myself how to do some more advanced ffmpeg conversion the other week, so I know a bit more what I am talking about here. I think we'll see how video dupe filtering goes, once I've added that, and then chase up with various systems for easy conversion/optimisation. Big gifs are stupid almost all the time, and I should think most users would rather have video if we can figure out a decent conversion regime (the dupe checker tech could help with this). And I wouldn't do it myself, but I know some users are thinking of converting all their jpegs to webp or flif or whatever, and hydrus could facilitate that sort of thing as well. I wouldn't be surprised if these older file formats will be completely subsumed by better systems in future. Especially so as we move to higher colour depths in all formats. You may not have seen, but hydrus supports bmp importing, but it auto-converts to png.
hydrus_dev Board Owner 05/04/2019 (Sat) 19:40:59 Id: d3155b No. 12482
>>12480 Thank you for this report. Can you say a bit more about it? What sort of tags are you using in your duplicate filter search? If this is on 349, that had an undercounting bug, so if you were clearing your apparent queue down to '0 pairs' and then next boot it went up again, this is now fixed in 350. If you are seeing similar files in subsequent filters, is it possible that you are just seeing some 'similar' looking files, and not exactly the same ones? The way the current pair-processing queuing logic works, I have to separate decisions on 'groups' of similar files into separate batches of 'committed' so they do not step on each others' logic. Can you see if you can find out more about a potential group here? I am thinking something like this: Open the dupe filter and work until you find a file with some decent tags that you can find again (Even better if you think you have seen that file in the dupe filter before) Do not process that pair and exit the filter Find the file(s) again in a regular search page with their tags Right-click->view this file's duplicates->potential (you may need help->advanced mode on to see this) Are there only two files, or more? This isn't surefire, but if you discover you have seven very very similar files, that would explain this. You could have dupe-filtered A-B on one cycle and A-C on another and A-D and then B-D, all depending on what their relationships were. Alternatively, can you reduce the filter domain so it is only, say 10 pairs? Just add some tags to the search until it gets small. If you process those 10 pairs and refresh everything, does it stay '0', or does the count go back up? If it stays at 0 then your pairs are being processed, and that suggests you are seeing multiples of very similar files. In fact, in general, if your 150 changes (or even just 2 on a test run) reduce the 'potential pairs' count by at least 150, then I feel fairly confident it is working basically ok here. All that said, I know there are bugs in the system. I don't think there are any big enough to drop 150 pairs in a whole batch, but I suspect one or two transitive implications are under- or over-applying. I am looking forward to likely working on the db-level here as part of my next big job, as the poll seems to be going atm.
Anonymous 05/04/2019 (Sat) 21:49:58 Id: 33e4a2 No. 12488
>>12482 v349 I didn't use any tags. I just let the duplicate filter try to find potential results. I'm 90% sure that it was giving me the same files I already sorted. Does the dup filter only move tags or does it put them in the trash? I couldn't find any recently filtered ones in the trash and it gave me the same files after running it again, this time it did move some to the trash though so maybe it wont show up in the dpilicates processing again.
Anonymous 05/05/2019 (Sun) 16:44:34 Id: 8fad24 No. 12498
>>12458 >and deleting files without leaving a delete 'record' Right when I was going to ask about that. Thanks. I think a better term for this action is 'forget'. How does deletion reason/forget/… work with "archive/delete filter"?
Anonymous 05/06/2019 (Mon) 08:16:50 Id: 93a027 No. 12499
>>12481 With delete, I don't mean delete with reason, what I mean is in current hydrus, there are various hotkey presets depending on what you are doing So when you get into a delete menu, its hotkey preset takes over, and like I said in my case it would be 1-8, pressing the number could select one of the reasons. Like I said before, my use case for this is still quite a bit down the road because mass culling is still easier and will be for several months. no idea if this is possible in the current shortcut method but if it is, it would be something to do to fill time if its easy enough. as for manga i'm leaning on keep the 'money shot' pages If I read the manga, it will make me laugh/make me want to re read it, if I didn't, I still may get something out of it, if its porn, well… that one obvious. as for conversion… personally with that 112mb attrosity, I got it down to 1.5mb after fucking with settings in handbrake for a while For what to do in the future… that a a bit harder first you would need a 'lossless' conversion, as some animated gifs are bloated to hell and back, but they have very clean lines, and a conversion even as lossless may end up at a lower file size then non converted. second you would need a near placebo preset, not quite there but still high quality third would be a high quality compression, a good catch all for everything, forth would be another relatively high quality preset, lower but still a catch all and below that some fuck this presets, I have a few videos where they made a music video of sorts, but it's over 200mb and the only thing that's worth anything from it is the audio, but the visuals are a nice touch. a good variable bitrate could/should crunch that down to acceptable levels, but at least with handbrake currently, I don't think there is a way for me to do that without fucking with audio, whatever presets you make, if its possible to do nothing to the audio, all the better. at least as far as I can see, these are the only presets you would need in hydrus, as if anyone wants to do more, an outside conversion would likely be better. as for other formats, If I can go lossless to lossless and save space, I would convey everything immediately, then convert back on a share. for jpeg or lossy, if I can convert to, get 95%+ of the quality, and save a receivable amount of space, it's another no shit for me, but that's just me. as for bmp to png, I think it may have come up a few times dealing with my older dbs, but for the most part, bmp has never been something I collected/found. On the topic of sfm, I know a little about 3d rendering, you don't render out a movie, you instead render out images, just because if the program crashes or fucks up 1 image, its a lossless way to re render. you can also stop the rendering and resume easier. so putting said images together into a gif opposed to figuring out editing program is probably a lot easier. granted im able to straight drop these into handbrake so the value of the gif is iffy at best. on the side of 14000x22000 artist images, again this is where I know quite a bit. working at a higher resolution, even if you dont save and distribute it at said resolution, you are able to do color mixing and are more easily able to ignore minor fuck ups then if you were making a 1080p at 1080p, it also allows you to use something like a textured brush so you get smaller details without trying without making it obvious you used a textured brush. on the side of the artists selling them, the ones who do it right are selling them as a teaching tool, and the morons who only buy them for the highest quality… got a very low opinion of these idiots. that said you make a great point. if duplicates is able to deal with video, then what ever I do on my end to shrink the shit, will get a higher quality version found if a higher quality version exists. was honestly not even considering that when I asked. that makes this a bit easier for me to move forward
hydrus_dev Board Owner 05/13/2019 (Mon) 16:22:23 Id: d3155b No. 12539
>>12488 I dove into the code here as part of my research into the dupe db overhaul yesterday and discovered some problems in my original 1.0. I feel less confident about it as I did for certain edge cases where 4+ files want to figure out what worst->best order they should go in. What happens to tags and whether the 'worse' of a pair is deleted is up to the duplicate merge options (check the cog icon on the dupe filter to edit these). I believe the defaults are to delete the worse of a pair, but if for some reason yours is not set this way, I think this will help your processing. Clearing 'bad' files out of the local file domain clears out some unusual AB, BC, AC pair comparisons the current system wants to do. My plans for the new storage system will do away with the over-autistic comparisons here and push for a simpler model with groups of dupe files with a defined 'best quality' King.
hydrus_dev Board Owner 05/13/2019 (Mon) 16:23:37 Id: d3155b No. 12540
>>12498 The archive/delete filter still uses the old delete system atm, just sending files to trash with a default reason. I would have added the new delete dialog to it, and likely will in future, but the batched way archive/delete filter stores up and commits its actions make it more complicated to integrate, so I put it off.
hydrus_dev Board Owner 05/13/2019 (Mon) 16:29:30 Id: d3155b No. 12541
>>12499 I will not hardcode number shortcuts for certain controls for now. I think macros to tab&space navigate the dialog are your best bet atm. That's interesting about SFM, I didn't know about the image-based rendering. I had heard the software tends to crash a lot, especially as artists are often working on older laptops etc… Sounds like a nightmare. I assume there is no easy one-click open source workflow–that for instance just neatly eats up their pngs and throws them at ffmpeg–for these users, unless Handbrake can do something like that?
Index Catalog Archive Top Reply
1
Forms
Delete
Report
Quick Reply