/hydrus/ - Hydrus Network

Archive for bug reports, feature requests, and other discussion for the hydrus network.

Index Catalog Archive Bottom Refresh
Name
Options
Subject
Message

Max message length: 12000

files

Max file size: 32.00 MB

Total max file size: 50.00 MB

Max files: 5

Supported file types: GIF, JPG, PNG, WebM, OGG, and more

E-mail
Password

(used to delete files and posts)
Misc

Remember to follow the Rules

The backup domains are located at 8chan.se and 8chan.cc. TOR access can be found here, or you can access the TOR portal from the clearnet at Redchannit 3.0.

US Election Thread

8chan.moe is a hobby project with no affiliation whatsoever to the administration of any other "8chan" site, past or present.
(10.75 KB 480x360 9rP1wjEsbak.jpg)

Version 356 hydrus_dev 06/19/2019 (Wed) 22:29:55 Id: 55d74e No. 12978
https://www.youtube.com/watch?v=9rP1wjEsbak windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.Windows.-.Installer.exe os x app: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.OS.X.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v356/Hydrus.Network.356.-.Linux.-.Executable.tar.gz source tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v356.tar.gz I had an excellent and busy week catching up after E3. Duplicate better/worse/same relationships are now stored more logically and efficiently, and a variety of bugs are fixed. Maintenance is also easier to control. duplicates The second phase of the big duplicates rewrite is now done! Rather than the old system of trying to store better/worse relationships for every possible member of a duplicate group, which is very complicated, requires many human decisions, and is not very useful in practise, this new system instead lumps all duplicate files into a single group with a single 'king' representing the best file of that group. The groups are transitive, so putting a file into a group with a better/worse/same command–or even merging a group into another–is now just a single action. Rather than comparing a file with every member of a group, it is now just merged as a simple member or the new king. The overall workflow of dealing with large duplicate groups is now much much faster. Furthermore, alternate relationships are stored between groups of duplicates, not individual files, so again the number of choices you need to make is automatically reduced as you process your normal duplicate filter queue. Searching and managing duplicate file groups is also easier. When searching for system:num duplicate relationships, you simply search for how many 'duplicates' a file has, rather than how many better/worse/same. Right-clicking on a thumbnail to open up the 'file relationships' menu–which is now available to all users–now presents the same simple 'duplicates' count-and-show-in-a-new-page entry and will say whether the file is the best file of its group. If the file is not the best of its group, you can view that king or set the current one as king with one click. Future updates will show king/not king status in the thumbnail UI. I will also write a new system search predicate to filter for king/not king. Your existing better/worse/same data will be converted to the new system on update, with appropriate kings chosen. If in recent weeks you noticed some files that seemed to be both alternates and duplicates, this will now auto-resolve as the duplicates are merged together. The new jpeg quality estimator, which will sometimes present 'high quality vs low quality'-style statements in the duplicate filter, now appears for all users. It isn't perfect, but it works ok. The last phase of the rewrite will be to update potential pairs. It will reduce the potential pair queue count even further and improve the pair-elimination efficiency of the decisions presented in the filter (essentially by forcing king vs king comparisons). I want to take a week to do some smaller duplicates work (like commands to undo/dissolve existing relationships intelligently), so with luck I expect to have that done for 358. I am overall very pleased with the work here so far, although there is still a good amount to do, and some proper new help to write up. the rest I fixed the popup spam that the new clipboard watcher was falsely making on many non-matching URLs. It was some test code that was misfiring–even though the URLs were not being added since they were not matched, a higher part of the system thought they were not being added due to a problem with pages. I am sorry for the confusion–it is now fixed. Also the unusual OS X shortcuts issue, where shortcuts were not working on some OS X installs until a menu was opened! This was due to a weird OS X specific problem in the recent main gui menubar detection code. If you are on OS X and notice your menubar stops updating, please let me know. I put a bit of time into the maintenance code this week. Most maintenance jobs, like repository processing and thumbnail regeneration, now use the same 'clock' to figure out if they should be working. They now know if they are running in idle time or shutdown time or forced by the user, and will cancel out faster depending on the situation as needed. The other upshot of this is that the file menu now has a 'exit and force shutdown maintenance' option, and the shutdown splash finally has a 'stop shutdown maintenance' button! I regret that this took so long to come in–but now it is, and it works well! You can now also force file maintenance from database->maintenance->file maintenance. It tells you if no work needs to be done. full list - duplicates: - moved better/worse/same quality duplicates relationships to the new 'king' group-based model. rather than tracking every relationship, duplicates are now stored in groups with a single 'best' file - as a result, duplicate relationships are now transitive! saying that one king is duplicate to another will merge groups. the 'better' king is the new king, and 'same quality' kings choose one of the kings pseudorandomly. advanced exceptions: saying that a king is better than a basic member of another group or saying that two members are same quality is still valid but will simply 'poach' the non-king member from the other group in order to ensure the wrong king doesn't end up on top in the eventual merge. saying KingA is same quality as MemberB will merge the groups with KingB as the new king (since it is presumably same/better quality to all members of A) - the thumbnail right-click 'duplicates' entry is now renamed to 'file relationships' and is no longer advanced mode only. the 'find similar files' entry is folded into this - the thumbnail 'file relationships' menu now shows a simple 'duplicates' count rather than the old messy better/worse/equal. it will show all the members of a duplicates group when clicked. the menu also notes if the focused file is the best file of its group. if it is not, you will get the option to show the best file or make the focused file the best - as a result, it is now much simpler to view a group of duplicates and overrule a 'best quality' member as needed - added the 'media' shortcut 'duplicate_media_set_focused_king' to shortcut setting a 'best quality' file
[Expand Post]- the system:num duplicate relationships now has the simpler 'duplicates' entry, to search on size of the entire group. searching for kings/not kings will come soon - due to the new duplicate transitivity rules, potential pairs are now eliminated at a much faster rate! - setting duplicate relationships will overrule false positive or alternate relationships already in place - manually setting alternate relationships to more than two thumbnails at once will now set each file as alternate to every other file in the selection, completely eliminating potential pairs within the group. if you try to do this to large groups of files you will get a longer yes/no confirmation message just to make sure you aren't overwriting some potential dupes by accident - all existing better/worse/same relationships will be converted to the new group storage in this update, with appropriate kings determined. potential pair queue counts will be reduce accordingly, and the temporary alternate/duplicate confusion from the alternates update will be auto-resolved by merging truly duplicate 'alternates' together - fleshed out the duplicate test code significantly to handle the new dupe groups and their interactions with the recent false positive and alternates changes - refactored some db test code into separate client/server/duplicates files and cleaned up dupe tests readability - potential pairs are now the only component of the new system still on the old pairs system. the duplicate filter will still serve up some inefficient (i.e. non-king) comparisons - . - the rest: - fixed the issue where many clipboard-watcher-caught URLs that did not match were producing false-positive 'could not generate new page for that URL' error popups - the clipboard text-fetcher now tests against incompatible clipboard types (like a screenshot) better, and all instances of text fetching now report errors more gracefully and with more information - fixed the unusual OS X issue where many shortcuts were not being processed after client boot until the top menubar was opened and closed. a variety of other blocking-while-menubar-is-open issues that were false-positive misfiring are now fixed as well, please let me know if you still have trouble here - the file menu now has an 'exit and force shutdown maintenance' option to force-run outstanding maintenance jobs - when shutdown maintenance work is going on, the shutdown splash screen now has a 'stop shutdown maintenance' button! - cleaned up some file maintenance manager maintenance locking and shutdown cancel logic - moved all the idle-mode maintenance checks to a new system that explicitly defines idle/shutdown/forced maintenance work and tests those states in a unified manner, checking idle mode and the new splash cancel button status and so on more reliably. a lot of maintenance should cancel out quicker when appropriate - misc shutdown logic cleanup - added a 'file maintenance' option to the database->maintenance menu that forces the new file maintenance manager to run its queue. it'll make a little popup as it works, or a note that no work is due - the 'regenerate' thumbnail menu is also available to all users - jpeg quality estimates are now available for all users in the duplicate filter. they only display when the two jpegs' quality have different labels - the jpeg quality estimator now handles some unusual jpegs that load with empty quantization table arrays - the duplicate filter now handles bad jpeg quality estimations gracefully - cleaned up some ffmpeg communication code - the ffmpeg debug text that spawns on a help->about call that fails to discover ffmpeg version information now prints stderr output as well. if you have been hit by this, please give it another go and let me know what you get - the same ffmpeg 'no response' error on file parse now popups and prints some debug info and returns a better error - dialogs and windows on the new panel system now support a new pre-close tidying system - the manage tags dialog and window will now cancel any pending large tag autocomplete queries on close - regular gui pages now support a new pre-close tidying system - search pages will now cancel any pending search results loading or tag autocomplete queries on close - improved reliability of the popup message manager chasing the main gui when it is sent to another screen by a keyboard shortcut (such as shift+win+arrow on Windows). it should work now if the mouse cursor is in either window. please let me know if this causes trouble for virtual display navigation - the network engine now waits significantly longer–60s–on connection errors before trying again, and with every failed attempt will wait n times longer again. when in this waiting state, a manual user cancel command cancels it out faster - I believe I have fixed/improved a situation where media viewer hover windows would sometimes disappear immediately after appearing on some Linux window managers - improved hover window report mode to state more focus info in case the above is insufficient - to better link the two requests and consume bandwidth under strict rules more precisely, the override bandwidth rule that kicks in when a file page has a single file is now 3 seconds instead of 30 - updated options->connection page to specify that 'socks4a'/'socks5h' is needed to force remote dns resolution - sped up tag parents initialisation - repositories now group tag sibling and parent petitions by the parent/better tag's namespace - removed some old network 'death time' code that is no longer useful and was interfering with heavy petition processing - the log now flushes itself to disk every 60s rather than 300s - misc fixes and cleanup next week This week was a crush, so I focused on the duplicates work and bug fixes and cleanup. I have a variety of small 'new stuff' jobs to catch up on next week. Some more shortcuts to add, some client api work, some quality of life like pasting tags into siblings/parents. And some smaller auxiliary duplicates work. I enjoyed taking a week off, but I am glad to be back to work.
Anonymous 06/20/2019 (Thu) 02:19:53 Id: 80b899 No. 12983
macOS, Version 356: Unfortunately, the issue detailed in >>12936 remains.
Anonymous 06/20/2019 (Thu) 15:46:45 Id: c5622a No. 12988
(51.41 KB 800x534 Injuries-Due-to-Repetitive-Job-Duties-.jpg)

Dupe system doesn't seem much better because it still asks me way too much about the same images over and over again. If Hydrus knows A and B are alts and that B and C are alts, can it not also assume A and C are alts? I just spent like 20 minutes right clicking in the dupe filter on the same set of images. It went from like 60 possible dupes to 50 to 40 to 35 to 30 to 25 to 20 to 15 to 11 to 8… *screams internally* About the new right side window in the dupe filter. Sometimes it coves the images. Perhaps it could have a pin/unpin button, to make it only appear on mouse over. Or perhaps it could automatically hide if the images are large enough that it coves them. Also, shouldn't smaller file size be considered better when comparing two png images since png is lossless?
Anonymous 06/20/2019 (Thu) 16:07:29 Id: 63994e No. 12989
Is this a p2p network to share any kind of files you wish, or just some kind of super indexer that can be used to post media to random image board sites without needing a web browser?
Anonymous 06/20/2019 (Thu) 18:28:00 Id: f6b2bf No. 12990
>>12989 Neither. It's a local file-management tool where all your media is organized by 'tags', instead of filenames and folders. Whenever you drop a file into Hydrus, it clones the file into Hydrus' own folder, auto-generates a specific filename, then lets you organize things through manual (or automatic) tagging. The quote-unquote main feature of Hydrus is the database of 'public tags' you can connect to; as soon as you import a file, if someone else already imported it on their client (based on the auto-generated filename) and tagged it, Hydrus can download and automatically apply / update the 'public' tags at your end. Right now, the repository recognizes over 30 million unique files; once you get things set up, Hydrus will automatically tag (and sort) most things you can throw at it. TL;DR - Grab a download from the OP, and skim through this: https://hydrusnetwork.github.io/hydrus/help/index.html Then, when you're ready to try the public database, go here: https://cuddlebear92.github.io/Quicksync/#three
Anonymous 06/22/2019 (Sat) 12:41:25 Id: 0e971e No. 12998
>>12988 while a and b are alts and while b and c are alts this means that a and c could also be a duplicate as its not an explicit a=1 b=2 c=3 what you said is b is not 1 and c is not 2, but you never told it that c is also not 1 as far as i'm aware, the program is still falling back on user input rather then computer assumptions Personally, I am cool with an in depth dup filter and a quick and dirty one, where if you say things are alts it assumes it down the line, as this is just to quickly parse shit, and an in depth dup for when everything is to be exactly known
hydrus_dev Board Owner 06/22/2019 (Sat) 18:52:28 Id: 6b8610 No. 12999
(85.27 KB 1024x768 wew.png)

(30.22 KB 540x540 f31ee3cba8dcc52543dc52effe581c225bcb3586e76c34f235299434b20f41ed.jpg)

>>12998 >>12988 Thank you for this feedback. This is the crux of this problem–while I can transitively assume alternate relationships, I don't know if two files that are alt because of transitivity are actually duplicate. Hence, when I add a file to an alternate group, I can't remove the remaining intra-alternate group potential pairs that remain without potentially leaving dupes within that group unmerged. My first alternates update a month or so ago did remove intra-alternate potentials and it caused a bunch of problems, so I swiftly undid it. I am thinking about this problem and talked in detail with some other users about ways to reduce the number of potential pairs remaining. There aren't excellent solutions, and since the whole alternates system doesn't have good DB or UI support yet (things like adding a 'WIP' label to an alternate, or displaying a perpendicular 'wheel' of alternates in the media viewer), we don't have easy ways of editing groups of files outside of thumbnails. The potential pairs update coming in a couple of weeks will reduce the decisions you need to make here, but large alternate groups, like 150 game cgs, remain an edge-case problem. As of this 356, you can remove all the intra-alternate potential pairs by loading up the alternates as thumbnails (do thumbnail right-click->file relationships->view->x alternates, then selecting them all and right-click->file relationships->set->alts. You need to be in help->advanced mode to see the 'set' side of this menu. Applying alts on a group in this way 'fixes' the group as all alts. Ugly as it is, I think it is the best solution for large alternate groups like game cgs. I still have work to do here. The main task is to update potentials, and then I will put two or three weeks into better workflow and handling of this stuff, and writing some proper help for it all. I can see different users will have different use cases here, so I think one thing I can do is just add an option for when adding a file to an alternate group, assume it is alternate with all other members of the group (and not duplicate) with some explaining text to talk a bit about the intra-group potential pairs–if I can explain that without it sounding like a lecture on information theory. One solution to deal with this right now is to set your duplicate filter search domain to 'system:num duplicate relationships: alternates = 0' and 'both files'. That puts off these alternate decisions until later. Please see my excellent diagram I was using to talk about this earlier.
hydrus_dev Board Owner 06/22/2019 (Sat) 19:12:29 Id: 6b8610 No. 13000
>>12983 Thank you, I am sorry. I had an email from another user that shortcuts were now ok on the main gui with thumbnails and so on, but still broken on media viewer. Are shortcuts ok for you with thumbs? I will give it another go for 357.
Anonymous 06/22/2019 (Sat) 21:04:44 Id: 204c29 No. 13009
>>13000 not the guy you're responding to but another mac user. Yeah shortcuts are good in the browser, but not in the viewer. Nice trips
Anonymous 06/23/2019 (Sun) 00:43:20 Id: 69aff2 No. 13010
>>13000 Everything in the browser (main GUI with thumbnails) seems fine to me. Some of the functions keys, like F3 and F9, have been unresponsive at times, but it's always been like that. Note: As I was writing this reply to your post, I opened the Hydrus client. The client opened to a page from a previous session. Every key was working fine. F3, F9, enter, and even the navigation keys in the viewer were functional. I then opened a new search page, and the problem reappeared. The function keys no longer worked, and the navigation keys in the viewer were unresponsive. Believing that the keys were fixed due to the page being from a previous session, I closed the client and reopened it, but the keys remained unresponsive. I'm guessing that I ran into the fix you implemented in 356.
Anonymous 06/23/2019 (Sun) 05:38:21 Id: 0e971e No. 13012
>>13001 ah, for games. never used the macro pad for games, well. aside from full keyboard games like the X series. as for razor, I cant say anything as my last experience was with my fucking naga, the think broke every 1-5 months, and had it replaced 6 times in warranty and after that I gave up and went logitech. currently have a white g600 and black g600 and the only thing wrong with the white one is sometimes it multi clicks, so if shit hits the fan, I can always buy some switches and swap them out, repairing the switches seems to be a bit to much of a temporary fix for my taste. with razor, the program would slowly start to use more and more cpu power till it would just nail a core to 100%, which was unacceptable to me. however, killing the process instead of shutting it down would give me the resources back and allow me to keep the presets. if I just shut it down I would loose the presets. complete fucking garbage last time. if you look into the teensy, you can attach a joystick to it, it should be possible to make that thing work exactly like a g13 just without the whole software backing it up. as for my db, just doing a search for everything will return all the 4mill+ files, so I don't think anything is wrong there, this is a hold over from my db corrupting itself some time ago. the db got fixed, but dup detection did not register anything before that time, and because at the time dup detection was a curiosity more so than a necessity (due to not being able to tell why files were deleted) I opted to leave it as is till I was actually able to use it. right now in my dup, I have 140,000 files over 1mb, and over 10000 over 2mb, so im good for a while on shit to do and parse through, just good to know there is a thing in the pipeline that may fix this issue.
Anonymous 06/23/2019 (Sun) 07:29:25 Id: c5622a No. 13013
>>12999 Thank for the reply, sounds good. Personally I don't think I would have any issues with Hydrus doing those C is alt to A decisions. I can't think of any situation where I wouldn't want all of the images in a collection of similar images be alts to each other. On a related note, after I go through all my dupes at search level 0 and change to level 2 it asks about the same alts again. Not sure if you had fixed that or are planning to (I think the last update thread mentioned it?) but either way, it still happens.
Anonymous 06/24/2019 (Mon) 09:51:01 Id: 0e971e No. 13019
(2.34 KB 283x74 2019-06-24_04-41-32.png)

have a question hdev With the duplicate work, are you going to be able to implement something that when files have different ratios due to file sizes, see image an example, the zoom level you choose/how big the image is, tries to stay the same. Right now the two files need to have the same ratio to maintain zoom between flipping back and forth, this makes trying to find out which file is better, or if the larger file is better and not just an upscale very hard because by the time i flip back to the prior image, zoom in and find a spot, the memory of the old image is gone with exceptions for when there is small text that is night and day different I can see 2 ways to address this, 1 an arbitrary zoom that tries to match between the two, or, a tem screenshot hotkey to compare various areas… honestly I think getting an arbitrary zoom would be the easiest, but also not sure if this doesn't require a larger redo of other aspects and not just duplicates.
Anonymous 06/24/2019 (Mon) 10:47:34 Id: 0e971e No. 13020
>>13019 using duplicate filter at the moment, I had a thought of what if I just dealt with every image that was either the same size or the same ratio, but at least inside of duplicate filter there is no clear way to do this. is it possible to add with little effort a 'same ratio' or 'same image size' option to the duplicate filter?how useful this is really depends on if refered to post is going to happen, but even then, for at least a few versions this would be very helpful to have it it takes little effort to implement.
Anonymous 06/24/2019 (Mon) 21:59:12 Id: 86505b No. 13021
>>13019 A while ago I reported this same problem and suggested that it'd probably be best if it zoomed to the same relative area. If it's x:2/3 y:2/3 150% zoomed in, you get the same on the other image. It's the best guess for which areas might be the same, right? There might also be a button to lock/unlock this relative zoom so for images that have a white border or something it doesn't become too annoying.
Anonymous 06/24/2019 (Mon) 22:07:21 Id: 8c6f9d No. 13022
Apparently Gelbooru locked some content (at least loli) behind logins. Can I add a login to my subscriptions or something?
Anonymous 06/24/2019 (Mon) 22:21:53 Id: 95f9c8 No. 13023
>>13022 God dammit. Thanks for the tip off
Anonymous 06/24/2019 (Mon) 22:33:13 Id: 86505b No. 13025
>>13022 Why are all the websites so insistent on making the internet more annoying to use?
Anonymous 06/25/2019 (Tue) 08:53:15 Id: 8372de No. 13026
If we're going to use this thread to suggest ideas, then the option for a "These are alternates, but delete one of them" option would be highly apreciated, even if as a custom action. Also, a "These files are not related" option that we could select from the right click menu as before would be lovely.
Anonymous 06/25/2019 (Tue) 13:18:58 Id: 9b7275 No. 13028
>>13026 > "These are alternates, but delete one of them" option would be highly appreciated I second this. Or even potentially adding a persistent "custom action", where you could click the cog and "edit duplicate metadata merge options for custom action"?
Anonymous 06/27/2019 (Thu) 09:36:33 Id: 13dd46 No. 13041
>>13025 >>13023 >>13022 You can import cookies, better help available on the discord
hydrus_dev Board Owner 06/30/2019 (Sun) 16:21:40 Id: 9825f6 No. 13079
>>13013 Thanks. Yes, there will be a fix to seeing same alts again. Might be in this week as I do the main potential data, or in next couple of weeks if it proves complicated.
hydrus_dev Board Owner 06/30/2019 (Sun) 16:25:14 Id: 9825f6 No. 13080
>>13019 >>13020 >>13021 Yeah, I'm going to try to fudge something like this. I'm not totally sure on all the edge cases yet, but I will do something. Maybe matching the smallest x or y to be the same virtual zoom height, with some clever exception for portrait vs landscape pairs. and offsetting drag delta by the (0,0) point on both files. I haven't gone into this exact code, so I can't speak cleverly yet, but it is in my big job todo. Thank you for your feedback.
hydrus_dev Board Owner 06/30/2019 (Sun) 16:33:03 Id: 9825f6 No. 13081
>>13022 Yeah, hit network->downloaders->manage logins. There should be a gelb entry there by default. Add some credentials, set is 'active', and do a 'do login now' test. (it is worth doing a manual download as well there and then to test it worked for real–this stuff can be unreliable due to all sorts of factors). After then, if your login expires (I think gelb login lasts a year?), hydrus will try to login again automatically without you having to do anything. More info here, at the bottom: https://hydrusnetwork.github.io/hydrus/help/getting_started_downloading.html
hydrus_dev Board Owner 06/30/2019 (Sun) 16:39:19 Id: 9825f6 No. 13082
>>13026 >>13028 If you have help->advanced mode on, then clicking 'custom action' in the dupe filter should let you select 'alternate' as the dupe type, which then gives you the dupe merge options and a choice to delete A, B, or both. If you do it from the thumbnails, you should get the dupe merge options too. I agree that having some sort of 'favourites' custom merge actions will be useful for alternate processing. I think I will do it when I get around to making a proper alternate processing workflow. I am trying to shy away from significant alternate processing in this primarily duplicates data structure overhaul, since it is a black hole of work that I could get lost in for months. I took away the 'set as not related' from thumbnails because in data terms, this duplicate type is very inefficient if not applied to files that already have potential status. I have left it to the dupe filter and 'show some random pairs' quick workflow. I am still thinking about bringing it back for thumbnails, but at the moment probably won't. It really is a 'set as false positive' rather than 'dismiss any potential info' as it has been previously intepreted. I'm still thinking about it.
Index Catalog Archive Top Reply
1
Forms
Delete
Report
Quick Reply