/hydrus/ - Version 316

/hydrus/ - Hydrus Network

Archive for bug reports, feature requests, and other discussion for the hydrus network.

Mode: Reply

Name
Options
Subject
Message	Max message length: 12000
files	Drag files here to upload or click here to select them 0.00 / 50.00 MB Max file size: 32.00 MB Total max file size: 50.00 MB Max files: 5 Supported file types: GIF, JPG, PNG, WebM, OGG, and more

E-mail
Password	(used to delete files and posts)
Misc

Remember to follow the Rules

The backup domains are located at 8chan.se and 8chan.cc. TOR access can be found here, or you can access the TOR portal from the clearnet at Redchannit 3.0.

US Election Thread

8chan.moe is a hobby project with no affiliation whatsoever to the administration of any other "8chan" site, past or present.

Version 316 hydrus_dev 08/01/2018 (Wed) 21:40:07 Id: 615bc5 No. 9533

https://www.youtube.com/watch?v=GKpKvnSjOtU windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v316/Hydrus.Network.316.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v316/Hydrus.Network.316.-.Windows.-.Installer.exe os x app: https://github.com/hydrusnetwork/hydrus/releases/download/v316/Hydrus.Network.316.-.OS.X.-.App.dmg tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v316/Hydrus.Network.316.-.OS.X.-.Extract.only.tar.gz linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v316/Hydrus.Network.316.-.Linux.-.Executable.tar.gz source tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v316.tar.gz I had an excellent week. I was able to cleanly 'do' the big gallery overhaul, resulting in many changes. This is an important release for anyone who does a lot of downloading. gallery update So, the meat of the gallery overhaul is complete! Where available, subscriptions, the gallery downloader, and the url downloader will now use the new parsing system to parse 'thumbnail pages'. The new pipeline has numerous improvements behind-the-scenes and is another important step in allowing users to create and share their own downloaders for new sites. I won't go into all the details here, but anyone who has poked around the new parsing system's advanced dialogs will now see that gallery pages are now featured. The important change for most users is that the main gallery downloader page is now a 'multi-gallery' page. It looks much like the multi-watcher for thread-watching, where the one page manages multiple queues simultaneously. Gallery download pages now list every query you enter separately and work on them simultaneously, reducing overall ui load significantly and making it much easier to pause one query without affecting others, spot queries that return 0 results, avoid big chokes and delays caused by larger queues, and manage the results you finally get as different pools. The old single-download page can no longer be created–your existing gallery pages will be updated to multi-pages on update (your existing queues will survive as generic 'updated from' entries, but everything will start paused, so you'll want to go through them, verify it all looks good, and then unpause when you are ready). And like the multi-watcher, gallery pages can have a 'highlighted query'. Double-clicking on a list entry will show its current thumbnails in the media panel and show its full details in a panel below the main list. This new panel is very similar to the old gallery page's ui and lets you see downloads as they happen and inspect the file import queue and gallery log and so on in more detail. Gallery download pages are now also downloader-agnostic. You open them just as 'gallery' from the F9 new page popup (there's also a new shortcut for them, under the 'main_gui' set), and they have a new 'gallery selector' button that lets you change the source for the next queries you enter. The gallery selector means that one gallery page can handle queries for multiple sites simultaneously! This new button will always default as 'deviant art' for now, but when I do the last stage of this downloader overhaul–the 'searcher' step–I will be able to add some reasonable 'favourite' options here. Subscriptions can also use the new gallery parser, and they have the neater 'gallery selector' as well. You shouldn't notice any difference with how they operate–they should just run as before. But after some thought, I have reduced subscriptions' maximum 'first time' and 'regular' file limits to 1000 and written some help on the edit panel to explain this change. tl;dr is: use download pages to do big one-time syncs, use subs to keep up with new stuff. If this 1000 change affects you, please check out the help and feel free to ask me about it. URL downloader pages also support parsable gallery urls. They only do the single page you add (i.e. they don't start a whole new multi-page search, but this capability will come), but they'll suck up any drag and drop for any working parser. For this week, I've written gallery parsers for danbooru, gelbooru (0.2.0 and 0.2.5), and e621 to get us started. Like with 'file pages', I will roll out more gallery parsers over the coming weeks until all default downloaders are using the new system. A lot has changed. I have done a whole bunch of testing here and I am overall happy with it, but if you encounter problems with an unusual query or site or anything else, please let me know. If you are the sort of user with 200k items waiting to download in your current session, I strongly recommend you ensure you have a good backup before you update. This multi-gallery page also makes it easier than ever to create a session with lots and lots of files, but I will reiterate that the client tends to get laggy at 50k+ items, so take it easy and make a schedule for big stuff if you can! There is a little more work to do here to support some unusual situations like tumblr, but the big hump of the downloader overhaul is now done. The 'searcher' change will be much simpler (it is basically just converting samus_aran -> http://safebooru.org/index.php?page=post&s=list&tags=samus_aran and some ui to manage it) and then it'll just be ui cleanup and figuring out an easy-mode share object, and redoing some help, and then we can talk about what we want out of a login manager (including whether it should be put off for something else). multi-watcher The multi-watcher also has a new 'highlight' panel just below its list. In a similar way, it has similar ui to the old single-page watcher and lets you inspect the file list and gallery (thread) log. Single watcher pages are removed this week, so all existing single watchers will be updated to multi-watchers. A variety of related single-watcher 'page naming' options and so on are also removed. Rather than be redundantly complicated, hydrus now just has 'url downloader', 'watcher', 'gallery', and 'simple downloader'. full list - gallery: - gallery url classes can now be linked to parsers! - if parsers are linked, gallery pagewalk can now work on the new parsing system. gallery import pipeline has been significantly updated to reflect this - gallery import objects are now 'multiple' gallery imports, much like the multi-watcher, with each separate query having its own entry in a list (they also run in parallel!) - the multi-gallery list will show file/gallery pause status in slender columns, and will show a 'stop' character when gallery parsing is done - wrote a 'gallery selector' button and added it to the new multi-gallery page, so you can spawn queries for ~different sites~ on the same import page! it always defaults to 'deviant art' for now, but when the next 'searcher' overhaul step is done, this will be customisable - the new page selector and related 'pages' menu is now simpler–with the new selector, you just select 'gallery' - added 'new_gallery_downloader_page' shortcut action to the 'main gui' set to allow quick opening of this new page type

[Expand Post]

- wrote a 'gallery import panel', which reviews a single gallery import stream, and added it to the multi-gallery page to show the current highlighted query - as all gallery imports now run in parallel and work on the new system, the now almost-useless 'cancel' gallery pagewalk button is now removed - with the wider availability of the new gallery log for file count and error reports, shifted around and smoothed out some gallery status text presentation - improved the auto url_class->parser linking 'try to fill in gaps' logic to work with gallery urls (this was surprisingly complicated) - fixed a misc stupid waste of time in auto url_class->parser linking - many misc updates to gallery pipeline - . - subscriptions: - wrote a new gallery pagewalk pipeline for subscriptions, which still does oldest-to-newest url addition and obeys file limits and so on - numerous subscription pipeline and error handling tweaks and improvements, particularly in regards to the new code - subscriptions now have max initial and periodic file limits of 1000. existing subs with >1000 or infinite will be cut to 1000. there is a help button on the edit sub panel to explain why you should do large syncs with the manual downloader and not subs - the ugly and dangerous-if-you-scroll-in-the-wrong-place gallery selector mismash control in the edit subscription panel is now replaced with the new gallery selector button - fixed an issue where the edit subs panels could sometimes say '48 years ago' (i.e. displaying a literal time delta since 0, 1970) on initial timestamps - juggled some 'periodic limit' reporting logic to skip an unusual false positive that affect hentai foundry subs for now and more in future - . - urls: - the url downloader now accepts gallery urls and will receive drag-and-dropped gallery urls. at the moment, it only parses the one page (i.e. it doesn't start a new 'searching' pagewalk) and sends the parsed links to its file queue - . - watcher: - finished 'watcher panel', which reviews a watcher, and added it to the multi-watcher page to show the current highlighted watcher - the single watcher page is completely removed–it is only the multiple watcher now. all singles will be converted to multiples on update - some single-watcher options (like watchers naming their own page tabs and the [404]-style page name prefixes) are removed - multiple watcher panel now lists file/checking pause status and has separate buttons to control these paused statuses - fixed some misc watcher highlight code–highlighted watchers should correctly publish to the page from the start of session load now - improved some 'repage' logic in how highlighted threads get removed - . - misc: - discovered a scroll-setup parameter that stops janky scroll-to-click-focus behaviour on all the new scrolling panels, thank the LORD - improved some 'can't parse' error handling for post url parsing - reworked how all importers present their network jobs to the ui, including fast response if the switch happens during a job - in prep for searcher switchover where all downloader sources will be harmonised into one system, booru identifiers now present in several ui locations as 'name', not 'booru: name' - updated the danbooru parser to deal with the new 'next page' markup they use - wrote a gelbooru gallery parser that works with 0.2.0 and 0.2.5 gelb sites–an ancillary issue where gelb-related downloaders could sometimes not accurately figure out the magic '42' next-page offset is hence now fixed - wrote an e621 gallery page parser - these sites hence now support single-page gallery drag-and-drop - added url classes for artstation gallery url and its api counterpart, but didn't go further yet–we aren't quite there with api pagewalking just yet - updated deviant art gallery url classes - added an e621 'search initialisation' gallery url class to improve some future drag-and-drop stuff - url normalisation no longer cuts off 'www.'-style prefixes - url comparison is more careful to test 'www.'-style prefixes, so a file import cache should recognise that 'http://www.blah.com/blah' is the same as 'https://blah.com/blah' - did a bunch of refactoring to further split up the bloated ClientImporting.py - fixed some misc downloader layout that may have been hiding some texts previously - some multi-watcher and multi-gallery events like add/pause query should be a bit snappier - in the parsing ui, url and title priorities are now 50 by default - prepped a little subscription unit test code for when searcher object is done - misc downloader layout improvements - misc listctrl refactoring next week A lot of stuff I overconfidently said "I will check this this week" about has been left in the dust of this unexpectedly large job. I want to spend a week focusing on this smaller stuff and catch up back to regular schedule.

Anonymous 08/02/2018 (Thu) 05:09:51 Id: 98be01 No. 9536

Ok, 600k image person here, just updated, with this update I should be able to knock around 200k images off the opened image list. Good I tried doing that. A new gallery search, will highlight and unhighlight just fine, works great The new shit added to the lineup… again, good job, it helps a lot, though I have a 4k monitor and even if I make the preview image portion as small as possible you still have to scroll. I think making this portion of the window tabbed is going to need to happen rather than would be nice if it happened, but its manageable. as I don't need that stuff much of the time. however already opened galleries will not clear highlight and highlight, the moment you clear them its gone for now. all the files are there, it remembers everything, you can look at the long, but the files won't display. I'm going to restart the program now and see if its a quirk of a first time open or if the functionality is currently not working.

Anonymous 08/02/2018 (Thu) 05:42:41 Id: 98be01 No. 9537

>>9536 Nope, still not highlighting. Im going to leave this as is for now, neither of the searches are things I will miss if this doesn't get fixed/working, but killing off 200k worth of images on a 'maybe this will work in the next version' isn't something i'm willing to do. that said, are you going to move simple downloader over to a multidownloader? Not so much anymore, but I use to get a fuck ton of tabs that went through that, and it would be nice if I could consolidate that too.

Anonymous 08/02/2018 (Thu) 07:09:09 Id: 98be01 No. 9540

So looking a bit more, I noticed that other galleries had 'open selected import files to new page' on right clicking them on the file import, so thinking this would be an acceptable work around, I looked at the files, they have nothing there saying they can do this. if I have a small file import from a gallery pre update im going to try one more thing, but I doubt it exists…

Anonymous 08/02/2018 (Thu) 11:06:33 Id: 98be01 No. 9541

Ok noticed another thing, downloaders seem to be going fairly slow. 1 image every 30 or so seconds, and doing everything I can think of to remove any possible limit does not work outside of telling it no, override everything every 5 seconds. Just added a few threads to download, yea, its 1 file every 30 seconds, but its limited to each watcher, so 20 watchers can get 1 file every 30 seconds, give or take if the file is already in the archive, as those ones seem to make the downloader immediately go for the next one.

Anonymous 08/02/2018 (Thu) 11:07:33 Id: 41d87e No. 9542

So I'm trying to make a blog parser when I encountered a problem. The website I'm trying to scrape does not have a next page button on the gallery page, only a bunch of page links. Is there any way to only get the link below the <strong> tag? The blog is https://blog.naver.com/PostList.nhn?blogId=doremi4704&categoryNo=8 I've tried to get one of the links and then replace the page number. But I can't find the page number of the next page, only the current page number.

Anonymous 08/02/2018 (Thu) 12:23:19 Id: 98be01 No. 9543

Ok, so I missed downloading a thread before it 404ed, something from a few days ago, and I went through the simple downloader, it immediately got to work and went fast (well fast ish for 1 download at a time) looking at it closer it appears that gallery multi and simple downloader both go relatively fast, using threadwatcher on 8chan seems to also go fast, though gallery watcher for 8chan seems to break with very low traffic 'this thread has been here for year' threads, it seems like simple downloader 8chan setting managed to get everything. Ok after much dicking around, I set a custom rule for 4cdn.org 100 requests a second and 2gb every 10 seconds. this appears to have solved the issue, however why was there an issue in the first place? I don't understand what was the hang up at all, all my settings globally have impossible to meet requirements unless hydrus starts doing multiple files at once per watcher. problem seems solved, but I have no idea why there was a problem in the first place.

Anonymous 08/02/2018 (Thu) 13:08:52 Id: 98be01 No. 9544

One last thing before I forget, file import options. I said last time that it wasn't working, however this version it does… kind of. The first file import the threadwatcer does not give a damn about this. the second one below that it does care about and you can set it, close then re open for the files. after some more dicking around mid sentence, I hit apply options and it worked without opening the watcher. This is going to be personal preference, I have use for being able to see everything, and then only seeing new, opening options and changing it then applying it, and then reverting it back is a pain in the ass. If its possible to have a 'highlight new' option, that would be much appreciated.

Anonymous 08/02/2018 (Thu) 15:20:46 Id: 402ca0 No. 9545

Add vim shortcuts and I will suck you're dick.

Anonymous 08/02/2018 (Thu) 15:28:40 Id: 614fca No. 9546

>>9533 Pretty cool update. The multi gallery downloader thing makes me wish I could also use it with filesystem paths, it's in most ways more elegant than waiting for the popup dialog to populate the specified folder / files and only then importing.

Anonymous 08/02/2018 (Thu) 15:42:31 Id: 614fca No. 9547

Editing my subscriptions [which by now has dozens of entries and encompass some hundred thousand downloaded files] seems quite liable to get Hydrus crashed/ OOM killed. Even if it does not for once [mainly works if no pages are open and nothing else is going on], it takes tens of seconds to load this dialog and most sub-dialogs, or to close them with apply. And it stops all subscriptions from downloading meanwhile, not only those being edited.

Anonymous 08/02/2018 (Thu) 17:07:13 Id: 98be01 No. 9548

found something concerning 21 days, 12 hours ago it looks like the program did not stop checking 404 threads. found one from 29 days ago one month 12 days pretty much everything from /b/ was checking every 3 minutes. various other boards watchers also did not stop checking but most of them fell to once a day. There is some amount of I want the program to make damn sure the thread 404ed and its not a website hang, but never stopping? It may be a setting I put in, i'm not sure, but shouldn't once the program confirms it 404ed or is dead that it stops checking? also, If I highlight a thread, it starts to check it again.

Anonymous 08/02/2018 (Thu) 19:51:58 Id: e4027b No. 9549

I did: new page->download->gallery->Gelbooru "afrobull" The result: Files get downloaded but no tags applied. I made sure that default tag import options have "get tags" marked. Using the same with danbooru or sankaku works. I have this problem since 314 but am scared to downgrade in fear of losing my db.

Anonymous 08/02/2018 (Thu) 19:56:36 Id: e4027b No. 9550

>>9549 I forgot to mention, I nuked the db folder in hopes that it was my config but it didn't help.

Anonymous 08/03/2018 (Fri) 01:19:02 Id: 5aec71 No. 9552

The 1000 file limit wrecks my workflow. On one of my hydrus installs I'll come back after a month and have it catch up on what I've missed and add it to the queue to pull in over a few days.

Anonymous 08/03/2018 (Fri) 11:58:55 Id: e60abf No. 9554

Can I combine multiple old download pages? I have some unsorted threads and I'd like to keep them in one page. Also, when I went to a converted download page with a lot of deleted files, this happened.

Anonymous 08/03/2018 (Fri) 16:28:40 Id: e6d908 No. 9557

Arch user here, I just wanted to thank you for this update and the mention that we should use the download page for initial sync. Can you also make the subscriptions work in pages? I think the popups were the reason I got graphical glitches and system freezes in the past. I don't know what wxphoenix is doing with popups but they seem to spam X11 memory/vram until it can't deal with it anymore. I'm kind of on the low end with this laptop and vram is shared with system memory so it could also be shitty hp engineering again. Now that I don't have subscriptions running for minutes, it doesn't takes my system down anymore, thanks dude.

Anonymous 08/03/2018 (Fri) 17:32:47 Id: 98be01 No. 9558

>>9554 IfI had to take a guess, its because it read the download list and tried to display the images but the images were gone. I don't mind this personally, as its better than something not working in a several thousand selected file move

hydrus_dev Board Owner 08/04/2018 (Sat) 16:21:42 Id: 615bc5 No. 9566

>>9537 >>9536 >>9540 >>9541 >>9543 >>9544 >>9548 I am not sure what is going on with your files not showing with highlight, but in the conversion, some legacy stuff may have been lost. The highlight consults the file import cache directly every time it loads (and it obeys file import options 'publish' rules), whereas the old system just remembered whatever thumbnails were in the panel (so you could do something crazy like drag and drop thumbnails in there from another page and it would remember). So if you maybe culled the files from your file import list, they may now be forgotten, or if your file import options say not to 'publish' archived files or something, then those files may not appear now on subsequent 'highlight' calls. For your 30s delay, do the affected galleries say 'overriding bandwidth in 30s'? That's a common bandwidth delay that comes into play in certain situations when the client is near the bandwidth limit for a domain. If you open up network->data->bandwidth, do the relevant domains look blocked? Or, on that same panel, if you check the 'default' button's options for a 'downloader instance' entry, does that have anything particularly restrictive. There are a bunch of different bandwidth rules to stop shit running away if you run like 100 queues at once or accidentally set up a queue 100k long, so maybe these are firing strangely or maybe they are working fine and stopping your client going completely bonkers. Ah, reading your next post, I see it was a bandwidth rule issue. I do not recommend 100 rqs a second. The default limit is 1 rq per sec per domain, which is a nice bottleneck that slows down big stuff and stop accidents that might hang the ui as well. If you had 30 threads running, they'd likely slow down to each, on average, getting one file per 30s. If you start banging servers ten times a second for like two hours straight, you may run into automatic cloudflare-or-whatever filters and get yourself IP banned. If you do a lot of downloading, I recommend letting the client download slower over a longer period. Yeah, I think I'll add an option for something like 'if I enter a new query/url and none is highlighted, highlight it please'. This 404 issue does not look good. Thank you for noticing and reporting it. Obviously it should stop checking after 404–and as far as I understand, is usually good and does stop–so I will make sure to check the logic here. Again, I recommend not having high bandwidth rules, as if something goes wrong and the client does do something stupid like checking the same url over and over (or getting lost in a subscription and checking a whole bunch of needless gallery pages), you can get very large numbers like this. Since you are a heavy user, I do not recommend checking every 3mins as a flat rule–instead let the checker check in a range of say 3mins to 24 hours, then if a thread falls off, the checker will only be hitting it once a day until it dies, and if shit goes wrong like this, you'll only tick up a few bad hits rather than thousands. The 'default' buttons in the checker options offer some reasonable rules that will work for most situations. This spammy requesting is likely another cause for the slowdown you saw with more reasonable rules. I'll also be adding options to let you customise the characters there–you are supposed to see pause/stop characters in unicode in those columns, but I see they aren't rendering right for you.

hydrus_dev Board Owner 08/04/2018 (Sat) 16:32:26 Id: 615bc5 No. 9567

>>9542 Yeah, I am encountering a similar problem with json APIs, which are requested with like 'offset=50' but then don't give that position as something parsable in the reply, and trying to write some pipeline for 'count how many posts were added and then add that to this number in the url we started with' is just way too complicated to even work right. In your case, even if you can grab that strong, I don't have a 'get next sibling' in the html parser yet. Even if I did, it feels a little dangerous, I am not sure. So, I think the better solution I am going to go for is adding a 'position index' definition in the gallery url class. So you'll define 'the third path component' or 'the "page" parameter', or in this case 'the "currentPage" param' as the index, and then say 'it increases by 1/24/50 every page', and then if the parser fails to find a link, the system will supply what the url class guesses. This is basically how the old system did it, but I'll just use it as a fallback. I have to think about this a bit more (I may want to let the parser raise a veto to say 'I am certain there are no more pages, don't fallback'), but I think it is what I will do.

hydrus_dev Board Owner 08/04/2018 (Sat) 16:32:52 Id: 615bc5 No. 9568

>>9545 Hey, I do not know much about vim–can you say what sort of shortcuts you would like?

hydrus_dev Board Owner 08/04/2018 (Sat) 16:38:36 Id: 615bc5 No. 9569

>>9546 >>9547 Yeah, the original file import dialog is one of the oldest things I wrote, and now I have the file import cache, I wonder if I should pass the mime parsing step to the import, where it is redundantly re-done anyway. This would give us better error preservation for the unparsable files and all sorts. For your subs, I recommend trying the 'compact' button on the dialog. This should cull old files from your subs and make the whole thing snappier. If your subs are all separate (i.e. 1 query per sub), you may want to 'merge' them as well. Having 10 subs each with 100 queries is easier to handle than 1000 each with 1. Yeah, the 'pause subs while dialog open' is a necessary evil–it is way to complicated to try to update the dialog with new data while the subs are working and handle the consequences of editing via the gui and then hitting ok on a running sub, so I have to atomise it to doing one task at once. Let me know if the 'compact' does it. I expect to make it happen automatically in future.

hydrus_dev Board Owner 08/04/2018 (Sat) 16:46:40 Id: 615bc5 No. 9570

>>9549 >>9550 Thank you for this report. Can you please check each of these locations: Under network->downloaders->manage default tag import options: - The 'default for file posts' - any default for 'gelbooru file page' A gelb default would override the basic default, but it will only exist if you created it, so you can ignore it if none exists. If it has 'get tags' checked, does the button aside say 'all tags'? If you click the 'cog' icon, are tags being applied to all files, and not 'only tags that already exist'? If you 'highlight' the afrobull query and scroll down to look at the highlight panel, do the tag import options there look good (i.e. either set as 'use default' or have the same explicit 'get all tags' settings)? If it is a help, the tag import options set on the multiple watcher only apply to new queries, so if you set the good tag options after you added 'afrobull', it wouldn't have applied. Clicking the query and then 'set options to queries' will do this, however. Lastly, if none of that works, please hit network->downloader definitions->manage parsers, find the gelbooru 0.2.5 file page parser, open it, and then put in an example url to fetch from and try the 'test parse' button–do tags come up?

hydrus_dev Board Owner 08/04/2018 (Sat) 16:49:36 Id: 615bc5 No. 9571

>>9552 Thank you for this report. What kind of limit would work for you, and what kinds of tags are these subs for? Is it stuff like 'blue_eyes' and 'smiling', some generic tags, or is there a different combination of tags I am not thinking of that produces more than 1000 files per month or so? What would be the 'worst' of these firehose tags, in your opinion, and how many does it produce per day? And would it be reasonable to run this client faster, like once a week, or that not possible for technical or whatever reasons?

hydrus_dev Board Owner 08/04/2018 (Sat) 16:54:05 Id: 615bc5 No. 9572

>>9554 >>9558 Thanks, yeah, I'd like to do both of these things. There's a bit of code already in to do shuffling watchers and queries between the pages, so I'll just have to figure out a drag and drop or cut/paste solution. But it might be a too-big pain in the neck to do drag and drop quickly, so I might end up having a 'make this page the kind of queries, have it suck up all the queries from other pages' in the meantime. Would a 'consolidate everything here' work for you, or would you like to have multiple pages running at once and shuffle between them? For the dead files, yeah, the 'publish to page' stuff doesn't filter them out yet. It is the same for the multi-watcher. I am going to revisit the way that queue loads this week, mostly to make it faster, but I'll see if I can ditch these missing files as well.

hydrus_dev Board Owner 08/04/2018 (Sat) 17:00:32 Id: 615bc5 No. 9573

>>9557 Thank you for this information. Yeah, I've had trouble getting floating stuff to work right in several flavours of non-Windows. Having subs run in pages is interesting. I don't think I have time to write a whole new page for them and figure out all the new back-end that would be needed, but if the popups have been a pain, maybe I can figure out a debug-tier test option to hide sub popups while they run. You can have them 'publish' their files to a page now rather than a popup button, so if you are always letting them run in the background anyway (I certainly am), maybe the popups aren't really needed for advanced users unless something goes wrong. Btw: if your existing subs are pretty heavy with thousands and thousands of files, try selecting them and hitting the 'compact' button in the edit subs dialog. This will cull old files that aren't needed for file velocity calculations any more and make the whole thing snappier. I am going to give the outdated subscription help file a pass in the next week or two. It will more explicitly lay out how I mean for the whole system to work now with initial syncs and then subs for catchup. Did you ever get pointed to that, and if not, where would be a good time for me to put a big arrow to it for you? The edit subs dialog has a little help in the top corner, but would it be better if I say tracked the first time a user opens that dialog and pop up a thing saying 'yo, check the help here m8'? Or do you have any other thoughts on this?

Anonymous 08/04/2018 (Sat) 19:02:38 Id: 55c85b No. 9575

>>9570 Thanks, i did everything you asked and every tag option looks fine and is the same as in the parsers that download tags. I thus did the parser test and no tags showed up in the parsed results while the html preview did contain them. Here is the data: >network->downloader definitions->manage parsers, find the gelbooru 0.2.5 file page parser, open it, and then put in an example url to fetch from and try the 'test parse' button I added https://gelbooru.com/index.php?id=4341231&page=post&s=view to example urls and as test url, I fetched it via "fetch test data from url" and "test parse" "fetch test data from url":

PREVIEW:
<!DOCTYPE html><html>
	<head>
		<meta charset="UTF-8">
		<title>1boy 1girl arm arm at side arm behind back back bare arms bare back bare legs bare shoulders barefoot blonde hair blue eyes blush bracelet breasts cliff closed mouth commentary covering covering ass dress dress lift english commentary feet female floating hair from behind full body green hat hat hiding highres jewelry lasterk legs link long hair looking at viewer looking back medium breasts neck necklace nintendo outdoors pointy ears princess zelda rock semi-transparent shadow sideboob sky standing strapless strapless dress sundress the legend of zelda the legend of zelda: breath of the wild white dress wind wind lift - Image View -  | 4341231 | Gelbooru</title><link rel="stylesheet" type="text/css" media="screen" href="responsive.css?14" title="default" />
		<link rel="stylesheet" type="text/css" media="screen" href="css/jquery-ui.css" title="default" />
		<link rel="stylesheet" type="text/css" href="css/jquery-ui.icon-font.min.css"/> 
		<

"test parse":

*** 1 RESULTS BEGIN ***

downloadable/pursuable url (priority 75): https://gelbooru.com/images/7d/67/7d67a721357cc49258551834444582ed.jpg
downloadable/pursuable url (priority 50): https://simg3.gelbooru.com//images/7d/67/7d67a721357cc49258551834444582ed.jpg
md5 hash: 7d67a721357cc49258551834444582ed
source time: 2018/07/28 15:00:31
associable/source url (priority 0): http://www.pixiv.net/member_illust.php?mode=medium&illust_id=69908210

*** RESULTS END ***

Do you think it serves different pages depending on region/timezone? It also often stopped downloading after 42 images and I'm not sure how to deal with that. Thanks for your patience

Anonymous 08/04/2018 (Sat) 19:12:20 Id: 55c85b No. 9576

>>9573 I started a new db with only one danbooru sub with 12 queries after downloading each of them. The freezes ONLY happen when I switch back to the hydrus window, sometimes for a few seconds and sometimes for a long time. It doesn't matter if it is finished syncing or not it seems that it waits patiently for me to get back to the window to give me a nice dose of freezing. I started hydrus while beginning writing this and it froze for a few seconds when I took a quick peek at it while it was syncing. Syncing is now finished, I can't see the results of the sync due to the popup glitching out but I think it downloaded 10 pics. It doesn't have any freeze problems for now. In essence: Syncing while window in background or if it is active/foreground: No freezing Switching active state of the hydrus window when it synced while in background: Freezes everything, including mouse and desktop for whatever amount it deemed necessary. Additional note: There is no system ram shortage on my end and the taskviewer also doesn't show any unusual spikes. I think it must be vram related due to the graphical glitches that happen. It mosaic-ed my whole desktop one time and I had to completely kill X11 to get back on track (the kernel terminal was fine). I think a quick and dirty way to try and fix this for shitty laptops like mine is to just not show popups at all during syncing, perhaps replacing it with a busy indicator, and only output the result when nothing is scheduled anymore. Thanks for your efforts, dude. I really appreciate the work you put into this program.

Anonymous 08/04/2018 (Sat) 20:20:15 Id: 98be01 No. 9577

>>9566 as far as bandwidth goes, I have had everything reset every minute for a while, the first time I ran into bandwidth not letting me download at all for I forget when that was, every single page should have been under something like a 1000 requests every second limit and 2gb every minute, but somehow on the update, 4chan went to every 30 seconds it would download a new file. as for override, that worked but was slower then I wished. the problem of solved when I made a custom rule for that domain to do 100 requests a second and 100gb every 5 seconds. I have no idea what was going on with 4chan, as it was the only domain to have that issue. as it comes to downloading form 4chan, I use to do FAR worse to its servers to make sure I got the images in cache. most of what I download, at least after the initial 'lets add every thread i'm interested in' event where hydrus could not keep up to save its life and I broke downloading completely, its maintenance imports, 10 or so threads per board im interested in, relatively low file counts, and just let the watcher do its job. Now for the 404, I have the checker going every 3 minutes for /b/ because threads at peak times only last 7 minutes, and the default program controlled logic decides 'this thread is going fast, lets check again in 2 hours' for the most part, boards that are not /b/ or /trash/ work well with the hydrus controlled timing, and I have every other one on a check at least once a day rule set, There may be a chance its the sheer volume of watchers I have going, 7125 including dead ones, that caused this behaviour. as for the slow down, I doubt it, I solved/made a work around for the slow down issue I want to say 3 hours before I discovered the never stop checking thing, It may be possible that when I updated the client, that's when the check all the fuckin threads behavure started, and the program filled in the blanks making it look like it was an issue for a longer time then it was, and once the program started to do that, it preformed slow as hell, this would explain why in 315 it was going fine and in 316 it wasn't. ah, so that's what those are, I just assumed it was a tick box like thing ————– No just to be clear, the image below is the samus auron safebooru that I did as a test, the right is god knows what I used as a search in gel The gel isn't highlighting, but it has all the files known, there may be some amount of bullshit I can do to recover the files in a to do list, in fact I know there is shit I can do and it wouldn't even be all that difficult. however ways I know for a fact I can recover them dont allow me to clear highlight and re highlight. I just looked at my tabs a bit, I apparently had quite a large amount of small galleries, and some of them fucking highlight and clear highlight… correct that all of them highlight correctly. Im both happy and incredibly pissed as these smaller ones would be very easy test cases for any method I can think of to recover. If there is something simple on your end that can get them to highlight, i'm willing to wait, however if your suggestion is to just treat them not like gallery downloads but like normal files, it makes it a bit easier to say fuck it and dick around, either manually by opening 5 images and seeing what the common ground is and re doing the download from the gallery, this way will work, however its time consuming, or by other methods that may let me do this rapidly without to much dicking around. ————–

Anonymous 08/05/2018 (Sun) 00:53:33 Id: acf26b No. 9579

>>9568 I would like to be able to navigate with J K L H instead of the arrow keys. Pretty much that. H Left J Down K Up L Right I would be nice if i/I (Insert) would open the tags windows. D (Shift+D) to Delete. O to open a new page.. o to open externally. / to quickly focus the Search bar. u Undo. Ctrl+r to Re-do. ZZ and ZQ to exit

Anonymous 08/05/2018 (Sun) 00:55:46 Id: acf26b No. 9580

>>9579 Also yy to Copy the file to the clipboard. https://vim.rtorr.com/

Anonymous 08/05/2018 (Sun) 01:08:46 Id: acf26b No. 9581

>>9579 O open externally o open a new page Sorry.

Anonymous 08/05/2018 (Sun) 08:57:32 Id: 0e8ff8 No. 9585

>>9579 Could you write a whole specification for that?

Anonymous 08/06/2018 (Mon) 00:17:31 Id: 8a5e55 No. 9586

>>9585 Let me try, Sorry I didn't explain myself very good. Vi is a very well know text editor that uses simple keyboard shortcuts, I use them in almost every program I use. I think it would be nice if you could navigate pictures in the main GUI with vim movement shortcuts: h Left j Down k Up l Right That's it I would be happy with just that,but if you look more in-depth , there is a lot of vim shortcuts that you apply to Hydrus. Fore example: i to open the manage file tags. yy (yank) to copy a file to the clipboard. D (Mayus D/Shift + D) to Delete a file. O and o to open a file externally and a new page. / To quickly focus the search bar. u Undo an action. Ctrl+R re do an action. ZZ and ZQ to exit. They mimic the function they have in vim, and it would have to make Hydrus more easy to use with just the keyboard. I hope this is clear enough. Anyway this is just a recommendation, sorry for being too insistent.

Anonymous 08/06/2018 (Mon) 13:14:42 Id: 98be01 No. 9587

I have a thought granted with highlight coming to multi watcher and gallery the use is now more limited than before but it may be something worth thinking about. would it be possible to have a low resource mode? let me put it this way, let's put it this way, my current session is eating 13.5 gb, but while i'm not actively doing things with files, I really only need the function of the watchers I could branch off the program put all this shit with images in a page of pages, save it and thread watchers in another, save it, and just reopen the watchers… now that I say that, I may do that till im able to properly cull images, but putting that aside, would it be possible to have a loading mode that uses the current session, but just doesn't load images… it keeps all the shit necessary to load them remembered, it will append any new tabs or things added, and when you want to get into images, you can load the full image set? granted a full move to highlight on all modes would kill any need or use this has, but I mention it because it seems like it could be an easy to implement feature.

Anonymous 08/07/2018 (Tue) 01:56:29 Id: f874ca No. 9589

>>9586 gg To go top would be nice.

hydrus_dev Board Owner 08/11/2018 (Sat) 16:54:55 Id: 615bc5 No. 9636

(3.04 KB 512x85 gelbooru 0.2.5 file page parser.png)

>>9575 Thanks. If I do the exact same URL, I get the tags ok. If you are on linux (and maybe running from source?), it might be that you have odd parsing libraries and the parsing tree is getting cut off. Please check help->about–what does it say about 'html5lib' and 'lxml'? Are they both present? If html5lib is missing, you should be able to add it with: pip install html5lib From the terminal. Then reboot the client and try again. It could also be that you somehow have a wrong/bad version of the gelbooru parser. You might like to try importing the attached png through the same 'manage parsers' dialog and testing again. If it works correctly, then delete the old one and fix up the gelbooru 0.2.5 link under manage url class links in the same networking submenu.

hydrus_dev Board Owner 08/11/2018 (Sat) 16:57:02 Id: 615bc5 No. 9637

>>9576 Thanks. Yeah, I think I am just going to write a simple alternative for popups for non-windows people. Floating windows are just the biggest pain to get working multi-platform. I apologise for the inconvenience.

hydrus_dev Board Owner 08/11/2018 (Sat) 17:10:59 Id: 615bc5 No. 9638

>>9579 >>9580 >>9581 >>9585 >>9586 >>9589 Thank you for this information. The new shortcuts system I am slowly migrating over to does not support double-shortcuts like ZZ, but I hope to eventually move pretty much all events over to it, so remapping arrow keys for movement should be doable in the future. I am drowning in work atm, but as the downloader overhaul comes to a close I should be able to revisit some of these longer term rewrite/cleanup jobs. Let me know how it works for you as I roll it out.

hydrus_dev Board Owner 08/11/2018 (Sat) 17:14:34 Id: 615bc5 No. 9639

>>9587 I would like to rewrite session loading to make is less resource intensive and do delayed loading like you suggest here. But yeah also moving to multi- pages should massively reduce the memory burden for downloaders that would otherwise have like 3000 thumbnails attached. Now the db just has to load the file import status list, which is basically just a URL and a couple of numbers per item, rather than all the tags and so on that go with it. I have some more ideas here to relieve gigantico-clients, but it will have to wait for the next pass on the downloader overhaul.

Anonymous 10/01/2018 (Mon) 04:27:39 Id: 92090e No. 10150

>>9579 >>9580 >>9581 >>9586 >>9589 Let me get this straight. You people LIKE vim? Hell, I use it every so often, but only if I need to make a super-minor change and don't feel like waiting for a graphical text editor to load. Or, more likely, it can't load because I FUBARed my system and need to fix it. Hell, I've always thought that all of those people talking about and comparing text-only editors were either old farts who worked on mainframes in the 1980s or people just joking around. I'm not arguing against improvements, mind you. I'm not even arguing about priorities. I'm just confused as all hell. >>9587 Low-resource mode is a great idea. With the old download system, I've long wished to be able to somehow page out only part of the memory Hydrus was using (yes I know paging doesn't work like that, but let me dream goddammit). At one point I actually bought more RAM so Hydrus worked better, especially when also doing something with my computer that's not Hydrus.

Anonymous 10/01/2018 (Mon) 06:23:40 Id: 92090e No. 10152

>>10150 Shit, I thought this was the new thread.

Anonymous 10/02/2018 (Tue) 05:17:01 Id: 3d1991 No. 10159

>>10150 >I'm just confused as all hell. Why? There is a lot of people who uses vim as their primary text editor, people really really like it, there is even a game to learn Vim, the community is kinda like Emacs (but I think Emacs users are more dedicated to their text editor), there is a lot of programs that use vim or emacs shortcuts, one example is the Terminal, by the default they use the Emacs shortcuts but you can set it to use Vi shortcuts, I prefer to only use the keyboard so most of my programs use the vi keys (I use them even in my web browser) , I try to do the same with hydrus but it ins't completely possible at least yet.

Index Catalog Archive Top Reply

Manage Board Moderate Board Moderate Thread

Forms

Delete

Password Unlink (Removes file reference from posts) Delete (Removes file from the server)

Report

Reason Category Global

No Cookies?

Quick Reply


Sage Bypass Check