/hydrus/ - Hydrus Network

Archive for bug reports, feature requests, and other discussion for the hydrus network.

Index Catalog Archive Bottom Refresh
Name
Options
Subject
Message

Max message length: 12000

files

Max file size: 32.00 MB

Total max file size: 50.00 MB

Max files: 5

Supported file types: GIF, JPG, PNG, WebM, OGG, and more

E-mail
Password

(used to delete files and posts)

Misc

Remember to follow the Rules

The backup domains are located at 8chan.se and 8chan.cc. TOR access can be found here, or you can access the TOR portal from the clearnet at Redchannit 3.0.

US Election Thread

8chan.moe is a hobby project with no affiliation whatsoever to the administration of any other "8chan" site, past or present.

(16.41 KB 480x360 sOTQsdwEdk8.jpg)

Version 335 hydrus_dev 01/10/2019 (Thu) 03:09:25 Id: d494fe No. 11238
https://www.youtube.com/watch?v=sOTQsdwEdk8 windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v335/Hydrus.Network.335.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v335/Hydrus.Network.335.-.Windows.-.Installer.exe os x app: https://github.com/hydrusnetwork/hydrus/releases/download/v335/Hydrus.Network.335.-.OS.X.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v335/Hydrus.Network.335.-.Linux.-.Executable.tar.gz source tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v335.tar.gz When I first made this release, Github’s file upload was not working right, and I used Mediafire instead. Github is now working and I have updated the links above. I had a great four weeks updating hydrus to python 3. The update went well, and the releases today are ready for all users, but there are special update instructions just for this week. python 3 The client and server now run completely and exclusively on python 3, updating from python 2. The new version has a variety of benefits, mostly in better unicode vs. data handling, but for hydrus it also runs a little faster, uses less idle CPU and significantly less memory, and has a little less ui-jank. I am pleased with the update. None of it was extremely difficult, but there was a lot to do, a few headaches, and some new stuff to learn. I am glad I took the four weeks. I also appreciate the users who tested the preview releases in the past couple of weeks. I have squashed a ton of little bugs, and everything normal like file downloading and browsing seems to work completely fine, but there are likely a couple of small issues still out there. If a dialog breaks for you and you get some popups about some datatype being mishandled, please send it to me and I'll get it fixed up for v336! some notes Unfortunately, for technically difficult reasons, I could not compile 'debug' versions of the executables, so these are gone this week. I will revisit this, but the original debug builds were a bit hacky and may no longer be practically possible in py3. Also, I am not certain if the database menu's 'set a password' will have kept correct password data for unusual keyboards, so if you use this function, the client will, just for this v335, forgive incorrect passwords! If this happens to you, the client will give you a popup with more information and tell you how to try to fix it. I would appreciate feedback here, if you encounter it. Due to a natural library update unrelated to py3, your hydrus sessions will likely be wiped (this also happened to some running-from-source users a little while ago), which means Hydrus will have to re-login to any sites you have logins set up for. If you have special cookies you need to save or re-import from your browser, get set up before you update! Now, because py2 and py3 are incompatible, the new version cannot be run in a folder that still has old .dll files hanging around. Please follow the following to update: update instructions for windows installer Just for this week, I have added a routine to the installer to delete the old files (but obviously saving your db directory where your database and files are stored!), so you shouldn't have to do anything. I nonetheless recommend you still make a backup before you update. Backups are great, and if you don't make one yet, this week is a great time to start. If you are a particularly long-time user and the installer fails to clear everything out, you may need to delete the old files yourself, like the extract users will have to: update instructions for windows and linux extract You will have to perform a clean install, which means deleting everything in your install folder except the db directory before extracting as normal. This is simple, but do not get it wrong. Do not delete your db directory–this is where your database and files are stored. As always, if you have a recent backup, you don't have to worry about any possible accident, so make sure you have one. update instructions for os x Due to technical limitations, the OS X release is now App only. Furthermore, this App release will no longer store the db inside itself! The default location for your db is now ~/Library/Hydrus (i.e. /Users/[you]/Library/Hydrus). This also means that the future update process will be as simple as replacing the existing Hydrus Network App in Applications, just one action. I apologise that this important change has taken so long to come out, but we got there in the end.
[Expand Post]If you are updating this week, you will need to make the Hydrus folder under your Library yourself and move your existing db there so the new Hydrus can pick up where you left off. If you use the tar.gz, you'll be moving the contents of your install_dir/db, and if you use the App, you'll want to right-click->Show Package Contents on your old py2 App and navigate to Hydrus Network/Contents/MacOS/db. You want the contents of db, not the db folder itself, so the path to your new client.db file should be ~/Library/Hydrus/client.db, for instance. If you cannot see your Library folder, check this: https://www.macworld.com/article/2057221/how-to-view-the-library-folder-in-mavericks.html If you have trouble with this, please let me know and we'll figure it out together. update instructions for running from source You'll need to make a new py3 venv and make new shortcuts. I now use pycryptodome instead of pycrypto and dropped some libraries, so I recommend you go to the 'running from source' help page again and just paste the new pip line(s) to get set up. I don't think 3.4 will work, but 3.5, 3.6, and 3.7 all seem ok. Obviously contact me if you run into trouble. I'm also interested in success stories! full list - important: - hydrus now runs completely and exclusively on python 3! - for users who are updating, the client has special install instructions for just this week: - if you are a windows or linux user who extracts to install, you will have to delete your old install's files (but keep your db folder!!!) before installing/extracting the new version so there are no 2/3 dll/so conflicts (don't delete your db folder!) - if you use the windows installer to install, this v335 installer will do the clean install for you! there is absolutely no way this could go wrong, so no need to make a backup beforehand :^) - if you are an os x user, I am now only releasing the client in the app. furthermore, the default app db location is now ~/Library/Hydrus (i.e. /Users/[you]/Library/Hydrus). you will have to move your existing db to this location to update, and thereafter you'll just be replacing the app in Applications! - if you try to boot a non-clean mixed 2/3 install, the client will try to recognise that and give an error and dump out - please check the release post for more detailed instructions here - . - semi-important: - the db password feature may be one-time broken for unusual keyboard languages, so failures this version will be forgiven with an appropriate error message explaining the situation. feedback from чики брики lads appreciated - I may have fixed the issue some linux/os x users were having launching external programs, including OS ffmpeg (it was a child process environment issue related to pyinstaller) - although I did most of my devving here on py 3.6, the client seems to run ok on 3.5. I doubt 3.4 will do it, if you mean to run from source - I moved from the old pycrypto to the new pycryptodome, so users who run from source will want to get this. I also dropped some libraries - . - misc bug fixes: - fixed the 'load one of the default options' button on manage tag import options when a set of default options is orphaned by a deleted url class - removed some popup flicker related to long error messages - fixed some parsing testing ui error handling - cleared up some bad text ctrl event handling that could sometimes cause a recursive loop - listctrls should now sort text that includes numbers in the human-friendly 2 < 10 fashion - cleaned up some bad external process calling code and improved how child process environment is set up - finally figured out the basic problem of a long-time nested dialog event handling error that could sometimes freeze the ui. I may have fixed it in one case and will keep working on this - . - boring details: - ran 2to3 to auto-convert what could be done - updated environment to python 3 - went over a whole ton of unicode encoding/decoding manually to update it to python 3 - removed all the old tobytestring/tounicode calls in favour of new python 3 handling - fixed all the file io to do bytes/str as appropriate - corrected a bunch of / vs // int/float stuff - fixed up twisted, which has some str/bytes stuff going on - fixed all the listctrls to deal with column sorting None values amongst ints/strs - fixed png export/import, which had some fun bytes/bytearray/int/str issues - updated the swf header parsing code to py3 (more str/bytes stuff) - misc float/int fixes - fixed up some http.cookies handling, which has changed in py3 - improved some ancient loopback connection code that was only really checking to see if ports were in use - cleaned up a bunch of now-invalid parameter tuples that 2to3 helpfully marked - numerous misc other refactoring and so on - updated the new network engine to now decode non-utf-8 responses correctly based on actual response header - removed some old py2 manual http multipart code - removed the old py2 'matroska or webm' parsing py, replacing it with some direct ffmpeg format inspection - replaced all % formatting with the new .format system. I will slowly move to this rather than the current endless concatenation mess - deleted some more misc old code - tightened up some spammy network error reporting - converted all /r/n to /n in my environment project, ha ha ha - the ui seems to better support rarer unicode characters like - updated some of the install/update/backup help for all this, and some misc other stuff as well - fixed misc bugs next week A lot of small stuff piled up over the holiday. I will spend a week or two trying to catch up and also planning out the client API, which will be my first big job of the year. I hope you had a good Christmas and New Year. Mine were great, and I am looking forward to 2019. Let's keep pushing and see if we can do some fun stuff. Thank you for your support!
>>11274 Those kind of programs can only salvage deleted files whose physical location on the disk hasn't been overwritten by something else. Most likely the disk where Hydrus is installed saw a lot of writing after the fact, possibly by the installer itself.
Everything works fine OS is arch linux and I'm running from source.
I'm the linux user with Filter Tag Dialog issues. Still getting them in 335. Additionally, just now hydrus crashed on me when I attempted to open the subscription manager, after having had the filter tag dialog open some. Otherwise the update works nicely. Had expected more issues after such a large refactor.
>>11290 Nope, I take it back. I'm getting a lot of: client: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.0., even outside of the filter system. Basically, anything that can open a separate window now have a chance of crashing Hydrus, even the status reports in the lower left corner. I also got following error when just clicking around in a gallery:
The program 'client' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadPixmap (invalid Pixmap parameter)'.
(Details: serial 14451 error_code 4 request_code 139 minor_code 5)
>>11291 Are you running tarball version or built it from source?
Gentoo Linux, updated via git & built from source, everything works so far
>>11267 >>11261 >>11269 >>11274 I am very sorry you have had this problem. I cannot explain it, and I hope you can recover as much as possible from your backup. It is stupid to ask, but just to be certain, is the install location correct and as you expected? It didn't change or typo for some reason? If you do a drive-level search for 'client.db', does that turn up a new unexpected location? And you never made a shortcut to launch the client with the -d switch? Also, is it possible that your install location became/stopped-being read-only? The newer version of hydrus will test if its install location is writeable, and if not, it will instead use ~/Hydrus (i.e. C:\Users\[You]\Hydrus). Is the screen at >>11267 at install_dir/db, and is there anything at ~/Hydrus?
>>11260 Thank you for this report. Is there a crash.log or any .db files in install_dir/db? Is your Win7 an updated version, or does it not pull updates? Unfortunately, I don't have debug builds available, but how about server.exe in the install dir–does this run, or give you a better error? It should be runnable from the console, and print back to it.
>>11263 Yeah, shouldn't be. As you probably know, the options for syncing those are unavailable and have been for a long time. The data record may still exist though. I would say try moving that folder somewhere and then boot the client and see if you can tag on the service(s) they were ever attached to. If it gives you an error, let me know and move the folder back, and I'll improve the error handling for next week.
>>11265 >>11271 >>11282 Thank you for this report. I will have it fixed for next week.
>>11266 I am glad it worked out. Let me know if you run into any trouble setting things up in future.
Damn, 335 is smooth. Updated from 332 and everything is smoother now.
>>11274 >>11287 If it helps, a filename to look for, either in a different location or for undelete purposes, that would definitely be associated with the old install would be 'client - 2018-12.log', and maybe a bunch of earlier ones like 'client - 2018-6.log' and so on, depending on how long you have been using the software.
>>11297 >Also, is it possible that your install location became/stopped-being read-only? The newer version of hydrus will test if its install location is writeable, and if not, it will instead use ~/Hydrus (i.e. C:\Users\[You]\Hydrus). Holy shit, my files are there! I just copied everything over and all seems ok. The logs go back to September, which is when I had a db issue, so whatever the problem was must have started then. I just checked, and the Hydrus directory is marked read-only. I don't know why it would be that way, but that may be the cause like you suggested. Thank you so much!
>>11277 This is great, thanks for letting me know. Did you have to do anything special to get wxpython to compile right? I've had the biggest trouble getting the new versions going on Ubuntu. I get 40mins of compile and then a -1 error. Seemingly no matter how many gstreamer1.0-dev-bullshit packages I install. I end up just installing the wheel from their site.
(61.22 KB 605x500 fuck.png)

>>11304 Great, that's fantastic, thanks for letting me know. I am glad we could figure this out. I am sorry for the confusion and stress here. I have a longer term plan to have better recognition of db locations. It would be nice in future if the client had a 'I think there's a db in your home dir m8' actionable label under the database menu or something.
>>11281 >>11279 Thanks, yeah, I am getting a bit of this. I have to double-check how I am logging some shit. I thought py3 would be utf-8'ing the whole time, but I think the OS default CP1252 is overriding. I think I just need to tweak the various file open calls. Should be fixed in 336.
>>11305 >Did you have to do anything special to get wxpython to compile right? The pip way of installing it takes about 5 minutes on my Ryzen 2400G and works fine. I'm using pip install –user rather than Portage since WxPython 4.x.x isn't in Portage yet (there's an issue on Gentoo's bugzilla tracking a proposed version bump ebuild, but I didn't use that). > Seemingly no matter how many gstreamer1.0-dev-bullshit packages I install. I end up just installing the wheel from their site. If it's any consolation, Ubuntu / Ubuntu Server never worked out well for me either. It broke all the damn time, and the way of creating and/or patching packages there was rage-inducing when I tried it. I guess I'm going to try to compile the ebuild anyhow to see if I can provide any information.
>>11284 Thanks m8. I am feeling better now we sorted >>11240 's problem. Really happy overall with the reports of better speed and so on. These various encoding bug reports are as I expected and should be clearable-up by 336. Keep on pushing through 2019!
>>11285 Thank you for this report. Is that on Win7? When you start the program, can you watch client.exe's CPU and HDD during that preboot phase before the splash screen appears? Is it doing a lot of work, or idling a lot? Is there any chance an anti-virus program is checking the files being loaded in memory or something? The boot loader is a little different, so maybe it is triggering something the py2 one didn't. Also, are you running off an HDD or an SSD? If a HDD, is it defragged? If you hit file->restart, is it just as slow again (maybe with high CPU), or is it much faster (suggesting the program was then cached, and this is a hard disk issue)? That screenshot suggests your public tag repo doesn't have any siblings–can you go services->review services->remote->ptr? Do the numbers look good there, or does it think, say, that it hasn't processed any content yet? Does it have any tags at all? You can take another screenshot of it if you like.
>>11286 Thank you for this report. I will have it fixed for 336.
>>11290 >>11291 Thank you for this update. I am sorry the py3 didn't do it. I have some ideas on the tag dialog and archive/delete filter that I'll be working on in this and coming weeks. As >>11292 says, are you on source? I am sorry that I forgot if you were. Some Linux users get much better stability when they aren't trying to cross their OS flavour's with my Ubuntu's .so files.
>>11308 Thanks, that's interesting. Don't worry about it too much though–the wheel here: https://extras.wxpython.org/wxPython4/extras/linux/gtk2/ubuntu-16.04/ Takes about 20s to install and works great for me. I'm just not experienced enough on the Linux side to figure out what wx wants.
>>11313 >Don't worry about it too much though No worries, doing just this is not much effort on Gentoo. Well, I compiled the wxpython 4.0.3 ebuild (with pypubsub rdepend commented out, don't need it) and of course uninstalled the version installed by pip. Took around 14 from extraction to install minutes with my current low-ish system load, works fine with Hydrus. The ebuild [it's simple, apart from the exact workings of the Gentoo helper functions] plus patches it applies (files/ subdirectory) is here: https://github.com/TheChymera/overlay/tree/master/dev-python/wxpython > I'm just not experienced enough on the Linux side to figure out what wx wants Maybe someone knows if you post the error. I don't think I'll be of much help without the ability to hands-on replicate the error [plus poor skills with Debian's / Ubuntu's package creation method].
>>11310 >Is that on Win7? Yeah >When you start the program, can you watch client.exe's CPU and HDD during that preboot phase before the splash screen appears? Yeah I can see the client.exe. >Is it doing a lot of work, or idling a lot? It seems like its running fine I guess, I mean I can see the RAM usage in the processes tab in the task manager slowly going up from a little under 10k and climbing. nothing big or anything. It gets to around 70-80k before I see the Hrdrus logo and everything is running fine The CPU usage goes up and down from 01 to 00. It just seems like its running like a normal program to me before it starts. Its just that it takes a really long time before I see anything outside the task manager to show up. >Also, are you running off an HDD or an SSD? If a HDD, is it defragged? I'm using a 3TB external HDD(Don't know why I still use the install version on Hydrus on it). I just got it a few months ago and haven't bothered to defragged it yet. >If you hit file->restart, is it just as slow again (maybe with high CPU), or is it much faster (suggesting the program was then cached, and this is a hard disk issue)? Tried it and its doing the same thing. >Is there any chance an anti-virus program is checking the files being loaded in memory or something? I wasn't really sure since I use a free version of Kaspersky and never had a problem with it slowing down any of my programs. Just tried pausing my protection and suddenly it worked. So yeah its definitely my anti-virus program slowing down hydrus, fuck. This is like the 3rd time an anti-virus program has messed with any of my programs and having to hunt for a new one. >That screenshot suggests your public tag repo doesn't have any siblings–can you go services->review services->remote->ptr? >public Wait a minute, I think I got something mixed up here. My last pic wasn't meant to be my public tags, it was suppose to be my local tags. I think I got confused, usually when I right click on a tag to take me to my siblings/parents, it'll take me to my local tags. I think I got mixed up since it was the public tab that was shown first(This would explain a lot of things for me actually). So everything is fine really, I just needed to pay attention more.
>>11298 Resolved this issue by installing the latest Windows updates, it seems that one of them is required by some binary file.
(63.39 KB 474x421 proxy.duckduckgo.com.jpg)

>>11276 Or is this just fantasy?
(14.11 MB 477x512 Unbenannt.gif)

Also thanks to the hydrus dev for yet another version. You're a real trooper, a big guy so to say.
>>11322 Great, thanks for letting me know. Maybe you can tell Kaspersky to ignore or treat the hydrus exe/install folder a little differently? Turn off 'active' protection or something so it isn't scanning every time it runs? >>11325 Great, thanks to you as well for letting me know! >>11327 UUUU Also, in this update, getting OS X PyInstaller to make a Framework-compatible Python 3 exe and app was, at times,extremely painful.
>>11328 Thank you, these are excellent. I also get problems with them. I will check them this week and see if I can improve the file parsing.
>>11334 >Great, thanks for letting me know. Maybe you can tell Kaspersky to ignore or treat the hydrus exe/install folder a little differently? Turn off 'active' protection or something so it isn't scanning every time it runs? Just letting you know I whitelisted Hydrus and it works, thanks.
>>11334 >>11328 Hey, on closer inspection, these were giving me 18KB 'webm' files that were actually some kind of CloudFlare html saying 'we think you are a bot, please solve this captcha'. Once I solved that in my browser, they worked mostly ok in the current dev version (which admittedly now has some unrelated ffmpeg fixes). The /trash/ ones are now 404, but https://i.4cdn.org/gif/1547302893499.webm is giving me a resolution problem that I think I can fix today. I think part of this problem is the result of the cloudflare-4chan setup giving the wrong http header to say the file was 'webm' when it was really an html interstitial. The other half is that hydrus has no capability to solve these sorts of captchas. I have never encountered this CF page before on a content fetch, and if it is going to be a new 4chan thing, that's a shame. Let's hope it is a temporary thing that was a misconfiguration or an accidental statistical spike on that set of files due to some mis-aimed DDoS or something. Please let me know if you encounter more of this.
>>11348 just solved the captcha… no fucking clue why it did it for that one alone, or why it didn't give me the captcha but yea, now im getting this error with bringing it into hydrus. that said, there has been some with captchas, there is an I believe open source project that can solve captchas without user interaction. it feeds the audio solution into google, and google spits out a solution and its 80-90% effective. I know with fur afinity the issue is captcha, would it be possible to implement that into the program? google apparently already knows about the issue, but the captcha team more or less said, nothing we can do.
>>11351 Thanks. I'd like to add captcha support, but diving back into the downloader code after all the work I did last year is not my top priority right now. It would be a lot of work, whether I would present login/download captchas to the user in the ui or try to solve them automatically. Current plan is to do it for the next big iteration on the downloader, whenever that will be. If 4chan or other sites start getting this CF captcha shit a lot, I'll bump it up the schedule, maybe as a hardcoded fix that recognises that CF page specifically. It is strange that both you and I got the same captcha issue, since I hadn't looked at that content/threads on my own machine/IP previously. That suggests to me the data source was the flagged problem, not the specific client, hence my DDoS-false-positive-on-that-thread suspicion. Again, we'll see if that starts happening more often.
>>11354 Thing is, I saw it fail, checked it, and was able to download it outside of client, then when you told me about the captcha and I checked It I got the captcha too. No idea what what the hell was going on with that one.


Forms
Delete
Report
Quick Reply