Rats! I was getting excited reading the commits.

>finally got upload working stable in new install of chrome >upload proceeds at buttery smooth 85KB/s >upload finishes, get this >it was a .tar.gz file I think Github may be fucked. I will try again tomorrow. Worst case, I'll throw them up on mediafire.

>>11236 Changed my mind. I want to hammer this shit out tonight. Uploading to mediafire now, release in 10-20mins.

>>11223 Nice.

Cool! Can't wait to see that commit's diffs

>>11226 here found a small artist I didn't import yet this screenshot was taken before any imports happened.

>>11227 >>11226 Thanks. I will make a job to check this code and have it occasionally clear out paths that no longer exist. I'll also be looking at changing some of these popup progress gauges to show progress more for the 'current job', like 1/17, rather than always showing entire-life progress like 4001/4017, which is always just a green bar with one pixel of grey, no matter how big the most recent sub sync or whatever was.

>>11231 lol asked about that somewhere too because that was a bit annoying seeing 'oh there are 1000 images here, the fuck happened' only to see the next stage of the sub have 975/983 and realized it remembers everything. not worth resetting the checkers for as to me it was a minor annoyance. I should also say, the weirdness where on saving the session where it would bloat from 12gb to 18+gb has been solved with the removing the dead watchers from active view, however with a little over 100k images in active view mode, I am unable to set it to every 5 minutes as it does start to bloat and lag a bit, no where near the extent it did before, but a bit.

>>11062 So are there any code that you feel is slow in Python 3? And if so would you consider Nim-lang (using NimPy) to speed up parts of the code? Or maybe golang (with GoPy)? Nim is basically a Python-like language that compiles to C, and Golang is made by Google to be a faster, better Java. NimPy: https://narimiran.github.io/2018/05/10/python-numpy-nim.html https://github.com/jboy/nim-pymod https://github.com/yglukhov/nimpy GoPy: https://hackernoon.com/extending-python-3-in-go-78f3a69552ac https://github.com/go-python/gopy https://github.com/sbinet/go-python

>>11083 I am happy with it so far. Some python-heavy computation like the lag on tag siblings/parents cache construction on client boot appears to be running much slower. Overall it feels it is running faster than py2. There's still a bunch of jank thread code on my end jittering things up, which is a long-term project to clean up. There's a lot of duct tape that makes hydrus run from the frozen executables, and I have no experience with non-vanilla versions of python. I expect it is possible to get them going, but I bet trying to install wxPython and OpenCV is an additional headache. For now, I'll stick to the simpler environment. If you have experience with them and try it, let me know how it goes!

>>11155 Whoops, I meant the siblings/parents is running much faster. No significant problems anywhere, is the experience.

iToddlers btfo.

>>11171 Reeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

Happy New Years hydrus dev!

>>11118 Borgbackup is your ticket to make it simple. >>11123 > https://github.com/bup/bup (Python, slow) Actually this one is a fine CLI tool too, but unlike borg it's not considered stable by the developers, and you likely do want some borg features like the ability to prune older database backups and staggered retention for the other files. Slow is extremely relative. Yea, python is not the fastest language around even if you use pypy (which however definitely makes most software that crunches some data quite a bit faster over standard python). But even that may not matter much if you're mainly waiting for IO coming from your HDD, which is usually what happens if you try to get data about your 40TB of image files (or whatever you might have on your array). > See https://github.com/gilbertchen/benchmarking for info A very limited test without the various configurables and apparently without any attempts to isolate how much time was spent waiting for IO or whatever, done only on the weirdly scheduled OS that is OSX? And I don't even see what interpreters they used for python (was the "python" symlink pypy or mainline python and not a link?) and such. Never mind there are actually a LOT of configurables in these tools, and for some weird reason they didn't even try to pick the same compression algorithms even when it was possible (why not run the same settings on the same zlib or lz4 or zstd compression on all the tools so they become more comparable?). Anyhow, use Duplicacy or restic over borg if you prefer, but I wouldn't pay too much attention to that benchmark.

>>11123 I'm using duplicati myself, it's still slow because OS chokes on the amount of hard drive seeks, takes around 30 minutes to back up a 100k set for me, probably going to be even slower when you're backing up from an encrypted volume. It doesn't even matter if there were any changes to the collection, just the act of checking 100k files on a hard drive is a very slow process.

>>11123 I haven't used any of those, so I can't comment cleverly. I expect they do the job well. I use FreeFileSync once a week on my laptop's IRL 1.2M-file hydrus db, backing up to a WD passport. It takes several minutes to compare all the millions of files and thumbs and then a few more to sync the backup. Unless I have a gonzo week, I'd estimate it is usually less than 20min total. Since I have it going in the background as I put the hydrus build together on my dev pc, it is no trouble at all.

>>11098 Merry Christmas and thanks for a great year with Hydrus!

>>11098 Merry christmas.

>>11113 Thank you for this follow-up. Since the db does seem fully shut down and disconnected, my best guess here is that one of the daemon threads (which do maintenance stuff in the background) isn't waking up to receive the 'program shutdown' signal properly. In this case, the process will hang on, with that one thread asleep, until it wakes according to its natural check period (which are typically on the order of hours). I will check this code. It may also magically fix in py3 due to the different way some thread signalling works as well, so please let me know if this improves/worsens after v335. In this case, as the db is completely closed, there is no danger in just killing the process in task manager when this happens again.

>>11045 >but first i wanna ask about what storage hardware is best for hydrus? I don't think it matters, but you may want to keep your media files on a HDD and the rest on a SSD. Just stay away from Seagate, their HDDs are known for having a lot of issues. I'd recommend HGST or WD. https://hydrusnetwork.github.io/hydrus/help/database_migration.html

>>10020 >See through gel lady >Hydra How embarrassing for a /monstershit/

I'd call that a naiad, not a goo-girl. Maybe she's hydrantish in the sense that, like a fire hydrant, she pours water?

fishfags fuck off

>10830 Imagine actually believing this is a serious post

>>10829 < Not valuing API because archiving is a waste of time When everything is deleted off the internet constantly you will regret that.

>>10992 here, this is a known link that doesn't pull images up, I tried going incognito to find ones, but apparently it finds the images that way too. for the artist, so I stuck with a known one that wouldn't work. https://mega.nz/#!KooGTIBC!aRlDdrI1XRA-iMDmkzye7J6Q-C3bTWnb3_74-8-0eDI Honestly, I think full stopping the download, program wide, till a log in finishes would be a good work around for this. or with the downloaders that these download from, a double tap when logging in, the first page or two of searches could get redone twice, because by then everything should be logged in, and if something didn't catch it would get them in the first two pages.

>>10991 I set everything to super high file and data limits there already though (like 100k files and 100s of TBs worth of bandwidth), and it's still "waiting on bandwidth" half the time. That change affected both my global and the specific tumblr rules. Also tumblr.com specifically (not its subdomains) has "yes" in the "blocked?" column. What does this mean?

>>11000 Oh, never mind, I had to edit the web domain rules as well as the subscription rules, seems like. All working now.

>>10859 Thanks, 0.8 works better, it scrolls a full page per tick now. I have my scroll length set to 5. One page is about 5½ thumbs high. I think this might also be related to the fact that I have Windows 10 set to 125% scale, Hydrus doesn't seem to handle that quite correctly (for example when resizing the width of the tagging panel to the left of the thumbnail view).

>>10917 Not yet, but I'd like to present that information better in several ways. Adding search would also be useful.

>>10933 I hope to check out high-dpi stuff in the new year. The new wx is supposed to be able to handle it better, so I'll have a look and see if I can figure out what flag I need to set or whatever. I have a 4k monitor now, so I can test this better.

>>10896 1/3 of the millenium… time goes by fast

>>10897 I don't really get what you're saying… it was simply a play on "three three three" with the image, since NS Germany is referred to as the "Third" Germany as well.

>>10899 333 is one third of 1000, i.e. the Millennium Reich

>>10843 I was afraid of updating as my hydrus db has surpassed my backup drive in size. I updated to 332 (from 317) anyway, everything went fine phew. Running from source now yields this error: Could not import lz4--nbd.
Traceback (most recent call last):
File "H:\Hydrus Network\client.py", line 20, in <module>
from include import ClientController
File "H:\Hydrus Network\include\ClientController.py", line 14, in <module>
import ClientCaches
File "H:\Hydrus Network\include\ClientCaches.py", line 1, in <module>
import ClientDefaults
File "H:\Hydrus Network\include\ClientDefaults.py", line 2, in <module>
import ClientData
File "H:\Hydrus Network\include\ClientData.py", line 3, in <module>
import ClientDownloading
File "H:\Hydrus Network\include\ClientDownloading.py", line 2, in <module>
import ClientNetworkingDomain
File "H:\Hydrus Network\include\ClientNetworkingDomain.py", line 3, in <module>
import ClientParsing
File "H:\Hydrus Network\include\ClientParsing.py", line 4, in <module>
import ClientNetworkingJobs
File "H:\Hydrus Network\include\ClientNetworkingJobs.py", line 9, in <module>
import HydrusNetworking
File "H:\Hydrus Network\include\HydrusNetworking.py", line 9, in <module>
import ssl
File "C:\Python27\lib\ssl.py", line 126, in <module>
from _ssl import HAS_SNI, HAS_ECDH, HAS_NPN, HAS_ALPN, HAS_TLSv1_3
ImportError: cannot import name HAS_TLSv1_3

>>10861 I removed the _ssl.pyd file that comes with the hydrus installer and it seems to be working after doing that. Same thing for the lz4*.pyd files. No more import errors.

>>10875 Great, I am glad you figured this out. Let me know if you encounter any more problems.

>>10868 I get the error of pic related. I already followed instructions to reinstall the .dll for this, so I am at a loss what to do now. Pls halp ;_;

>>10869 >>10868 Sorry for not putting this all into one post - but I tried to install "bionus" "Imgbrd grabber" by the way. Is something like this possible with hydrus!? I never tried…

>>10870 Imgbrd is a separate project that does similar things, this board is not responsible for it, best ask in https://github.com/Bionus/imgbrd-grabber/issues Bionus & pals has promised to make things easier for us to import data to Hydrus, but it is not yet well integrated. https://github.com/Bionus/imgbrd-grabber/issues/1001 and https://github.com/Bionus/imgbrd-grabber/issues/588 has some notes

>>10812 What were you trying to download? I sometimes open various(10-20) doujinshis at once and use a full page script to leave them all opening at once to then save them, but it's a short rapid burst. If you're downloading a lot of shit over an extended uninterrupted period of time that could be why.

>>10822 the last one I downloaded was an 8 page comic with everything spaced out 1 minute, I did all my testing on very small downloads and regardless of what I did, 1-1 week long lockouts. It seems they want you to pay them for the ability to download things, as 20$ gets you a weekly 5gb pass on downloads, something I would gladly pay, but I need to link bank accounts and do money transfers for cryptocurrencies, and i believe my bank was one that was one that was punishing people for even trying.

>>10811 >>10808 Thanks. I hadn't heard of the jpg-orig trick. Hydrus currently does the :orig conversion, which works well as long as the artist themselves didn't originally upload a 350x400 80% quality jpg. :^) I know that twitter can provide direct mp4 links for videos, but I haven't figured out a way to parse this info. This service provides it: https://twitter.com/this_vid But takes a couple of minutes to produce an answer, so my guess is they manually request it from twitter using a legacy "I'm an old phone" http header or something and then forward the link. Normal twitter video works with some DASH streaming or something that we can't handle (yet). If you discover a way to get simple mp4 links for twitter vids, please let me know!