/hydrus/ - Hydrus Network

Archive for bug reports, feature requests, and other discussion for the hydrus network.

Index Catalog Archive Bottom Refresh
Name
Options
Subject
Message

Max message length: 12000

files

Max file size: 32.00 MB

Total max file size: 50.00 MB

Max files: 5

Supported file types: GIF, JPG, PNG, WebM, OGG, and more

E-mail
Password

(used to delete files and posts)
Misc

Remember to follow the Rules

The backup domains are located at 8chan.se and 8chan.cc. TOR access can be found here, or you can access the TOR portal from the clearnet at Redchannit 3.0.

Uncommon Time Winter Stream

Interboard /christmas/ Event has Begun!
Come celebrate Christmas with us here

8chan.moe is a hobby project with no affiliation whatsoever to the administration of any other "8chan" site, past or present.
(42.63 KB 480x360 FfJ8SWDV7Xw.jpg)

Version 341 hydrus_dev 02/27/2019 (Wed) 23:49:51 Id: 229563 No. 11732
https://www.youtube.com/watch?v=FfJ8SWDV7Xw windows zip: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.Windows.-.Extract.only.zip exe: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.Windows.-.Installer.exe os x app: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.OS.X.-.App.dmg linux tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.Linux.-.Executable.tar.gz source tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v341.tar.gz I had a great week after being ill at the end of the previous week. There is a bit more than usual in this release because of the extra time. The Client API has some new commands, and there are a variety of fixes and stability improvements. client api The Client API now does some fun stuff. If you would like to play with one user's browser extension, which recognises hydrus-compatible URLs–including whether you have the page's file already–and lets you send them straight to hydrus for downloading, please check it out here: https://gitgud.io/prkc/hydrus-companion And the Client API help itself is here: https://hydrusnetwork.github.io/hydrus/help/client_api.html There is a little info at the top telling you how to turn the API on. This release brings some more API commands. You can now edit a file's tags, edit a file's URLs, and the 'add_url' command now lets you attach tags for any files coming from the given URL and also lets you determine which page (by page name) the URL will go to (for instance if you have multiple watcher pages working on different check timings for different boards). the rest I spent some time this week cleaning things up. Repository update processing uses less memory and now regularly saves its progress on big jobs, reducing transaction bloat, improving error recovery, and giving control back to you much faster when it finishes. Memory use all around should be reduced. I also replaced several areas of messy UI code, which has resulted in improved stability, particularly for Linux. Some CRITICAL logspam is gone, and I believe I have finally fixed a lingering ui crash on the duplicates processing page (at least for search distance 0–I will still work on >0, which seems to be a db-level problem for linux). Several boot crashes due to incorrect boot-error handling (such as when catching that some client_files directories are missing) are also fixed. I would appreciate your continued feedback here. Duplicate processing now treats ratings copy/move/merge actions a little different: Before, it would only make a change if the 'destination' file had no rating set–now it will do it if it has no rating or if the rating is smaller than what the 'source' has. For instance, merging two files that have rating 2/5 and 3/5 will result in both files having rating 3/5. After talking with a user about different ways of storing and presenting thumbnails, I put some work into a thumbnail drawing experiment that only needs one set of thumbs to draw on screen (the current thumbnail system stores two copies for different purposes). If you would like to try it temporarily, please turn it on under help->debug->gui actions->thumbnail experiment mode. Instead of using your 'resized' thumbs, it will only use the 'master'. It makes no changes to your file structure, and you can turn it off at any time. Try loading up some random files in your client and let me know if you get any errors or if any thumbs take an extremely long time to load. I've wanted to drop the two-thumb storage system for some time but never had good numbers to feel good about it, but my tests here are promising–the experiment mode only takes a few hundred µs more per initial thumb load for me, and if I were to switch over to it permanently, I could reduce it even further. Assuming there are no big problems here, I expect to chip away at this problem over the coming weeks and hopefully make a significant simplification in hydrus file storage. full list - client api: - added /add_tags/add_tags, which does several kinds of tag content updates - added /add_tags/clean_tags, which shows how hydrus will handle potential tags - added /add_urls/associate_url, which allows you to associate urls with files - added 'destination_page_name' to /add_urls/add_url, which will choose which destination watcher/url importer to place the url (or create a new one with that name) - updated client api version to 2 - updated client help and unit tests for the above - added a linked contents to the client api help - improved some server error handling, mostly moving 403s to more correct 400s
[Expand Post]- improved how missing parameter 400 errors are reported from the server vs deeper keyerrors that should be 500 - . - the rest: - tag repository update processing now saves progress to disk every million rows or every minute, whichever comes first. this reduces journaling bloat, improves recovery when the process quits unexpectedly, and makes for significantly faster cancel when requested by the user - when processing duplicates and copying/merging/moving ratings, the 'source' file will now also overwrite the 'destination' file's rating if that destination rating is lower (previously, the rating would only go over if the dest had no rating set) - added a new 'thumbnail experiment mode' under help->debug->gui. this will load fullsize thumbs and resize them in memory, please see release post for more details - reduced menubar replacement flicker while, I believe, keeping and strengthening recent menubar indexing stability improvements - the tag autocomplete dropdown will now always embed (instead of floating) in non-Windows - when data seems non-decodable, the fallback encoding format is now that given by chardet, rather than utf-8 - improved serialisability of some pending tag data - watchers can now hold and pass on fixed pending tag data - gallery log objects can now hold and pass on fixed pending tag data - file import objects can now hold and action fixed pending tag data - hard drive imports now store their paths-to-tags info in this new format, directly in the file import objects - improved some url-import page drop-target-selection logic - improved error reporting when dropping/api-adding urls - adjusted some url import workflow so big 'already in db' download lists should work a bit faster - attempting to start the program with some external database files but not the main 'client.db/server.db' file will now cause a boot-fail exception with an explanation before any stub db files can be made - tightened up some hydrus service login-capability-testing code that was previously stopping certain error states from recovering promptly, even on a force account refresh, while the service was maxed on bandwidth - fixed a source of linux CRITICAL logspam related to several common dialogs - improved ui stability on boot when file folders are missing (particularly for linux) - improved stability for the various async tasks on the duplicates processing page, particularly for linux. I am not sure I got everything here, but it is definitely better - did some more misc stability improvements, particularly in various boot fail scenarios - completely removed an ancient and janky focus catcher widget from the main gui frame - now various db caching is improved on the python side, removed a sqlite instruction to force temp information to always stay in memory–hot data returns to staying mostly in memory to start and then spools to disk if the transaction gets too large - fixed approx bitrate sorting for malformed video files with explicitly '0' duration - daemon_profile_mode now spams some more info about export folders - fixed an issue that meant client db maintenance was firing its jobs too aggressively, regardless of idle status - updated windows build to cv 4.0 - misc refactoring and fixes next week We are in the home stretch of the Client API now. I just want to write some simple file searching and it'll be done. I do not expect to be done in one week, but I think I should have something working for 342. It felt good this week to clear out some long-time bugs doing code cleaning, so I'd like to focus on that as well.
Anonymous 02/28/2019 (Thu) 00:21:05 Id: 8a8e99 No. 11734
>>11732 No immediate noticeable difference with the thumbnail mode on, if its pulling from my thumbnail library its pulling them off an nvme though so reguardless its going to be fast
Anonymous 02/28/2019 (Thu) 02:05:23 Id: 9f0fef No. 11737
hey dev i was just wondering wether or not the functionality to get/download the images by searching through them in a booru style fashion will be available through the api or if that will stay exclusive to the hydrus booru server
Anonymous 02/28/2019 (Thu) 03:38:08 Id: 08f8cb No. 11739
Hey dev, any chance of you adding in a way to exclude images with certain tags/ratings from searches unless they're explicitly searched for? Sort of like the tag censoring, but with the images that fall under those tags being excluded.So, if I were to have a namespace entitled lewd: and set that to be excluded, no images whose tags included lewd: would show in searches unless I explicitly included the lewd: namespace into my current search
Anonymous 02/28/2019 (Thu) 14:40:48 Id: b84c1b No. 11742
>>11739 This would be nice to have
Anonymous 02/28/2019 (Thu) 15:19:19 Id: 389540 No. 11743
Why does Hydrus need two sizes of thumbnails anyway? 🤔
Anonymous 02/28/2019 (Thu) 20:45:00 Id: 3a6419 No. 11744
Can you use the API to access hydrus features other than importing and exporting images/tags? I was thinking of making a script to automate the IQDB lookup script for multiple files.
Anonymous 03/01/2019 (Fri) 00:58:53 Id: 1af804 No. 11746
341 is broken for me, endless database errors DBException OperationalError: unable to open database file Traceback (most recent call last): File "include\HydrusThreading.py", line 342, in run callable( *args, **kwargs ) File "include\ClientGUICanvas.py", line 4172, in process_in_thread HG.client_controller.WriteSynchronous( 'content_updates', { service_key : [ content_update ] } ) File "include\HydrusController.py", line 790, in WriteSynchronous return self._Write( action, True, *args, **kwargs ) File "include\HydrusController.py", line 228, in _Write result = self.db.Write( action, synchronous, *args, **kwargs ) File "include\HydrusDB.py", line 976, in Write if synchronous: return job.GetResult() File "include\HydrusData.py", line 1519, in GetResult raise e include.HydrusExceptions.DBException: OperationalError: unable to open database file Database Traceback (most recent call last): File "include\HydrusDB.py", line 580, in _ProcessJob result = self._Write( action, *args, **kwargs ) File "include\ClientDB.py", line 12306, in _Write elif action == 'content_updates': self._ProcessContentUpdates( *args, **kwargs ) File "include\ClientDB.py", line 7990, in _ProcessContentUpdates self._DeleteFiles( service_id, hash_ids ) File "include\ClientDB.py", line 3121, in _DeleteFiles self._CacheSpecificMappingsDeleteFiles( service_id, tag_service_id, existing_hash_ids ) File "include\ClientDB.py", line 2353, in _CacheSpecificMappingsDeleteFiles self._c.executemany( 'DELETE FROM ' + cache_current_mappings_table_name + ' WHERE tag_id = ? AND hash_id = ?;', ( ( tag_id, hash_id ) for hash_id in current_hash_ids ) ) sqlite3.OperationalError: unable to open database file Database Traceback (most recent call last): File "include\HydrusDB.py", line 580, in _ProcessJob result = self._Write( action, *args, **kwargs ) File "include\ClientDB.py", line 12306, in _Write elif action == 'content_updates': self._ProcessContentUpdates( *args, **kwargs ) File "include\ClientDB.py", line 7990, in _ProcessContentUpdates self._DeleteFiles( service_id, hash_ids ) File "include\ClientDB.py", line 3121, in _DeleteFiles self._CacheSpecificMappingsDeleteFiles( service_id, tag_service_id, existing_hash_ids ) File "include\ClientDB.py", line 2353, in _CacheSpecificMappingsDeleteFiles self._c.executemany( 'DELETE FROM ' + cache_current_mappings_table_name + ' WHERE tag_id = ? AND hash_id = ?;', ( ( tag_id, hash_id ) for hash_id in current_hash_ids ) ) sqlite3.OperationalError: unable to open database file putting back 340 I get no issues
Anonymous 03/01/2019 (Fri) 16:59:11 Id: 2f5629 No. 11748
>>11732 > help->debug->gui actions->thumbnail experiment mode > Try loading up some random files in your client and let me know if you get any errors or if any thumbs take an extremely long time to load. So far, this seems to be working just fine here. I don't notice any difference to the extent where it's pretty hard to tell if the option is doing anything at all, actually.
Anonymous 03/02/2019 (Sat) 01:35:49 Id: bf140b No. 11750
thumbnail experiment mode loads them nearly instantaneously for me. it'd be great to be able to free up that space up.
Anonymous 03/02/2019 (Sat) 03:11:30 Id: 355098 No. 11751
(170.99 KB 1440x862 Hydrus.jpg)

(147.48 KB 1440x860 Hydrus2.jpg)

>>11732 Having issues with the new thumbnails mode on my 11 year old toaster of a system. They lag loading slightly while Hydrus is doing other stuff like subscription downloads. Also did this weird thing with my background color on the duplicates page…
Anonymous 03/02/2019 (Sat) 03:56:53 Id: 44eedb No. 11753
I'm getting a segfault in Python when clicking the duplicate '2 - discovery' play button
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `python -OO /opt/hydrus/client.pyw'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f1a954cb223 in sqlite3VdbeFindCompare () from /usr/lib/libsqlite3.so.0
[Current thread is 1 (Thread 0x7f1a6257a700 (LWP 9481))]
(gdb) bt
#0 0x00007f1a954cb223 in sqlite3VdbeFindCompare () at /usr/lib/libsqlite3.so.0
#1 0x00007f1a9542e157 in sqlite3BtreeMovetoUnpacked () at /usr/lib/libsqlite3.so.0
#2 0x00007f1a954c1b86 in sqlite3VdbeExec () at /usr/lib/libsqlite3.so.0
#3 0x00007f1a954c6920 in sqlite3_step () at /usr/lib/libsqlite3.so.0
#4 0x00007f1a955a10f0 in pysqlite_step ()
at /usr/lib/python3.7/lib-dynload/_sqlite3.cpython-37m-x86_64-linux-gnu.so
#5 0x00007f1a955a406b in _pysqlite_query_execute ()
at /usr/lib/python3.7/lib-dynload/_sqlite3.cpython-37m-x86_64-linux-gnu.so
#6 0x00007f1a95da7a78 in _PyMethodDef_RawFastCallKeywords () at /usr/lib/libpython3.7m.so.1.0
#7 0x00007f1a95dd30ff in _PyMethodDescr_FastCallKeywords () at /usr/lib/libpython3.7m.so.1.0
#8 0x00007f1a95e1e479 in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#9 0x00007f1a95dd4149 in () at /usr/lib/libpython3.7m.so.1.0
#10 0x00007f1a95e1a271 in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#11 0x00007f1a95da72eb in _PyFunction_FastCallKeywords () at /usr/lib/libpython3.7m.so.1.0
#12 0x00007f1a95e19c42 in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#13 0x00007f1a95d60b99 in _PyEval_EvalCodeWithName () at /usr/lib/libpython3.7m.so.1.0
#14 0x00007f1a95d61dec in _PyFunction_FastCallDict () at /usr/lib/libpython3.7m.so.1.0
#15 0x00007f1a95d71908 in _PyObject_Call_Prepend () at /usr/lib/libpython3.7m.so.1.0
#16 0x00007f1a95d6233b in PyObject_Call () at /usr/lib/libpython3.7m.so.1.0
#17 0x00007f1a95e1b43c in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#18 0x00007f1a95d60b99 in _PyEval_EvalCodeWithName () at /usr/lib/libpython3.7m.so.1.0
#19 0x00007f1a95d61dec in _PyFunction_FastCallDict () at /usr/lib/libpython3.7m.so.1.0
#20 0x00007f1a95d71908 in _PyObject_Call_Prepend () at /usr/lib/libpython3.7m.so.1.0
#21 0x00007f1a95d6233b in PyObject_Call () at /usr/lib/libpython3.7m.so.1.0
#22 0x00007f1a95e1b43c in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#23 0x00007f1a95da72eb in _PyFunction_FastCallKeywords () at /usr/lib/libpython3.7m.so.1.0
#24 0x00007f1a95e19c42 in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#25 0x00007f1a95d61c0b in _PyFunction_FastCallDict () at /usr/lib/libpython3.7m.so.1.0
#26 0x00007f1a95d71908 in _PyObject_Call_Prepend () at /usr/lib/libpython3.7m.so.1.0
#27 0x00007f1a95d6233b in PyObject_Call () at /usr/lib/libpython3.7m.so.1.0
#28 0x00007f1a95e1b43c in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#29 0x00007f1a95da72eb in _PyFunction_FastCallKeywords () at /usr/lib/libpython3.7m.so.1.0
#30 0x00007f1a95e19c42 in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#31 0x00007f1a95da72eb in _PyFunction_FastCallKeywords () at /usr/lib/libpython3.7m.so.1.0
#32 0x00007f1a95e19c42 in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#33 0x00007f1a95d61c0b in _PyFunction_FastCallDict () at /usr/lib/libpython3.7m.so.1.0
#34 0x00007f1a95d71908 in _PyObject_Call_Prepend () at /usr/lib/libpython3.7m.so.1.0
#35 0x00007f1a95d6233b in PyObject_Call () at /usr/lib/libpython3.7m.so.1.0
#36 0x00007f1a95e90aa7 in () at /usr/lib/libpython3.7m.so.1.0
#37 0x00007f1a95e38fa5 in () at /usr/lib/libpython3.7m.so.1.0
#38 0x00007f1a96196a9d in start_thread () at /usr/lib/libpthread.so.0
#39 0x00007f1a960c6b23 in clone () at /usr/lib/libc.so.6
 Running the Arch Linux AUR package with Python 3.7.2 Problem occurred with Hydrus 340 as well.
Anonymous 03/02/2019 (Sat) 04:14:10 Id: 389540 No. 11755
I think the gelbooru downloader grabbing the rating tag is broken because it seems to give everything the mature rating. This for example got a mature rating but it's clearly marked as safe. https://gelbooru.com/index.php?id=2475423&page=post&s=view (also is mature the new explicit or what)
Anonymous 03/02/2019 (Sat) 04:22:31 Id: 389540 No. 11756
The new thumbnail system runs fine here but I have a pretty powerful system. A 9700k, 16gb ram and the thumbs are on an encrypted NVMe SSD with the originals on an encrypted 5400rpm SATA HDD. However I did notice that while I had a page open that was constantly added to through subscription updates, some thumbnails didn't load and were black. Once I clicked on them they loaded, though.
Anonymous 03/02/2019 (Sat) 08:59:20 Id: 355098 No. 11757
>>11756 > some thumbnails didn't load and were black. Once I clicked on them they loaded, though. Yeah that's been an issue for me since before tumblr wiped all the porn. It's not really a big deal so I never mentioned it.
Anonymous 03/02/2019 (Sat) 18:45:02 Id: 3f70b4 No. 11765
Thank you based dev.
Anonymous 03/03/2019 (Sun) 00:24:56 Id: c7fd47 No. 11767
I really want to try hydrus but you have 3 detections on virus total ;-;
hydrus_dev Board Owner 03/03/2019 (Sun) 01:04:49 Id: 229563 No. 11768
>>11737 Yep, that should be what I am working on next. It will support simple tag searches to begin with and give basic file metadata and file and thumbnail fetches. If you want to talk to the API to do this stuff, then with a bit of wizardry on your end you could write up a simple booru much better than my old prototype, with proper tag search. You'll be able to restrict API access a bit as well, btw, saying that it only works for searches that include certain tags (think maybe a tag something like "share this on my booru"), so you don't have to give full access to every booru 'window' that is looking at your client. Let me know how this works for you. I haven't been able to find time to improve my old booru, so I'd love it I could offload that to other users who can do it justice. It is much easier for me to expand the API to now allow filesize searches or whatever than to keep up with a whole booru codebase.
hydrus_dev Board Owner 03/03/2019 (Sun) 01:09:12 Id: 229563 No. 11769
>>11739 >>11742 That's an interesting thought, and I hadn't considered approaching file censorship/hiding from that perspective before. My original thought here has been to add multiple local file services, so you can stick all your nsfw in a completely different file domain. This way, when you type in 'fu' for a 'futurama' tag search with a family member looking over your shoulder, you won't get a variety of other 'fu-' beginning tags that you might not want appearing in the results. I haven't made much progress on this because it will be a complicated job. It is still the best solution here–although I do like you idea of only allowing certain files if the search itself includes a magic word, and I'll think about it a bit more, I still think it will be a bit whack-a-mole trying to keep it secret/separated. The old solution here for now has been to run two clients–one for safe stuff, and one for spicy stuff. It is clunky, but it works.
hydrus_dev Board Owner 03/03/2019 (Sun) 01:17:38 Id: 229563 No. 11770
>>11743 When I first wrote it, SSDs were still pretty rare, and I was working with the slower PIL library, so many thumbnail actions were still pretty slow. It was nice to have a fast backstop thumbnail at 200x200 to regen new 'resized' thumbnails rather than going back to the master file to regen. There's also some Hydrus Network legacy stuff to fit in, since a file repository always transfers the master to the client for resizing to whatever is desired locally. Now the hardware and software has all moved on, fitting a more dynamic "Hey, that thumb we loaded is the wrong size, let's regen a new one from an appropriate source, save it back to disk, and deliver the correct one to the user all in one step" action is more feasible. We also have much bigger hydrus installs (I've got more than a million files on my IRL client, so I'm up to my ears in thumbs atm), so it is worth putting the effort in.
hydrus_dev Board Owner 03/03/2019 (Sun) 01:19:05 Id: 229563 No. 11771
>>11744 Please check the file search stuff I'll be rolling out in this final 1.0 phase. It might be able to do what you want–something like you tag a file in the client with 'do iqdb on this', and your script fetches those files and does its work, writing back what it finds.
hydrus_dev Board Owner 03/03/2019 (Sun) 01:24:14 Id: 229563 No. 11772
>>11746 Thank you for this report. I reversed an old memory-only db mode in 341. SQLite will now attempt to write a journalling file to your temp directory when it writes changes–is there anything about your temp dir (maybe a ramdisk?) that might cause permission problems here? I assume you were able to boot the client and update to 341, right? So these ProcessContentUpdate calls were you trying to manage some tags once it had booted? It is odd that client.db presumably accepted the simple 'You are now 341' write but client.mappings.db had trouble. This is odd, but did your client.mappings.db somehow get set read-only? OperationalError is a pretty low-level error. What happens if you run the sqlite3 executable in the install_dir/db directory and then type: .open client.mappings.db
SELECT * FROM sqlite_master;
.exit Is it ok reading that, or does that also give an error?
hydrus_dev Board Owner 03/03/2019 (Sun) 01:26:07 Id: 229563 No. 11773
>>11772 Thinking this through more, it doesn't make sense that the file would be read only and also 340 works ok for you. Please explore any permission errors in your temp dir, but I'll also do a bit of searching on my end.
hydrus_dev Board Owner 03/03/2019 (Sun) 01:29:19 Id: 229563 No. 11774
>>11751 Thank you. I should be able to clear up most of the lag here if I commit to this new system. The band of colour is odd–did you happen to turn on help->debug->debug modes->thumbnail debug mode? That shades the 'tiles' of the thumbnail grid differently so I can see where the borders are.
hydrus_dev Board Owner 03/03/2019 (Sun) 01:32:10 Id: 229563 No. 11775
>>11753 Thank you. This is the last crash in this system. Search distance 0 (exact match) works ok. I worked with another Linux user today and we believe we have this crash fixed for 342. It seemed to be unhappy with how I was doing some similar-hash comparison during the SQLite query iteration that was fetching them. Please let me know how that works for you.
hydrus_dev Board Owner 03/03/2019 (Sun) 01:34:54 Id: 229563 No. 11776
>>11755 Thank you for this report. The parser appears to be pulling rating info from the wrong location. I will fix this myself and update it for next week.
hydrus_dev Board Owner 03/03/2019 (Sun) 01:38:19 Id: 229563 No. 11777
(5.15 KB 200x200 hydrus.png)

>>11756 >>11757 Can you talk more about this black thumbnail issue? Could they have been dark with the hydrus image (pic related) transparently on top? This is the default image used when hydrus can't find a thumb on your disk, but it usually comes with an error popup. Normally, thumbs should be ready to display before a 'this file is imported, throw it on screen' call goes through, but I have been changing some of this recently.
hydrus_dev Board Owner 03/03/2019 (Sun) 01:48:41 Id: 229563 No. 11778
>>11767 Thank you for this report. I cannot promise with absolute certainty that my dev machine is clean, but every instance of this happening so far has been due to false positives. My code is unprofessional and unsigned, with unusual frozen python packaged into InnoSetup, and it has a bunch of downloading and uploading and UPnP components that I presume may appear to a scanner to be bad business. Every 4-6 months, a user reports that Kaspersky or Avast or whatever considers a particular dll (or often the upnpc exes in install_dir/bin) to be bad, and then it goes away in a bit when they update their definitions. I recommend you wait a bit and scan the latest release in a few weeks. My guess is the report will be different. I would obviously be very interested in any evidence of a true-positive. Out of interest, does the scan say which files are bad? If it is upnpc again, I can point you at the files I use on their original site, and you can even delete them and still run hydrus fine.
Anonymous 03/03/2019 (Sun) 02:23:16 Id: 3a6419 No. 11779
>>11769 >I haven't made much progress on this because it will be a complicated job. Couldn't you make so the user can blacklist/whitelist files based on tags and use those as search terms hidden from the user? Like blacklisting rating:explicit and whitelisting "rating:safe" makes it so whever you search something, it automatically also searches for "rating:safe" and "-rating:explicit" as well, except it doesn't show it to the user?
Anonymous 03/03/2019 (Sun) 02:28:16 Id: 3a6419 No. 11780
>>11769 >>11779 Also, I don't know how hydrus works, but if it just throws basic SQL around, how about having a whitelist/blacklist string and appending it to the end of searches? Every time the user adds another whitelist/blacklist it automatically generates the string, then when you do searches you just use python concatenation to do "SQLshit + search_terms + whiteandblacklist".
Anonymous 03/03/2019 (Sun) 02:37:31 Id: 5af3c2 No. 11781
>>11773 I tested a thing, backup with FFS and then deleted the install folder. Install 341 again. Quick check that permissions look fine (removed the "read only" that windows loves SO MUCH) and tried to start the client: A serious error occurred while trying to start the program. The error will be shown next in a window. More information may have been written to client.log. 2019/03/03 03:20:45: Traceback (most recent call last): File "include\ClientController.py", line 1351, in THREADBootEverything self.InitModel() File "include\ClientController.py", line 605, in InitModel HydrusController.HydrusController.InitModel( self ) File "include\HydrusController.py", line 501, in InitModel self.db = self._InitDB() File "include\ClientController.py", line 99, in _InitDB return ClientDB.DB( self, self.db_dir, 'client', no_wal = self._no_wal ) File "include\ClientDB.py", line 191, in init HydrusDB.HydrusDB.init( self, controller, db_dir, db_name, no_wal = no_wal ) File "include\HydrusDB.py", line 200, in init self._InitDB() File "include\HydrusDB.py", line 437, in _InitDB self._CreateDB() File "include\ClientDB.py", line 2987, in _CreateDB self._AddService( service_key, service_type, name, dictionary ) File "include\ClientDB.py", line 335, in _AddService self._CacheSpecificMappingsGenerate( file_service_id, service_id ) File "include\ClientDB.py", line 2452, in _CacheSpecificMappingsGenerate self._CreateIndex( cache_pending_mappings_table_name, [ 'tag_id', 'hash_id' ], unique = True ) File "include\HydrusDB.py", line 375, in _CreateIndex self._c.execute( statement ) sqlite3.OperationalError: unable to open database file
Anonymous 03/03/2019 (Sun) 05:19:50 Id: 355098 No. 11782
>>11774 >The band of colour is odd–did you happen to turn on help->debug->debug modes->thumbnail debug mode? Ah, I did! I just saw thumbnails and thought that was it. Trying the correct option now, they seemingly load around the same speed either way. >>11777 My background is set to black so it's possible. The title/artist bar and inbox icon still res'd in the grid like they should, just the thumbnail wasn't there. But clicking on the tile to highlight it instantly loaded the thumbnail. I noticed it when doing tumblr downloads and stuff so it was mostly on pages with a fair number of files ~100-500. Ironically now I can't replicate it after moving to 341. I was sitting on 339 and don't recall seeing it but I wasn't really using the client much either. If I had to describe it, it was like when you normally load up a page of files and sometimes certain thumbs take longer than the rest of their row to load. So the slow loading thumb will be the last thumb to load despite it being in the middle of your scroll space. It was pretty much that except the thumb just never loaded. Actually, now that I think about it, something else seemed off too, like the border was darker or something. I'm probably just mis-remembering that. I'll let you know if I see it again.
Anonymous 03/03/2019 (Sun) 07:21:17 Id: 389540 No. 11784
>>11777 My background is black so it is possible that image was there, but I don't know. There were no error popups though.
Anonymous 03/03/2019 (Sun) 07:24:33 Id: 8a8e99 No. 11785
hey hdev, I thought I would bring this to your attention. http://comicrack.cyolito.com/ So far, outside of acdsee, this is the best manga program I have found. as much as I like it, it seems abandoned. Ok, not horrible but not good either. I think that when you do get rar/zip/manga version of the program going, you may want to look at this for inspiration, there are aspects I really like, like tabbed comics so you can have more than one book open at a time, personally this is VERY useful for when I have some shit open that i'm reading, I check the new stuff and I want to read that more, but I want to go back to the older book How I am able to have multiple searches/lists open at once, this is something that may even be useful for images, but I think manga suits it more especially for manga, the way it does tags is REALLY good, as there is a fairly limited range of tags you can do for it unless you want to really go into details, at least far more limited then with single images where people start to tag eye color and hairstyles. I think this program could very well be a blueprint for how to handle manga. granted, this program will let you keep shit where it is, which if it was hydrus, that wouldn't fly, however, use parseable is a requirement I have for using hydrus for manga. thought you may want to check it out.
Anonymous 03/04/2019 (Mon) 05:07:43 Id: 43a746 No. 11789
>>11732 Problem with dark mode: the white boxes for tags and others don't change color, they stay white.
hydrus_dev Board Owner 03/04/2019 (Mon) 22:39:35 Id: 229563 No. 11792
>>11781 Right. If a fresh install cannot create a new db, and given the use of the OS's temp location for journalling files is I think the only thing I really changed, I suspect this is a temp directory problem. Can you try opening the sqlite3 executable in install_dir/db and doing this: .open test.db
CREATE TABLE t ( n INTEGER );
INSERT INTO t VALUES ( 13 );
SELECT * FROM t;
.exit You should be able to paste that in line by line. If a test.db file is created, you can delete it afterwards. If a very simple sqlite db on stock settings cannot be created here, please check that your temp location is 'correct'. I had presumed you were on Linux because this is where users have had temp directory problems before, but if you are on Windows, you are probably looking at C:\users\(You)\AppData\Local\Temp . Is there anything odd about that directory? Is C drive completely full, or could there be any user permissions problem related to writing to that folder? Has C drive (or wherever your temp folder is) had any problems recently, maybe that could have set a 'dirty bit' on the drive? If 340 is still ok for you, you may get a hint at what SQLite is using for its temp location by looking at what hydrus has under help->about. It should be that …\local\temp location, but if you have an unusual environment, perhaps it has been relocated? If the test.db goes fine, please check your temp location anyway and let me know and I'll see what else might be going on here.
Anonymous 03/05/2019 (Tue) 08:17:57 Id: 044f62 No. 11793
341 and 340 (haven't tried earlier versions) are prone to locking up on Arch Linux when processing repository updates, forcing me to kill hydrus.
Anonymous 03/05/2019 (Tue) 12:35:18 Id: 44eedb No. 11794
>>11775 You're a bloody legend mate. >>11779 >>11780 I've observed that the blacklists on boorus just run a computationally cheap filter after the query. The query gets 30 posts per page normally, then the filter knocks out 9 on page 1, 13 on page 2, etc. Hydrus doesn't have to paginate the results so this is way less ugly than a web page. >>11793 Repo updates are quite slow on my Arch system with /home on btrfs on a hard disk (I usually do them manually in services->review services where the progress bar locks up but eventually pulls through). It doesn't locked up more than usual with v340+ for me. How many hours are you waiting for the 30 min shutdown process?
hydrus_dev Board Owner 03/05/2019 (Tue) 23:01:35 Id: 229563 No. 11796
Sorry lads, I have had a busy week and haven't found time to catch up with all my messages. I'll try to do a bit tomorrow morning before putting the release together, otherwise it may delay to Saturday. >>11781 For tomorrow's 342, I have added a launch parameter to the program, '–no_db_temp_files', that will tell the database to return to 340's behaviour and operate entirely in memory. Please try this when it is convenient and let me know how it works. If it works, that means this is a temp directory permissions/whatever problem. If at that point you are still interested in figuring out what's going on here, we can, or you can just stay in memory. If so, you might like to disable db vacuuming in the options. If you are not sure how to launch the program with this parameter, try going to your install directory and creating a new shortcut to client.exe. Edit the shortcut's options and change the 'Target' line to: "X:\path to hydrus\client.exe" --no_db_temp_files If you already use the '-d="path"' parameter to relocate your db, it is fine to use both.
Anonymous 03/07/2019 (Thu) 22:07:41 Id: fe42ee No. 11810
>>11796 help>about has shown the issue… it used the install directory as temp. Don't ask how, this is as far as I know. In the settings I tried to put it in the correct folder (nope) and it worked in a new, different folder purpose made. 342 still spits errors with that, HOWEVER! –no_db_temp_files does work The correct temp directory seems fine, several temporary looking files and there's plenty of free space so I really have no idea wtf is going on. Also I don't really have other issues (that I know of at least!) so uh… yay I guess?
hydrus_dev Board Owner 03/09/2019 (Sat) 21:25:12 Id: 229563 No. 11826
>>11780 >>11779 Thanks. Yeah, putting in an invisible 'explicitly required in search to appear' is a nice simple solution to the problem I had not thought of before. I have usually been thinking of 'sfw' modes and so on that would try to blacklist or whitelist according to complicated rules. The complicated job is making multiple local file services. There would be a bunch of new verbs added to the program like moving files from one local domain to another, and formerly fairly simple ideas like deleting and undeleting would get an order more complicated. The reason why I generally favour the latter over the former is all the edge cases. My running assumption in these cases is that you have a normie looking over your shoulder, so you don't want any nasty surprises. There are many places where whitelists and blacklists can go wrong, from false-positive tags to autocomplete suggestions (which are too complicated to filter), or even something like the 'recent tags' column in manage tags. If I have a strict database split between Content A and B, I can enforce a hard division on the caches that back up all these things, but if instead it is all mixed together, it is whack-a-mole.
hydrus_dev Board Owner 03/09/2019 (Sat) 21:30:05 Id: 229563 No. 11827
>>11783 Thanks. I see one of those definitions is unhappy with UPnP. Feel free to delete the three upnpc executables from install_dir/bin–they aren't needed to run the program, only if you want to explictly tell some hydrus service to keep a NAT port open for itself. They are just copies of the exes available here: http://miniupnp.free.fr/ The others, I suspect, are as I said before. I haven't had a true positive here yet, but I'd appreciate any future information you have here.
hydrus_dev Board Owner 03/09/2019 (Sat) 21:33:07 Id: 229563 No. 11828
>>11785 Thank you, I really appreciate this recommendation and your thoughts about it. I will definitely revisit this when work on hydrus multi-page support starts.
hydrus_dev Board Owner 03/09/2019 (Sat) 21:36:28 Id: 229563 No. 11829
>>11789 Thank you for this report. My dark mode is a bit of a fudge–I only have control over the colour of my custom objects, whereas everything else (like window borders and backgrounds and button colours) is supposed to be inherited from your OS's current system colours. If you have a window manager that changes your OS colours, hydrus's dark mode switching can be helpful to make sure that my custom objects also change with it. There's a very speculative long-term plan to move from wxPython to Qt. If this ever happens, theming support will get significantly better, but until then it is all duct-tape, I am afraid.
hydrus_dev Board Owner 03/09/2019 (Sat) 21:39:38 Id: 229563 No. 11830
>>11793 Thank you for this report. Are the updates processing during idle time or shutdown? If you have problems during idle time, can you change your settings under options->maintenance and processing to be shutdown-only, and, say, only 5 minutes? If that relatively small amount of work can go through, can a bit more, like 15 mins? Is there a significantly 'cleanup' lag, like 5 mins of work takes 3 mins to commit?
hydrus_dev Board Owner 03/09/2019 (Sat) 21:45:25 Id: 229563 No. 11831
>>11810 Thank you. That is very strange. Here's SQLite's and Python's info on this: https://sqlite.org/tempfiles.html#temporary_file_storage_locations https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir So, for some reason, a different folder was not permitting a program launched from it to access any system temp location. Windows has simpler temp structure than Linux, so perhaps this really just meant stopping you from accessing the one under AppData/Local/Temp. Could somehow that directory (or say the client.exe) have got a compatibility mode or some other protected state applied to it? I am afraid this is way beyond my expertise.
Index Catalog Archive Top Reply
1
Forms
Delete
Report
Quick Reply