I have this one.

>>11545 hot!!!

>>11545 Wait… If that's supposed to be a bee giving honey shouldn't it be coming out of her butt? That's how real bees do it

Also, computer bug. Getting this error when trying to commit tags or anything related to the PTR, including refreshing the account. >>11553 Wrong. >https://honeybee.org.au/education/wonderful-world-of-honey/how-bees-make-honey/ >When her nectar “sacs” are full, the honeybee returns to the hive. Nectar is delivered to one of the indoor bees and is then passed mouth-to-mouth from bee to bee until its moisture content is reduced from about 70% to 20%. This changes the nectar into honey. Sometimes the nectar is stored at once in cells in the honeycomb before the mouth-to-mouth working because some evaporation is caused by the 32.5°C temperature inside the hive. That's really fucking hot in a way.

>>11555 Thank you for this report. I got one of these on Thursday, but it cleared up after a refresh account call. I am not sure if it is related to me rebooting my server on Wed to update it, or if it is related to some recent power cuts I had that maybe knocked my network about for a few 60s-segments that maybe my client just happened to try something on, or something else. This sounds stupid, but if you try refresh account again, does it work now? I've had this stuff fix itself over time before. Otherwise, if you check your client log under install_dir/db, is there any traceback related to the login error? You can prob search the log for "public tag repository" to help find anything relevant. And if not anything good there, what happens if you try a 'refresh accounts' with help->debug->report modes->network report mode on? It should dump some popup messages and eventually write it all to your log again. What requests is it making to my server, and what responses is it logging, at all?

Having trouble closing any page of pages; I don't use this feature often so I don't know if it's new on latest release or what AttributeError 'PagesNotebook' object has no attribute 'TestAbleToClose' File "include\ClientGUIMenus.py", line 156, in event_callable callable( *args, **kwargs ) File "include\ClientGUIPages.py", line 907, in _ClosePage page.TestAbleToClose()

>>11526 >Do you have, say, more than 100 pages open? I've never had anything close to even 50 pages open because I'd find myself overwhelmed by it. I do however have a good number of downloads from galleries, I download artists I like entire gallery and then sort through it so that could easily be the cause of the huge number of pending tags. I have not tried page of pages, but since it's been a week on the new version (still on 338) I haven't run into the bug since then and I think I would have run into it again by now. I believe it's been fixed from moving to wx4.0.

Clearing the deleted mappings record doesn't seem to be doing anything. The deleted tags still show up in the tags manager.

>>11569 >This sounds stupid, but if you try refresh account again, does it work now? Refreshing it just tells me to refresh my account. >Otherwise, if you check your client log under install_dir/db, is there any traceback related to the login error? Clicking show traceback on the error(it also shows on the bottom right) gives me this. CancelledException
Download cancelled: This hydrus service (hydrus service: public tag repository) recently failed to log in. Please hit its 'refresh account' under 'review services' and try again.
Traceback (most recent call last):
File "include\ClientGUIPanels.py", line 846, in do_it
self._service.SyncAccount( force = True )
File "include\ClientServices.py", line 993, in SyncAccount
response = self.Request( HC.GET, 'account' )
File "include\ClientServices.py", line 876, in Request
network_job.WaitUntilDone()
File "include\ClientNetworkingJobs.py", line 1174, in WaitUntilDone
raise HydrusExceptions.CancelledException( message )
include.HydrusExceptions.CancelledException: Download cancelled: This hydrus service (hydrus service: public tag repository) recently failed to log in. Please hit its 'refresh account' under 'review services' and try again. >And if not anything good there, what happens if you try a 'refresh accounts' with help->debug->report modes->network report mode on? Nothing changes except pic related shows.

I restarted my client because the usable area turned blank (though I could still see my tabs and they highlight blue when I hover over them; didn't try clicking this time since this happens every so often) .. and on restart it said "OSError" ; "File not found!" OSError
File not found!
Traceback (most recent call last):
File "include\HydrusThreading.py", line 342, in run
callable( *args, **kwargs )
File "include\ClientGUIDialogs.py", line 1021, in THREADParseImportablePaths
mime = HydrusFileHandling.GetMime( path )
File "include\HydrusFileHandling.py", line 315, in GetMime
if HydrusVideoHandling.HasVideoStream( path ):
File "include\HydrusVideoHandling.py", line 309, in HasVideoStream
lines = GetFFMPEGInfoLines( path )
File "include\HydrusVideoHandling.py", line 132, in GetFFMPEGInfoLines
CheckFFMPEGError( lines )
File "include\HydrusVideoHandling.py", line 37, in CheckFFMPEGError
raise IOError( "File not found!" )
OSError: File not found!
If it matters, what i was doing immediately before the client area (except tabs) turning blank- which was what prompted me restarting the client- was importing an old windows folder I was too lazy to sift through manually. I never learned how to really tackle this stuff, so I was running irfanview's (image viewer) batch rename command to rename all the images to "date (ascending)" to import under a unique namespace. I let that do its thing into a new folder it created, while at the same time I was importing the original folder with original filenames to Hydrus. Hydrus said it couldn't access two images, probably because they were in use, but I couldn't be arsed to import the folder again over two images, so I just let it go. I assumed that was what this error was about but I don't know. I'm on version 339 (latest version as of the time of submitting this).

>>11592 Actually the two images Hydrus didn't have access to on importing this folder, that appears to be independent of my running an irfanview batch rename command alongside importing to Hydrus. The folder irfanview's batch rename thing produced has two fewer images than the original folder. So two of them must just be broken. In short all my text after pasting the error was probably irrelevant and independent of the error.

>>11582 false alarm still get the bug. showed up again after heavy downloading today after many crashes almost immediately after restarting, I've since cleared the 800k pending tags yesterday so that can be scratched off the list as the cause.

Since 339 i am getting an error when uploading data to repositories on the linux client, now with 340 it still happens. first i got this C++ assertion "win" failed at /home/vagrant/wxPython-4.0.4/ext/wxWidgets/src/gtk/menu.cpp(84) in DoCommonMenuCallbackCode(): event for a menu without associated window? File "site-packages/wx/core.py", line 3259, in <lambda> File "include/ClientGUICommon.py", line 3351, in WXDoIt self._func( *self._args, **self._kwargs ) File "include/ClientGUI.py", line 4940, in RefreshMenu old_menu = self._menubar.Remove( old_menu_index ) then the pending menu get gray and vanish and i get this Menu not found! Traceback (most recent call last): File "include/HydrusPubSub.py", line 141, in Process callable( *args, **kwargs ) File "include/ClientGUI.py", line 4756, in NotifyNewPending self._DirtyMenu( 'pending' ) File "include/ClientGUI.py", line 1164, in _DirtyMenu menu_index = self._FindMenuBarIndex( label, name ) File "include/ClientGUI.py", line 1236, in _FindMenuBarIndex raise HydrusExceptions.DataMissing( 'Menu not found!' ) include.HydrusExceptions.DataMissing: Menu not found! until the upload is complete

Found some exporting fuckery. I have some files I want to tag with pages 1 to 18. First I tried exporting using (page) and it got really fucking weird. Page 1 got a blank filename, pages 2 - 18 got a page, except it was the number of the previous one. Using page led to the same result, except instead of a blank name, 1 was just named "page.jpg", whereas page 2 was then semi-corretly named "page 2.jpg". I tried just saying fuck it and using {tags} which worked for the pages, but then the tag order fucked up on some files for some reason, all in the same pattern.

>>11620 >whereas page 2 was then semi-corretly named "page 2.jpg". *whereas page 2 was then semi-corretly named "page 1.jpg".

>>11555 >>11569 >>11589 Also, I'm possibly retarded. I went to check on the services again just now and it was paused, I unpaused it and it worked. Either I accidentally paused it, or something fucked up and it paused itsel,f either way it's working now.

Not sure if this happen on new versions but right now i'm using Hydrus 334 on windows 7. When i load a lot of images it doesn’t load every image on my archive, there is always missing a few images to load (10,12,5,6,etc)

>>11620 Use "[page]" not "(page)"

I have a problem on linux where the popup window with all the system searches like in the pic in >>11631 doesn't show up. Same with the popups in the media viewer. V.338

>>11589 >>11555 >>11629 Thank you. I am sorry for the inconvenience here. When I wrote the new network engine, I had to fold in the old hydrus account checking stuff into the new login system, and it was a little messy. I suspect there are some dead-end error states in there somewhere that I am not catching properly. In your case, perhaps after several days some timer clicked over and it realised that the login status really was 'unknown' instead of 'failed' or something, so it was able to try one more time in the background, failed cleanly, and then set the service paused in a 'true' unknown state. You were then able to honestly retry and it all sorted itself out. Please let me know if this happens again or you find out anything new.

>>11581 Hey, I am sorry for this bug. It was a mistake I made in 339. I have fixed it for the most recent 340 and written a new UI test to stop this stupid stuff happening again.

>>11582 >>11607 Ok, thank you. I will keep working on this. I rejiggered some menubar stuff this past week for 340, so maybe if >>11607 was on 339, it will now be improved, but I am not sure. I don't think this is menubar but instead something else overflowing. If you are also getting crashes, I recommend you scale back your client in any other easy ways. If you have a lot of thumbnails sitting there waiting to be processed, give them a local tag for like 'process later' and close the tab. Make your client leaner is probably the best bet here for now.

>>11585 Was this through the advanced content update dialog, where you say like '(copy) (all mappings) from (x) to (y)'? That had a counting bug that meant it would not perform enough update rows (often it would do 0) in many cases. This is fixed in 340, so if this is so, please try again in that and let me know if it works.

>>11592 >>11597 Thank you for this report. A couple of users have got boshed by this. Please check the chain at >>11594 . I am interested in answers to all the questions in that.

>>11615 Thank you. I will revisit this code this week. I am not sure what is happening here, but I think it is OS- or window manager-specific–is this on Linux, as running from source?

>>11620 >>11624 Hey, I think >>11633 has it. Try with brackets, [ and ], and not parentheses, ( and ). With parentheses, it is looking for 'page' rather than the x in 'page:x', and since the files do not have 'page', the resultant filename is the empty string. Each file gets the same empty string, so my 'duplicate-filename de-deduper' is putting ' (n+1)' on the end of each one.

>>11631 Thank you for this report. I do not recommend you try to load that many files at once! I recommend you set a limit under options->speed and memory. 10k is the default. That said, this bad number may be due to some file miscounting in the past. Please try running help->debug->data actions->clear db service info cache and then restart the client (you might like to note down your exact numbers here beforehand). Afterwards, have the inbox/archive/everything numbers changed? What about the 'all local files' and 'my files' file domains under services->review services panel? 'all local files' should have the same number of files as the sum of 'my files', 'repository updates', and 'trash' domains.

>>11634 I have had a lot of trouble getting those floating windows to work right in non-Windows, so by default I have them 'embed', like in the manage tags dialog. Please check the 'always embed autocomplete …' option under options->gui and restart the client–does that work better? There is an ugly BUGFIX option on the same page to make the hover windows on the media viewer always show. It isn't nice, but it works better on the Linux window managers that do not like my code. Did you by any chance migrate from Windows?

>>11652 not running from source, i am running the the executable. To be more specific, The problems happens if i rover the mouse over the pending menu after i put one to sync. I am runing ubuntu 18.04 with KDE plasma, if i go over GNOME the problem is worst, most of the menus get greyed out

>>11655 Cool thanks. Yes I did switch from windows. >>11275 Followup on this. While the gui doesn't freeze at random anymore I noticed it will always freeze irrecoverably when I export files.

pressing "restore from a database backup" on client ver 340, newly install debian 9 causes the client to crash, output in terminal is "Aborted."

>>11667 I've got 341 to do some more intelligent swapping during fast update events here. It may fix your issue, or it might move it somewhere else. Please keep me updated once you have tried 341, and I'll give it another go if needed.

>>11679 Thank you for this report. Is this when exporting via the export files dialog (i.e. from the thumbnail right-click menu), or drag and drop exporting? If you haven't tried one way, can you try it and let me know if that works? Does it freeze at the beginning of the action, like as soon as you click, or at the 'end', when the copy is completed and control passes back to the program? Do the files ultimately get exported or not?

>>11686 Thank you for this report. I will revisit this code. It was always a jank way of doing it and something I did not maintain, so the solution here may be to remove it and write a better guide on how to do it manually.

>>11707 It's from the export files dialog. I need to export the tags with the files so drag and drop isn't really an option. I did test the drag and drop and it seems to work just fine. The freeze happens while running the export i.e. at 7000/30000 files exported. Luckily it will successfully export all the files, it's just I have to kill the client whenever I think it's done.

>>11706 ok m8, you are doing gods work, i will report wfter i try 341

Since I updated from 338 to 340 I've been having a problem with file discovery crashing my client. I don't get an exception message, no logs, just a message in my terminal: [1] 19988 segmentation fault (core dumped) hydrus-client -d ~/mnt/data/hydrus/hydrus/ So if I click the discover duplicates button, my client crashes. Anything else on the page works fine (except the button to launch the filter, I don't have any potential duplicates and trying to find them crashes my client, so maybe it does, maybe it doesn't.) I figure I somehow fucked up my duplicate processing, unfortunately for me I overwrote my previous backup with a 340 one because I'm retarded. Even more unfortunately for me, on a fresh db this doesn't happen which probably means I fucked up. Also I'm running 341 on Linux. Any suggestions for unfucking duplicate discovery?

Version 340, downloading from SVTFOEbooru Source: http://svtfoe.booru.org/index.php?id=23786&page=post&s=view <html>… (Copy note to see full error) Traceback (most recent call last): File "include\ClientImportFileSeeds.py", line 1071, in WorkOnURL network_job.WaitUntilDone() File "include\ClientNetworkingJobs.py", line 1133, in WaitUntilDone raise self._error_exception include.HydrusExceptions.ServerException: <html> <head><title>503 Service Temporarily Unavailable</title></head> <body bgcolor="white"> <center><h1>503 Service Temporarily Unavailable</h1></center> <hr><center>nginx</center> </body> </html>

I was trying to make collections out of my newly downloaded files, it was fucking up and turns out it had downloaded an mp3 and that's what was fucking up. Looks like the program is set to accept mp3 files but not programmed to collect mp3 files properly or something. Here's the file: yiff.party/patreon_data/20184639/Whatevs_01.mp3 TypeError
'>' not supported between instances of 'NoneType' and 'int'
Traceback (most recent call last):
File "include\HydrusPubSub.py", line 141, in Process
callable( *args, **kwargs )
File "include\ClientGUICanvas.py", line 2585, in MediaFocusWentToExternalProgram
self._MediaFocusWentToExternalProgram()
File "include\ClientGUICanvas.py", line 1674, in _MediaFocusWentToExternalProgram
elif self._current_media.HasDuration():
File "include\ClientMedia.py", line 1702, in HasDuration
def HasDuration( self ): return self._media_result.GetDuration() is not None and self._media_result.GetNumFrames() > 1
TypeError: '>' not supported between instances of 'NoneType' and 'int'

>>11542 Still a relatively new Hydrus user. This is also my first time posting here, but I found this bug where part of the file search box sticks around on whatever page I switch to. (Here's a screenshot of it on top of a watcher page I opened). Anyone else experiencing this issue?

>>11736 Are you using binary distribution from tarball or built version from github? Pre-built hydrus is often broken on Linux because of the plethora of libraries shipped with it that might be incompatible with your system libraries.

>>11758 I use a Arch based distro so I get Hydrus from the AUR, the pkgbuild appears to compile hydrus from source. Regardless I've been running it out of pycharm for debugging purposes, so unless I have a library that's so new it's breaking stuff, I don't think that's the problem. Now as I just mentioned I've been running hydrus out of pycharm so I could debug it. Tl;dr I debugged the shit out of hydrus, eventually we get to the Write method in /hydrus/include/HydrusDB.py: I figure either line 967: job = HydrusData.JobDatabase( job_type, synchronous, action, *args, **kwargs ) is producing an incorrect job which in turn causes this: line 974: self._jobs.put( job ) to flip out. Or if we go with what gdb says (even though I can't even find a single piece of sqlite related code in the entire set of methods I followed), then sqlite is flipping out which probably means I need to manually edit a table in my database which I am quite capable of…if I knew which table to look at. So if anyone has any knowledge on where exactly the aforementioned write method is calling to sqlite, I'd love the info.

Found an image that was incorrectly detected as a decompression bomb by the importer, here is the original link https://yiff.party/patreon_data/24364306/3223347/01-30.png and here is a mirror https://anonfile.com/71ufV4v0b8/01-30_png Fair warning: pic is 13 MB of furry porn.

>>11788 Well I figured it out. sqlite 3.27.1 can't execute the following statement: SELECT phash_id, phash, radius, inner_id, outer_id FROM shape_perceptual_hashes NATURAL JOIN shape_vptree WHERE phash_id IN (123778, 113226, 135502, 94192); Every other version can. Guess which version of sqlite my distro upgraded to when I performed the system update that upgraded hydrus to 340?

>>11706 ok, i tried to replicate the error. now the menu stay greyed out and the error that appear is this 2019/03/06 00:45:02: wxAssertionError C++ assertion "win" failed at /home/vagrant/wxPython-4.0.4/ext/wxWidgets/src/gtk/menu.cpp(84) in DoCommonMenuCallbackCode(): event for a menu without associated window? File "site-packages/wx/core.py", line 3259, in <lambda> File "include/ClientGUICommon.py", line 3351, in WXDoIt self._func( *self._args, **self._kwargs ) File "include/ClientGUI.py", line 4959, in RefreshMenu self._menubar.Replace( old_menu_index, menu, label ) sometimes the menu comes backup, sometimes not. but i can state that it is more stable, specially on the gallery download page and on my gallery(+- 18k images)

I am sorry lads, I had a mixed couple of weeks so I am late getting to all this. I have a new tablet for hydrus that I hope will help me keep on top of this better when I am away from my dev machine, even if I put out shittier shorter posts at times. >>11714 Thank you for this follow-up. There is some really shit ui code in the client that can lead to hangs during big operations. I have a long-term plan to fix it all, but for now I'll see if I can make large export files jobs less susceptible to the problem. The 'solution' for now is just to wait it out. It will get there in the end.

>>11736 >>11758 This should be finally fixed in the current release. It was an odd low-level issue related to the order in which I was doing some CPU stuff and database iteration for distance>0 searches, and specific to some flavours of Linux. I am sorry for the inconvenience. Please let me know if you still have it.

>>11741 Hey, 503 is a server-side error. I presume booru.org was overloaded at that time. It works for me now. Please try the download again and let me know if you still have problems.

>>11747 Thank you for this report. I have been doing some work here recently, so it might be fixed already, but I will check it out again.

>>11752 Thank you for this report. This is odd. It happened a bit a while ago, but I thought I had it cleared up. If you are on the search page and click on the thumbnail area, does it dissappear as expected then, or does it always stay up no matter where the focus is? Which version of Windows are you on? This is not an ideal solution, but you can make the dropdown always 'embed' into the panel under options->gui->always embed autocomplete….

>>11788 >>11797 Continuing from >>11899 . Yeah, this way the problem query, which should now be fixed. If you are interested in the details, the problem seems to have been the subsequent HydrusData.Get64BitHammingDistance call being interleaved with the row fetching. This Hamming function does some mickey-mouse CPU stuff to count 1s from an XOR, and when I did it during the SQLite query iterating its results, it crashed in some flavours of Linux. When I instead complete the query and then do the phash distance searching, all in smaller batches, it is ok. I am not expert enough in this stuff to know what the problem was, but I presume SQLite was still hanging on to a buffer pointer or something of the phash bytes, to be used in the next row fetch iteration, and when python then did either the struct or n & 0xF0F0F0F0F0F0F0F0 bullshit on it and SQLite returned to it, it caused a memory access violation. Again, seems to be fixed now. I am sorry for the mess–this was a pain to figure out because I could not replicate it on my test machine. Let me know if you still have trouble.

>>11795 Thank you for this report. Yeah, this seems on the lower end of bombs. I don't really see any memory explosion as it loads, even if it is a big file in itself. I recommend you turn on 'allow decomp bombs' under file import options and reattempt the import. The decompbomb test isn't actually mine–it is something internal to one of my image loading libraries. I think I can bump up the limit–especially since python 3 is better at handling memory all around. We'll see how that works out.

>>11800 Thank you. I will keep working on this.

>>11899 >>11903 I'm also having the same issue with it crashing on >0 duplicate searching on Arch Linux. I'm using the latest v343. Was it supposed to already be fixed or do you have it queued up for the v344 release?

>>11925 F U G G U G G Yeah, this should be fixed in 343, which means there is still more to go here. Ok, please keep the search at 0 distance for now. I will see if I can do something about this this week. Maybe make a super-safe debug mode or similar for you to test out.

>>11934 For some reason it's working now that I restarted my computer. It's weird because it's not like I was still running an older version. I turned on my computer, updated all my packages, then opened hydrus and encountered the error. I even made sure to double check the version I was running before I posted. Today I restarted both of my computers for a kernel update and it's working fine on both of them.

Latest version on Linux is constantly crashing. And doesn't show the tags on the search menu. I'm on Manjaro. And one question: when I asked to export some files to a folder, only a few are actually are. Like there is 138 files under "X" tag and only half of them are exported to the folder. Am I missing something?

>>11943 >And doesn't show the tags on the search menu I have figured it out what's happening here. The search box is squashed on a tiny box. When I resize the client window to some ridiculous size I can actually see it.

>>11941 Thanks. Please keep me updated–there is just the smallest off-chance this was a crash due to a different problem that either happened to occur at the time you clicked, or maybe the click fired off some garbage collection that triggered it otherwise. Or perhaps the kernel update actually fixed some deeper memory-management issue! Anyway, let me know how you get on from now on.

>>11943 >>11944 Thank you for this report. Are you running from source or the built release I put out? If you ran 342, the previous week's release, was that ok for you? I see you have the hidden button borders as well. Is that new as well, or have you had this before? Unfortunately, the frozen build I release is made on Ubuntu 16.04 and is not very compatible with certain flavours of Linux and window manager. If you are running from that, you may like to look into running from source: https://hydrusnetwork.github.io/hydrus/help/running_from_source.html A bunch of UI weirdness is generally fixed, and stability improves as well. As for file exporting, I am not sure. Is this an 'export folder'? Is there any chance the filename phrase you set may result in duplicates? Is it possible it is writing two different files to the same 'blue eyes.jpg' or whatever filename?

>>11949 >Are you running from source or the built release I put out? Built release. To be honest, I got no crashes after I made that post. Maybe I was doing something stupid and causing it to crash. >If you ran 342, the previous week's release, was that ok for you? No, this is the first time I'm using hydrus on Linux. >I see you have the hidden button borders as well. Is that new as well, or have you had this before? I haven't change anything, maybe is just manjaro theming. I had some issues kinda like this one on Firefox, maybe it has something to do with it. https://wiki.archlinux.org/index.php/Firefox/Tweaks#Unreadable_input_fields_with_dark_GTK+_themes >Is there any chance the filename phrase you set may result in duplicates? Yeah, I think that was the problem. I added the {hash} on the filename options and I think it fixed. Thanks for the help.

>>11957 >No, this is the first time I'm using hydrus on Linux. On options check the "Always autoembed dropdown results".

>>11960 This one? There's no checkbox for me.

>>11957 >>11960 >>11961 Yeah, that's more of the hidden ui elements. The checkbox is there, afaik, but just not rendering right. The embed autocomplete is now forced always-on for non-windows, you don't have to worry about it. My guess it is squashed because your WM isn't reading/storing my SetInitialSize etc… calls right. Thanks for that info about Firefox sometimes doing similar with themes. I do not really set any colours for my controls, and I am not sure I even can set colours for different parts of checkboxes–I think wxPython just uses the system defaults. This borderless stuff seems to be related to my Ubuntu .so files clashing with whatever your system wants. I believe the best path is to try running from source (or maybe try switching theme/WM for hydrus, if that is possible?). Let me know if you figure out a particular behaviour seems to cause a crash, or if otherwise avoiding something seems to eliminate it completely.

Here's a couple of webms that hydrus fails to add into itself

And here's the last one, alongside an error that appeared with it. Error counting number of frames!… (Copy note to see full error) Traceback (most recent call last): File "include/HydrusVideoHandling.py", line 507, in ParseFFMPEGNumFramesManually l = frame_lines[-1] # there will be several of these, counting up as the file renders. we hence want the final one IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "include/ClientImportFileSeeds.py", line 831, in ImportPath self.Import( temp_path, file_import_options ) File "include/ClientImportFileSeeds.py", line 788, in Import ( status, hash, note ) = HG.client_controller.client_files_manager.ImportFile( file_import_job ) File "include/ClientCaches.py", line 1098, in ImportFile file_import_job.GenerateInfo() File "include/ClientImportFileSeeds.py", line 283, in GenerateInfo self._file_info = HydrusFileHandling.GetFileInfo( self._temp_path, mime ) File "include/HydrusFileHandling.py", line 234, in GetFileInfo ( ( width, height ), duration, num_frames ) = HydrusVideoHandling.GetFFMPEGVideoProperties( path ) File "include/HydrusVideoHandling.py", line 211, in GetFFMPEGVideoProperties num_frames = ParseFFMPEGNumFramesManually( lines ) File "include/HydrusVideoHandling.py", line 520, in ParseFFMPEGNumFramesManually raise HydrusExceptions.MimeException( 'Error counting number of frames!' ) include.HydrusExceptions.MimeException: Error counting number of frames!

>>11907 >I recommend you turn on 'allow decomp bombs' under file import options and reattempt the import. That was how I worked around the issue, yes, I reported mainly because it was pretty weird behaviour.

>>11974 >>11976 >>11977 >>11978 Thank you, these are great. I will try to check these out this week.

>>11651 Dude, I'm sorry. I am dummy depressed all the time and I feel stupid even making this post. I didn't reply to you for now over a month cause I was kinda panicking over your reply. I don't use Linux, and I didn't get all the stuff you mentioned, and I wasn't doing so well anyway. I figured maybe later it'd be easier. But I still can't do it. It's hard. That doesn't justify anything but I don't know. I'm on version 341 now, and I was about to install version 344, which is the latest right now. I am on Windows 7 Ultimate. I don't really fetishize ignorance so I don't really need you to say "it's ok to not provide context". I'm not really saying anything for any purpose

I'm having an issue with the API. When I GET /get_files/file_metadata with the parameters file_id and only_return_identifiers, I get this error: Traceback (most recent call last):
File "threading.py", line 916, in _bootstrap_inner

File "threading.py", line 864, in run

File "site-packages\twisted\_threads\_threadworker.py", line 46, in work

File "site-packages\twisted\_threads\_team.py", line 190, in doWork

--- <exception caught here> ---
File "site-packages\twisted\python\threadpool.py", line 250, in inContext

File "site-packages\twisted\python\threadpool.py", line 266, in <lambda>

File "site-packages\twisted\python\context.py", line 122, in callWithContext

File "site-packages\twisted\python\context.py", line 85, in callWithContext

File "include\ClientLocalServerResources.py", line 1255, in _threadDoGETJob
file_ids_to_hashes = HG.client_controller.Read( 'hash_ids_to_hashes', file_ids = file_ids )
File "include\HydrusController.py", line 588, in Read
return self._Read( action, *args, **kwargs )
File "include\HydrusController.py", line 183, in _Read
result = self.db.Read( action, *args, **kwargs )
File "include\HydrusDB.py", line 931, in Read
return job.GetResult()
File "include\HydrusData.py", line 1577, in GetResult
raise e
include.HydrusExceptions.DBException: TypeError: _GetHashIdsToHashes() got an unexpected keyword argument 'file_ids'

Database Traceback (most recent call last):
File "include\HydrusDB.py", line 557, in _ProcessJob
result = self._Read( action, *args, **kwargs )
File "include\ClientDB.py", line 9265, in _Read
elif action == 'hash_ids_to_hashes': result = self._GetHashIdsToHashes( *args, **kwargs )
TypeError: _GetHashIdsToHashes() got an unexpected keyword argument 'file_ids'
The request returns OK if I get rid of the only_return_identifiers parameter or if I use hashes instead.

>>11983 No worries m8. I think I read your error report wrong in the first place and assumed you were having 'finding ffmpeg executable' problems, whereas it looks actually like your ffmpeg is ok but could not find the file hydrus was pointing it to. If you had a batch rename thing going on at the time, this may well have been related–maybe hydrus queued the file up to be imported, and then irfanview renamed it before hydrus got around to trying the import. Or maybe hydrus had a hiccup when trying to do some temporary storage stuff and lost track of the file mid-import for other reasons. If you are still seeing this error over and over, then I am happy to chase it up. But if this was a one-time thing that may have been because of two programs accessing the same space at the same time, no need to worry about it. You are good to update to 344. Let me know if you run into any more trouble. Sounds like you are having a shit time. I understand this completely. If you can get a few bucks together, please consider getting this book: https://www.amazon.com/Feeling-Good-Handbook-David-Burns/dp/0452281326/ . It saved my life about ten years ago. You can find it in any big corporate bookstore as well, just say you are buying it for a friend.

>>11984 Thank you for this report. I fucked up the variable name here, and this is now fixed for 345. I have a background job already to improve my testing to catch this shit, which has been a blind spot previously.

I updated to 346 earlier today, but I seem to be having an issue with thumbnails. Today (after updating) I ran a couple concurrent searches for the same artist on different sites, using the gallery downloader, and, later, one search using a simple downloader (for yiff.party). I'm pretty sure everything was working fine before I started the simple downloader, but now when I check on one of the gallery download pages, many of the thumbnails are missing. The file count at the bottom also appears to be incorrect - stating 111 files, but when ctrl+A'd, says 149 selected. Refreshing the page causes the thumbnails to load (and the order changes too, strangely (actually, this happens on all of my gallery downloader pages)) Also, for some reason, if I F5 on certain other pages, it fixes the thumbs on the gallery downloader page too. Oh, opening the options menu fixes them too, apparently.

I'm not sure if this is a bug with Hydrus or what is going on. I downloaded the standalone Flash player so I could view swf through Hydrus' open externally menu. However, when I do that the program opens like it's hidden. I can only hear sound and find the process in the task manager, but I can't find a window. If I run the launch path (X:\Hydrus\flashplayer_32_sa.exe "%path%") manually it works fine.

Can you fix this? I know the program is made with a resolution of 1080p on mind but please can you make it a bit more friendly with lower resolutions? Not a bug but a related problem it's the tags dialog, can you make it scroll automatic when you scroll on the tags? I have to scroll manually if I want to see the system tags or picture's tag.

>>12087 Fuck, that's crazy. Thank you for this report. If you can repeat this now, can you try clicking on just one of those blank spaces? What info does it say in the status bar then, like 0 selected out of 149? What happens if you right-click on that ghost thumb, does it give any info or an error? What happens if you Ctrl+A and then try to drag the selection up to the greyspace tab area up top (i.e. moving it to a brand new page)? What's the 'file import options' 's 'presentation' options for that downloader? It looks like it is trying to present all 149 successful files, but some subset were maybe deleted or otherwise file-domain-invalid and so can't be sourced/counted somehow. Hitting F5 on a page without tag search (typically any import page) will force a resort according to what the page has set (rather than the default there, which is downloader-queue order), so looks like 'oldest first time imported' in that screen.

>>12096 Thank you for this report. That's odd–I assume if you double-click a swf in explorer, the standalone projector launches ok? The default windows 'open externally' call is very simple on my end–I literally just say "Hey Windows, open this path m8: ( path )", so I presume what you are seeing is an artifact of an odd Windows-side 'open with' setup. Perhaps your environment is unusual in some way, so if you would like to pursue this further, please try help->debug->report modes->subprocess report mode and then launch with your "%path%" command. It will popup a bit about the launch environment. If there is anything interesting here, please post it (and feel free to scrub out any personal info), and it might help here. There's an outside chance that maybe the hydrus exe is running as one user but the flash projector is owned otherwise, or there are some funny permissions going on here. EDIT: I didn't read your post right. Did you try setting up that %path% in hydrus's options->files and trash, or is that something you were doing from the command line? If you haven't tried it for hydrus, that may 'open externally' fine.

>>12101 Thank you for this report. I am afraid I am a little limited by my ui library here. I use wxPython for its ease of use (and I use it in a basic way), but that also offers a little less customisation as a result–it generally pulls from your system defaults for things like how to draw buttons and font sizes and so on, and I do not generally set those sizes myself. That said, I can improve the layout sizers in the left-side management panel for smaller screens, and I will see if I can make the system preds either fit better or scroll to view when you interact with it. If you want the left panel a bit wider, the program should remember the current page's current width when you close the program, so you might like to make it wider and go down to five thumbnails wide on your main thumbgrid. Also, if you want the extra verticality and are willing to sacrifice it, you can hide the bottom-left preview window under options->gui.

>>12110 I had it set up in options->files and trash, yeah. But I removed it from there and set it up as an "open with" association in windows and now it works fine. There appears to be an issue with setting a manual launch path. The path was "X:\Hydrus\flashplayer_32_sa.exe "%path%"", if that helps. The program opens and plays the swf file, but the window is hidden, and has to be closed in task manager.

>>12109 Okay, I'm just going to dump *everything* I've noticed/tested here - sorry for the long/rambling post ahead. So, it seems like it's probably something to do with duplicate images. The missing thumbnails are duplicates of thumbs/images that are already there, and it looks like one of the two images is randomly picked to be displayed. As of now, there's always the same number of missing thumbs: 31. Regarding clicking on one of those blank spaces: it reveals a thumbnail, and the status bar behaves as expected for selecting an image. Same with right clicking - no error messages. Clicking on the blank spaces always reveals the thumbnail, but sometimes it reveals it half "greyed out" (pic 1). Subsequent clicks off/on the thumb progressively brighten the image up to its normal value (pic 2). In a similar vein to that, clicking on one of the thumbs which *is* loaded (but has a missing counterpart), results in the "highlight" not fully disappearing when deselected (pic 3 - only Misty in the bottom right is actually selected). Also, *some* of the time, clicking on one of the thumbs that is already loaded will reveal its ghost duplicate counterpart. (see pic 4 - I clicked on the bottom right image, pic 5 was the result) >What happens if you Ctrl+A and then try to drag the selection up to the greyspace tab area up top? It creates a new page, but the new page exhibits the exact same behaviour as the Gallery Download page in question. The file count is lower than expected, and a bunch of thumbs are missing. The page title says files(92), and after a Ctrl+A, the statusbar says 92 files - 123 selected (I've done some archive/deleting since my first post). I noticed that immediately after Ctrl+A dragging the images into a new page, I'm left with the offending 31 images (no dupes, no ghosts) on their own in the "downloads" page. >What's the 'file import options' 's 'presentation' options for that downloader? The buggy behaviour is present for both >show all importers' presented files >show all importers' files but not for the last option: >show all importers' new files *However* I can't say for 100% sure that this was the case originally - as I said, I've deleted some files in the mean time. >trying to present all 149 successful files, but some subset were maybe deleted or otherwise file-domain-invalid At the time, I actually hadn't begun deleting anything yet. Also: Pressing Ctrl+A reveals a few of the missing thumbs, but then deselecting (clicking an empty spot) reveals many more (but not all). Lastly, I doubt it's related, but I randomly checked the known URLs of one of the seemingly non-bugged images on that page, and it displayed two urls on one line: https://gelbooru.com/index.php?id=2551390&page=post&s=view https://gelbooru.com/リタを触手で.avi | ranken [pixiv] http:/www.pixiv.net/member_illust.php?illust_id=48114073&mode=medium https://img2.gelbooru.com//images/dd/c6/ddc66a0d2ebb5cc3299a676d23dce121.png I hope this helps. I tried to be as clear as possible, but some of this shit is just weird. Let me know if I can try anything else, or if you need any screenshots to visualize wtf I'm talking about.

Had disk io errors, ran a chdsk and it fixed some things, now I have >sqlite3.OperationalError: table local_hashes already exists im completely fine with reimporting all my images, but is there a way to do that without losing my processed tags?

>>11908 hi, it's me again. got on 345 and for a long time i couldnt replicate the error. and got harder on 346, but i got to(dont know how tho). The erro message was this: 2019/04/08 13:03:48: Uncaught exception: 2019/04/08 13:03:48: wxAssertionError C++ assertion "win" failed at /home/vagrant/wxPython-4.0.4/ext/wxWidgets/src/gtk/menu.cpp(84) in DoCommonMenuCallbackCode(): event for a menu without associated window? File "site-packages/wx/core.py", line 3259, in <lambda> File "include/ClientGUICommon.py", line 3351, in WXDoIt self._func( *self._args, **self._kwargs ) File "include/ClientGUI.py", line 4973, in RefreshMenu self._menubar.Replace( old_menu_index, menu, label )

>>12117 Thank you, I will have a closer look at this next week.

>>12118 Thank you, this is excellent. I am not totally sure what is going on here, but there are issues in 346 with drawing certain thumbnails and making files appear even when they are not in the appropriate file domain. Neither of these issues is precisely what you are seeing here, although that 'thumb ghosts on click' makes me think the animation is a bit busted, perhaps as a result of it not being able to locate files due to dupes or whatever. Whatever the case, 347 should be better in several ways here. Please give it a try when convenient and let me know if the situation is any better or exactly the same. That bad 'known url' stuff is annoying. Some boorus prepend extra metadata in the known url area (in this case "リタを触手で.avi | ranken [pixiv] "), and it is parsing wrong. I'll see if I can auto-detect this stuff better and cut off everything up to any http.

>>12120 I cannot say with confidence without more information, but could one of your .db files been deleted or moved as you fixed things? There should be: client.db client.caches.db client.mappings.db client.master.db In install_dir/db. Are your three auxiliary dbs fairly large and old, but the client.db say only 200KB and brand new? Could the original client.db have been moved somewhere, and then the client was booted with it missing? That error looks like the client booted without a client.db, made a fresh empty one, and then ran into a roadblock when it discovered client.master.db already existed. If this is exactly what happened (it may help to compare the creation date for your four files to see if one was only made recently), your solution will be to move the stub db file (likely client.db) to a new safe location and then move the original client.db back to the database folder so all four files are together again. If you are absolutely certain you did not move your client.db, I believe it was deleted in the drive damage or the chkdsk. Please double-check your recycle bin, just to make sure. if you cannot recover a good client.db, I am afraid your client is lost and you will have to start again. The short steps for this are: Move current db folder contents and client_files folder somewhere safe. Boot a fresh client. Import contents of client_files's 256 'fxx' folders in batches. I am afraid I cannot help you recover subscriptions or other core metadata, as this is all stored on client.db. But with some work we should be able to migrate your tags to a new db. If your client.db is large and old, please let me know, as a more complicated error has occurred here. Do not delete anything, and please check if you have any backups anywhere, even very old ones.

>>12123 Thank you. I will keep pushing on this. I am glad it is happening less. I still have some ideas for code cleanup and reduced replace calls, particularly during pending tags commit.

>>12141 yeah, client.db is only 4KB, though the creation time is similar to all the others unfortunately no way to get the old client.db back, i didn't take the advice to backup :( images are back, but no tags yet

hello. i wanted to ask about a small issue i am having. i launched up hydrus to find certain images, but my subscriptions started to download stuff, but it kinda made finding the image harder as it was trying to process that meanwhile processing me searching for the image, so i paused the subscriptions. but apparently, the pause subscriptions button bugged out, as the subscriptions started to NOT PAUSE when i had checked pause subscriptions, but DID PAUSE when i unchecked. i checked the files in my inbox and 4 files were missing/broken, so i ran a file integrity check. at the end of the check, it removed the 4 missing files which was good, but it also said "a file indentifier is missing! this is a serious error that likely needs a repository reset to fix! think about contacting hydrus dev!" which now brings me here. i don't know if this missing file identifier problem formed when this bug happened or is something i've had all this time. if it is the case of me having it all this time, hydrus seems to work fine even with this bug. i am on hydrus client 340, network version 18. i don't know if i should update or not. what should i do to fix this missing file indentifier?

The yiff.party parser is technically broken, but not really. Nothing about the creator json on the site changed, but any and all attachments downloaded through hydrus result in either a 403 or a 404 for me, except the links are correct, viewing them in a browser works. My uneducated guess is yiff.party is blocking whatever Hydrus uses as its user agent or something similarly simple. Not that surprising since that site is in desperate need of funding and has limited bandwidth.

>>12147 im good now, just got tags from https://cuddlebear92.github.io/Quicksync/

>>12147 >>12231 I am sorry you had a hard time with this, but I am glad you are putting things back together. I know it sucks to hear right now, but this sort of problem can be a great driver to get a good backup routine going. It happened to me about ten years ago, and has happened to more than a few hydrus users since I started doing this–you lose a load of shit and previous effort and decide to spend a bit of cash to avoid that pain again. I highly recommend a cheap external usb drive, either from a store or second-hand from a friend. Even just having a copy of the .db files–rather than including all your client_files as well–can make a real difference. If you are interested and not sure how to do it: https://hydrusnetwork.github.io/hydrus/help/getting_started_installing.html#backing_up

>>12149 Thank you for this report. Have you had any crashes or power outages recently? The file identifier error may have occured due to a recent-ish hard shutdown of the client. As a precaution, I recommend you shut down the client and check the 'help my db is broke.txt' document under install_dir/db/ as background reading. You might like to just check the integrity of your db and maybe run a chkdsk to be sure your drive is ok. Can you also check install_dir/db/client - 2019-4.log? If you scroll way down to the timestamp where this was happening (ctrl+f -> 'file identifier' will probably get you to it too), are there any error tracebacks related to the pause/unpause or missing/broken file imports? I would be interested in seeing that information, as it may better explain why subs were acting wrong here. Subs may take a few seconds to register a 'pause' event, and will typically not stop working until the current gallery page or file being worked on is done, but they shouldn't flip behaviour entirely. If this was just a blip due to a power cut or whatever, and you are confident your drive is healthy and database is not corrupt, you I think have two paths: if you keep using your client and it seems totally fine, especially if those subscription files are reimported and are fine, I think it may have fixed itself, but if you keep getting it, particularly in relation to the same set of files that are tagged on the PTR, your solution is to go services->review services->remote->tags->PTR and hit reset processing cache. Let me know how you get on!

>>12189 Yeah, it looks like they started putting secondary post attachments to a new subfolder in the URL structure, and it was being recognised and incorrectly truncated hydrus-side. I am rolling out a new url class for 348 that seems to fix it. Please let me know if you still have trouble here. It wasn't a block, but the accidental hydrus truncation was doing something like this: https://yiff.party/patreon_data/9078764/24083586/3164951/coffee-girl--SFW.jpg -> https://yiff.party/patreon_data/9078764/24083586/3164951

I've noticed that the url import isn't attaching tags to gelbooru and pixiv links anymore. Is there a setting I'm missing? I am on version 347 now, I was on 343 before.

>>12271 It seems ok to me in a drag-and-drop test on a gelbooru URL I just did now, and I do not think I have changed anything in the past four weeks–can you say any more about your situation? How are you importing this URL: through a search or a drag and drop? Have you imported any new parsers for those domains? And what are your 'default tag import options' set up as under network->downloaders->manage default tag import options? Do you have one TIO for all domains to rely on, or specific ones for gelb and pixiv? Do they all look correct?

when I try to manually process tags, the entire client locks up when I wait for it to do it automatically, it immediately finishes it's about 200 tags behind

>>12352 i can only process tags using shutdown processing

>>12353 >>12352 Thank you for this report. Are you manually triggering repo processing through the 'process now' button under review services? This button has always been a bit debug tier and I have heard it increasingly working badly for some users with the current PTR. If you have it run in idle time, does it run ok? When you say it 'immediately finishes', does that mean it does no work at all? Is it 200 updates behind on the review services page, and it doesn't seem to be catching up, even in idle time?

Was doing some quick and dirty processing and when I attempted to set 2 images as alternates I got a DBException "IndexError: Cannot choose from an empty sequence" DBException IndexError: Cannot choose from an empty sequence Full Traceback: File "include\ClientGUICommon.py", line 708, in EventButton self._func( *self._args, **self._kwargs ) File "include\ClientGUIManagement.py", line 1370, in _SetCurrentMediaAs self._ShowSomeDupes() File "include\ClientGUIManagement.py", line 1385, in _ShowSomeDupes hashes = self._controller.Read( 'random_unknown_duplicate_hashes', file_search_context, both_files_match ) File "include\HydrusController.py", line 593, in Read return self._Read( action, *args, **kwargs ) File "include\HydrusController.py", line 183, in _Read result = self.db.Read( action, *args, **kwargs ) File "include\HydrusDB.py", line 931, in Read return job.GetResult() File "include\HydrusData.py", line 1582, in GetResult raise e Database Traceback (most recent call last): File "include\HydrusDB.py", line 557, in _ProcessJob result = self._Read( action, *args, **kwargs ) File "include\ClientDB.py", line 9520, in _Read elif action == 'random_unknown_duplicate_hashes': result = self._CacheSimilarFilesGetRandomUnknownDuplicateHashes( *args, **kwargs ) File "include\ClientDB.py", line 1285, in _CacheSimilarFilesGetRandomUnknownDuplicateHashes hash_id = random.choice( list( potential_hash_ids ) ) File "random.py", line 260, in choice IndexError: Cannot choose from an empty sequence

Just found out that the media was actually set as alternates, it just would not remove the media from the duplicate page. Closing the tab entirely and opening a new duplicate tab showed no potential pairs once again.

Zoom toggle seems to be entirely ignored on images of the same height as monitor, even if zoom level has been altered some other way. Like scrolling back from a lower res image in duplicate viewer, or having used ctrl+wheel

>>12417 >>12418 Thank you for this report. This is a very helpful error, and I think related to the 'ghost pair' issue I'll be tackling this week.

>>12422 Thank you for this report. I think I am forgetting how some zoom stuff works, or just misunderstanding. Can you talk about it a bit more? When you say the 'zoom toggle', do you mean the zoom button/action that switches between 100% and the full media viewer size, or something else? Can you say what you would like to happen when it is hit and the image is the same height as the monitor?

>>12412 >Are you manually triggering repo processing through the 'process now' button under review services yes >Is it 200 updates behind on the review services page, and it doesn't seem to be catching up, even in idle time? i mean 200 processed tags behind the number downloaded when idle time starts, the popup in the bottom right says it's starting to process tags, then immediately says it's finished

>>12427 Yeah, I mean the "switch_between_100_percent_and_canvas_zoom" action What happens is literally nothing, it just refuses to change the zoom level back to 100% I'll try to be a bit more formal and explicit: My environment: Hydrus version 349 on Windows 10 with 1920x1080 monitor. Example steps to reproduce: >open static image where height==1080, width<=1920 in the fullscreen media viewer >zoom in with ctrl+wheel up >press Z (switch_between_100_percent_and_canvas_zoom) Expected behavior: Zoom level changes back to 100%, as happens with images of other dimensions. Actual result: Nothing happens, image remains zoomed in. Same result happens with image of width==1920, height<=1080. Does NOT happen if either dimensions of image exceed the monitor/canvas.

Another thing: ctrl+shift+T / "unclose_page" When several pages have been closed, I expect it to re-open the most recently closed page, which is at the TOP of the "closed pages" list. Actual result: It re-opens the page from the BOTTOM of the "closed pages" list It feels very counter-intuitive to me and is not how browsers do it, so I assume it's a bug.

>>12087 Hydrus version: 348 (Installed via AUR) I'd like to report the same bug. The behavior is almost exactly the same as described. The only difference is mine has 35 images with 35 more "ghost" thumbnails – one for each image. 35 pairs of thumbnails. The layout of the thumbnails changes between highlights. I have included three images (bug1-bug3.jpg) showing different layouts after clearing the highlight, and highlighting again. The last image shows the layout of all of the thumbnails once I've revealed them all by selecting them. It's been two months since I downloaded these images so I don't remember exactly when the behavior started but I've been meaning to report it for a while now. Perhaps you've already fixed it because I'm running another query now and its images don't seem to have this problem, not yet at least

Ctrl + Backspace in Windows inserts a control character instead of deleting the previous word in a text box. Could that be fixed?

When entering incorrect login credentials for Pixiv, Hydrus falsely reports that the login is valid and the user is logged in even though they actually haven't logged in.

I think there's a bug in the options -> files and trash -> remove files from view when they are sent to the trash , where deleted images are not being removed from a page. I'm not sure how to reproduce it or anything, but I've experienced it at least once before. The first pic shows trashed images not being removed, and the second image shows a new page opened (after deleting a bunch of images), with the exact same queries: it has the expected behaviour; no trashed images. All the images I'm dealing with right now are animated, not sure if that makes a difference.

While in the Hydrus Client Media Viewer, if you open the manage tags window, the media viewer's hover-over popups (tags, known urls, and the top toolbar) still pop up and obstruct the manage tags window. This occurs even when the manage tags window is the current active window.

I am sorry, I let this thread get away from me last week. >>12431 What are your idle settings under options->maintenance and processing? If you turn the system CPU setting off, does that stop this behaviour?

>>12432 Thank you, I will check this this week.

>>12442 Thank you, I will check this as well.

>>12494 Thank you. I have been doing some thumbnail drawing work recently, and I am not done. It may be better now. I sometimes get something like this on my IRL client, but on only 4-6 thumbs, and it fixes itself with a ctrl+a. I will investigate this more.

>>12501 Thank you for this report. I had no idea this happened. It looks like some Rich Text something, as here: https://trac.wxwidgets.org/ticket/10344 I'll see if changing the style removes this.

>>12512 Thank you for this report. I think I wrote the Pixiv login script. If I remember right, Pixiv still gives you a cookie whether you are a guest or correctly logged in. I can't promise anything too fast, but I will make a job to see if I can add a login step to pixiv to check for login/logout buttons or whatever on the homepage once the login attempt happens.

>>12514 Thank you for this report. I'll check this code. The animated part might have something to do with it.

>>12516 Thank you, I am sorry. I had this from a couple of other people as well. In letting the hovers display even when the new duplicate filter always-on-top has focus, I accidentally added these dialogs to the permissable 'you can show' logic. I will fix it for 352.

When I manually stopped a dupe search at distance 8
RuntimeError
wrapped C/C++ object of type BetterStaticText has been deleted
File "site-packages\wx\core.py", line 3259, in <lambda>
File "include\ClientGUIManagement.py", line 1272, in wx_code
self._refresh_maintenance_status.SetLabelText( '' )
File "include\ClientGUICommon.py", line 944, in SetLabelText
wx.StaticText.SetLabelText( self, text )


>>12527 same thing im testing it with the settings in pic related

On image viewer, if I drag the image too fast, the mouse pointer show again and the drag is limited to the view only.

>>12535 Wow, thank you for this report. I am 97% confident I can knock that on the head this week.

>>12537 I have just had a look at the idle calculation code and nothing stands out. I am not sure what is going on here. This is a long-shot, but what happens if you hit help->debug->debug modes->force idle mode? Do it at least 30s after booting the client. It should wake the process daemon and do some work, and the 'should I stop early' calculation should be functionally the same as shutdown processing. Otherwise, if you have not written down the number of updates currently downloaded and processed, can you do that, and then check the numbers again in three days? I'd like to see if the idle processing is doing any work at all when you aren't looking, or if it is always stopping early.

>>12538 Thank you for this report. This was a compromise I added to support touch-drag for touchscreens, which sperg out at the hidden anchor mouse warping. Can you just confirm that under options->media, you have the 'WINDOWS ONLY: anchor blah' option checked atm? I will add an option here to override the touchscreen hack so it always anchors (probably, I'll change it so that is the new default, and you have to turn on touchscreen support, so hopefully you'll just be fixed here soon). I am 68% confident I can get this done this week for 352, and 95% confident I can get it done for 354.

>>12142 Me again. got to 351, error occuring less and overall better stability. the erro i was getting changed, but doing the same action, and with same outcome, greyed out pages and pending menus. here the error: 2019/05/13 14:51:40: Uncaught exception: 2019/05/13 14:51:40: wxAssertionError C++ assertion "win" failed at /home/vagrant/wxPython-4.0.4/ext/wxWidgets/src/gtk/menu.cpp(84) in DoCommonMenuCallbackCode(): event for a menu without associated window? File "site-packages/wx/core.py", line 3259, in <lambda> File "include/ClientGUICommon.py", line 3365, in WXDoIt self._func( *self._args, **self._kwargs ) File "include/ClientGUI.py", line 5003, in RefreshMenu self._menubar.Replace( old_menu_index, menu, label )

>>12549 opens the same window as if I hit process now from network -> services acts the same as well, crashing the client soon after starting (pics related) when I just waited for it a textbox opened in the bottom right saying it was finished

>>12558 >>12549 >I'd like to see if the idle processing is doing any work at all when you aren't looking, or if it is always stopping early my computer isn't on most of the day, so this isn't really applicable sorry

>>12494 What GTK theme and what distro?>>12494

>>12538 Yes, it's checked. >confident I can get this done this week Take your time man. Thank you.

>>12556 Thank you for the update. I will not have time today to look at this, but I am 95% certain I can put some time into it by 354. >>12558 >>12559 When you say crashing, does it actually quit the program unexpectedly, or does it only hang, 'Not Responding', indefinitely? Depending on the client, big maintenance jobs like this can sometimes block the UI thread until they are done (due to my old shit code not handling big asynchronous tasks properly). If your window is only hanging, greyed out, can you please check if the client.exe is doing CPU/HDD work under Task Manager? If so, it is probably working ok, just not updating the UI. It will eventually catch up and finish cleanly, even if it takes some time. If it is actually crashing and quitting the program, then something worse is happening. It might be related to running out of memory or another unexpected OS/hardware bottleneck. Processing is typically very stable, especially for Windows users atm. Nonetheless, if your client is having trouble doing processing work in idle time under any context, I think your most practical solution right now is to limit it to only do this work in shutdown time. I have plans to improve big maintenance task ui politeness, but it will not come quickly.

>>12572 yeah, seems like it was just blocking ui, it is still accessing disk annoying, but whatever

When shortcuts are used to open the manage tags window (f3 or a custom one), if you press spacebar while typing the tags, the window closes.

>>12578 Thank you for this report. I am afraid I cannot reproduce it unless I bind 'spacebar' to 'manage_file_tags' shortcut action under the 'media' shortcut set. Do you have something like this? Do you have spacebar set to any other shortcuts? And if you open manage tags from the right-click menu (i.e. not using a shortcut), do you not get this behaviour? What happens if you press spacebar on the similar manage ratings or manage known urls dialogs?

>>12580 Somehow I had spacebar binded to manage the tags. Removed and now it works fine. Sorry.

>>12581 Great, no worries. Thanks for letting me know.

Seems that whenever I have my file page collected(by series, chapter, title, etc.) and I refresh said page, it chooses a random image to show rather than showing the first image from my 'sort by.' So when I have a comic, and I sort by series-creator-title-chapter-page it should show page 1 (the title page) when I collect, but when I refresh the files, it shows a random page. If, when I'm seeing a random page from refreshing, though, I change what I'm collecting by(uncheck and recheck the box), it shows the first page again, correctly.

>>12514 >>12533 Just to follow up on this, I think it may have been my fault. It seems like the trashed images only still appear on pages which are/were opened as new page -> files -> all local files and do not appear on pages opened with new page -> files -> my files. I'm still not really sure what the distinction is between these two page types though. That being said, there may still be a bug here. On the all local files pages, the images will be removed from view right after being deleted, but then show up again after a refresh (with the little trash can icon).

>>12606 Thank you for this report. Yeah, it looks like I messed up some of the sort/collect publishing code when I added collect save this past week. I am 99.7% confident I can fix it for 353. >>12619 all local files is a catch-all I added that combines both my files and trash (and also the secret repository updates domain that you don't normally see). If a file is in all local files, it should be on your hard drive, and when a file leaves this domain at the db level, it is scheduled for physical deletion. It is a bit of a stupid technical difference, and I generally recommend you just search my files unless you are doing something special. When I eventually add support for multiple local files domains, it will gain importance. I think I will hide this file domain for users who have not turned on help->advanced mode as it is not important and has made for some confusion before as well. Thank you again for the report, it was not your fault. I looked through the code, and the remove on all local files pages was unintentional. The remove option is supposed to reflect a quicker 'sync' with what is true, so when you trash from a 'my files' domain, removing instantly is an accurate option since a search refresh will not reproduce the trashed file. But all local files pages include trash, so sending to trash should not remove them.

advanced operations > copy tags from ptr to local db is not working for me in 353. No error, just no effect. Happens for single and multiple images.

>>12697 Actually, does the mass copy not lock Hydrus anymore? It might have happened slowly in the background after all. A visual indicator would be a great addition if so. Separate bug: File URLs are not working in Firefox for me although it does for my chromium browser. Maybe it's because my Firefox path has a space in it?

Was scrolling through my library and got this error message: AttributeError 'ThumbnailMediaCollection' object has no attribute 'GetMediaResult' Traceback (most recent call last): File "include\HydrusPubSub.py", line 141, in Process callable( *args, **kwargs ) File "include\ClientGUIMedia.py", line 2414, in ProcessContentUpdates self._RedrawMedia( affected_media ) File "include\ClientGUIMedia.py", line 3000, in _RedrawMedia self._FadeThumbnails( thumbnails_to_render_now ) File "include\ClientGUIMedia.py", line 2717, in _FadeThumbnails self._DirtyAllPages() File "include\ClientGUIMedia.py", line 2628, in _DirtyAllPages self._DirtyPage( clean_index ) File "include\ClientGUIMedia.py", line 2642, in _DirtyPage HG.client_controller.GetCache( 'thumbnail' ).CancelWaterfall( self._page_key, thumbnails ) File "include\ClientCaches.py", line 2418, in CancelWaterfall cancelled_media_results = { media.GetMediaResult() for media in medias } File "include\ClientCaches.py", line 2418, in <setcomp> cancelled_media_results = { media.GetMediaResult() for media in medias } AttributeError: 'ThumbnailMediaCollection' object has no attribute 'GetMediaResult'

Simple downloader (all files linked by images in page) gives the below error with a fireden.net thread (for example: https://boards.fireden.net/v/thread/464181401/#464181401). Images will only import as normal if they are opened in a browser first. Client 353. The server's error text was too long to display. The first part follows, while a larger chunk has been written to the log.… (Copy note to see full error) Traceback (most recent call last): File "include\ClientImportFileSeeds.py", line 1274, in WorkOnURL self.DownloadAndImportRawFile( file_url, file_import_options, network_job_factory, network_job_presentation_context_factory, status_hook ) File "include\ClientImportFileSeeds.py", line 594, in DownloadAndImportRawFile network_job.WaitUntilDone() File "include\ClientNetworkingJobs.py", line 1137, in WaitUntilDone raise self._error_exception include.HydrusExceptions.ServerException: The server's error text was too long to display. The first part follows, while a larger chunk has been written to the log. <!DOCTYPE html> <!–[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]→ <!–[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]→ <!–[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]→ <!–[if gt IE 8

>>12697 >>12699 Thank you for these reports. I will check the UI for the advanced operations. It is debug-tier at the moment, so if it is completing asynchronously, I'll change it to wait and throw up a modal progress dialog or something. For the file urls, is this clicking on them in the top-right hover or similar? And they fail to open if you have firefox's path set in options->external programs? If firefox is in "Program Files" or something with a space, I think it is supposed to be ok–especially if Chromium is similar and works–, but you can try wrapping the path in quotes, maybe? "C:\Program Files\firefox\firefox.exe" "%path%" Or could it be a firefox profile issue? If you have multiple profiles set up and need to use -profile profile_path to launch your FF, maybe that is it? You can explore this further by turning on help->debug->report modes->subprocess report mode. That should spam a big set of popups if you have a custom url launch path set up. If it isn't obvious to you, feel free to take a screenshot of the popups and post them here, but make sure to exclude or redact any personal info.

>>12766 Shit, I am sorry. I have had this from a couple of other users as well. I fucked up the new clever thumbnail queueing with collections. This will spam any time collections that are about to fade get an update event like fast scrolling. I am 99.7% sure I can have this fixed for next Wed. I apologise for the inconvenience.

>>12768 Thank you for this report. When I put that URL in the simple downloader with 'all files linked by images in page' simple downloader formula it works ok. When I do '4chan thread (all linked images)' it downloads the thread apparently ok but don't parse any file URLs. That error says ServerException (typically a 5XX error), so is it possible the domain was having trouble when you tried it? If you go to install_dir/db/client - 2019-05.log and do a ctrl+f to find that section, what does the extended stuff say? That should be most if not all of the html returned by the server–does it have something like 'currently undergoing maintenance' or anything? Or maybe a cloudflare page?

>>12792 Thanks for your help. Seems to be working for me now too. The extended log message is 112 lines long, but the gist of it is that there was an error 522 (connection timed out), and, yeah, some cloudflare page stuff (short version below, tags removed). Weird because I had no such trouble connecting via Firefox at that time. … What happened? The initial connection between Cloudflare's network and the origin web server timed out. As a result, the web page can not be displayed. … What can I do? If you're a visitor of this website: Please try again in a few minutes. If you're the owner of this website: Contact your hosting provider letting them know your web server is not completing requests. An Error 522 means that the request was able to connect to your web server, but that the request didn't finish. The most likely cause is that something on your server is hogging resources. Additional troubleshooting information here. https://support.cloudflare.com/hc/en-us/articles/200171906-Error-522

>>12793 The Eternal CloudFlare strikes again. I'll update that ServerException result to put the actual status code, 522 or whatever, at the front of the text so it is more obvious that the solution here is to wait a bit and try again.

>>12789 Experimented some more, and I found out that it was purely due to a peculiarity in how I launch Firefox (from a batch file). The same problem happens no matter how I open a URL, so it is nothing to do with Hydrus :)

>>12821 Great, thanks for letting me know.

I keep getting these errors when trying to use collections after the last update. It only works if I set the collection then f5 the search. AttributeError
'ThumbnailMediaCollection' object has no attribute 'GetMediaResult'
File "site-packages\wx\core.py", line 3259, in <lambda>
File "include\ClientGUIPages.py", line 2234, in MediaDragAndDropDropped
source_page.GetMediaPanel().RemoveMedia( source_page.GetPageKey(), hashes )
File "include\ClientGUIMedia.py", line 2454, in RemoveMedia
self._RemoveMediaByHashes( hashes )
File "include\ClientMedia.py", line 854, in _RemoveMediaByHashes
self._RemoveMediaDirectly( affected_singleton_media, affected_collected_media )
File "include\ClientGUIMedia.py", line 3063, in _RemoveMediaDirectly
self._DirtyAllPages()
File "include\ClientGUIMedia.py", line 2628, in _DirtyAllPages
self._DirtyPage( clean_index )
File "include\ClientGUIMedia.py", line 2642, in _DirtyPage
HG.client_controller.GetCache( 'thumbnail' ).CancelWaterfall( self._page_key, thumbnails )
File "include\ClientCaches.py", line 2418, in CancelWaterfall
cancelled_media_results = { media.GetMediaResult() for media in medias }
File "include\ClientCaches.py", line 2418, in <setcomp>
cancelled_media_results = { media.GetMediaResult() for media in medias }


>>12828 Hey, thank you for this report. I apologise. I had this from several other people as well. I fucked up how collections are dequeued from the thumbnail 'waterfall' fade system in 354, so many update events that occur while collection thumbs are fading in (like fast scrolling) throw this error. It is fixed for 355. I don't use collections myself much, and I keep messing up their rendering and sorting, so I will add some basic tests to my weekly routine to stop this.

Cannot import webms on Ubuntu 18.04, client 355. Details: > fresh xubuntu 18.04 install, still a babby with it > extract hydrus, run client > import a few JPG, PNG, GIF; complains about ffmpeg missing > install ffmpeg, version 3.4.6-0ubuntu0.18.04.1, reboot > now can import those (and most) files, but a popup says ffmpeg version not detected; lines from "db/client - 2019-6.log" below. Cannot import webm (can as normal on Win7 install), but CAN import MP4.
2019/06/06 02:09:33: FFMPEG was recently contacted to fetch version information. While FFMPEG could be found, the response could not be understood. Significant debug information has been printed to the log, which hydrus_dev would be interested in.
2019/06/06 02:09:33: FFMPEG was recently contacted to fetch version information. While FFMPEG could be found, the response could not be understood. Significant debug information has been printed to the log, which hydrus_dev would be interested in.

{'universal_newlines': True, 'startupinfo': None, 'env': None}

environ({'GDM_LANG': 'en_GB', 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'MPLCONFIGDIR': '/tmp/tmpb0q9iesu', 'HOME': '/home/localuser', 'SSH_AGENT_PID': '1292', 'XDG_DATA_DIRS': '/usr/share/xubuntu:/usr/share/xfce4:/home/localuser/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share:/var/lib/snapd/desktop:/usr/share', 'QT_ACCESSIBILITY': '1', 'GPG_AGENT_INFO': '/run/user/1000/gnupg/S.gpg-agent:0:1', 'SSH_AUTH_SOCK': '/run/user/1000/keyring/ssh', 'XAUTHORITY': '/home/localuser/.Xauthority', 'DESKTOP_SESSION': 'xubuntu', 'MATPLOTLIBDATA': '/home/localuser/toplevel/Hydrus/mpl-data', 'LANG': 'en_GB.UTF-8', 'SESSION_MANAGER': 'local/xubuntu-laptop:@/tmp/.ICE-unix/1310,unix/xubuntu-laptop:/tmp/.ICE-unix/1310', 'LD_LIBRARY_PATH': '/home/localuser/toplevel/Hydrus', 'GDMSESSION': 'xubuntu', 'PWD': '/home/localuser', 'XDG_VTNR': '7', 'SHELL': '/bin/bash', 'XDG_CONFIG_DIRS': '/etc/xdg/xdg-xubuntu:/etc/xdg:/etc/xdg', 'QT_QPA_PLATFORMTHEME': 'gtk2', 'XDG_SESSION_TYPE': 'x11', 'XDG_SESSION_ID': 'c2', 'XDG_SEAT': 'seat0', 'USER': 'localuser', 'DISPLAY': ':0.0', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session0', 'GLADE_PIXMAP_PATH': ':', 'CLUTTER_BACKEND': 'x11', 'GTK_OVERLAY_SCROLLING': '0', 'XDG_GREETER_DATA_DIR': '/var/lib/lightdm-data/localuser', 'GLADE_MODULE_PATH': ':', 'XDG_MENU_PREFIX': 'xfce-', 'XDG_SESSION_DESKTOP': 'xubuntu', 'GLADE_CATALOG_PATH': ':', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'LANGUAGE': 'en_GB:en', 'LOGNAME': 'localuser', 'XDG_CURRENT_DESKTOP': 'XFCE', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'XDG_RUNTIME_DIR': '/run/user/1000', 'SHLVL': '0'})

Response:


>>12111 >you can hide the bottom-left preview window under options->gui. where

>>12864 Thank you for this report. I have been chasing up this same issue with another user. The PATH in your environment is all correct there, and FFMPEG is being found, but it looks like hydrus is getting no response back. Just no text at all. Ok, thank you for the debug data. I'll have another look at what I am doing here and why this is happening. I don't have my devstuff in front on me right now, but I'll check this and update the debug info to include more about what if anything was read back. If you are feeling cheeky, you can try downloading a static ffmpeg exe here: http://ffmpeg.org/download.html and then place it in install_dir/bin. The client may find that is ok after a restart–that's how Windows users get it, and it should work the same way for Linux. If you try this, let me know if it works.

>>12868 Fifth option down, 'Hide the preview window'.

>>12889 Oh hey, that does work (with 4.1.3-static). Appreciate that. Incidentally, I did try running hydrus via wine, which, again, I am using for the first time. I think I've set it up fine because Notepad++ runs, but hydrus crashes it immediately (see below). No big deal to me. Thanks for the great support and enjoy your E3. wine: Call from 0x7b44c1e7 to unimplemented function api-ms-win-core-path-l1-1-0.dll.PathCchCanonicalizeEx, aborting
wine: Unimplemented function api-ms-win-core-path-l1-1-0.dll.PathCchCanonicalizeEx called at address 0x7b44c1e7 (thread 002b), starting debugger...

Getting strange behavior where it hangs at committing for hours, then if you alt-f4 out, it closes the window and begins eating RAM in the background until manually killed. 2019/06/08 02:38:25: shutting down gui…
2019/06/08 02:38:25: waiting for daemons to exit
2019/06/08 02:38:25: public tag repository sync: processing updates
2019/06/08 02:38:25: fattening service info
2019/06/08 14:39:32: processed 5,072,024 content rows at 117 rows/s
2019/06/08 14:39:32: committing
2019/06/09 21:17:57: shutting down controller…
2019/06/09 21:17:57: hydrus client shut down
2019/06/09 21:19:10: hydrus client started
2019/06/09 21:19:10: booting controller…
2019/06/09 21:19:10: booting db…
2019/06/09 21:19:10: preparing disk cache
2019/06/09 21:19:14: preparing db caches
2019/06/09 21:19:14: booting db…
2019/06/09 21:19:14: initialising managers
2019/06/09 21:20:16: booting gui…


This is less of a bug report, but more of a fix for a longstanding known issue. So at some point recently Sankakuchan decided to upload every doujin they could get their hands on, which is pretty annoying considering the default hydrus parsers limitations as it effectively buries actual content. Also said doujins are only like 1/3 tagged which is a pain in the ass. Some light investigation and prototyping and I was able to fix the hydrus parser so it could just keep going forever in theory. I'm honestly surprised this was ever an issue considering how simple the fix was, basically I just made it get the next page correctly. More specifically an html tag, a div specifically, contained an attribute called 'next-page-url' that contains the url for the next page by id which has no limitation rather than &page which is limited to page 25/50 if authenticated. Some light testing on something I indexed a while back got through 1390 posts.

I've got two images, one of which i've accidentally tagged as a worse quality dupe to the other. Trying to set that image as an alternate image by right clicking on both, Duplicates > Set relationship > Set as Alternates results in both files still being tagged as one worse and one better. Can anyone else confirm?

>>12910 Yeah, although I'm like 95% sure hydrusdev knows, I recall reading something about it like a week ago. If you're on 355 (possibly 354 too), the duplicate system is in the middle of an overhaul and hydrusdev needs to fix the context menu duplicate commands. I've been having a fun time with cg sets due to this.

Using v355 on linux. I filter deduplicates bit it doesn't apply anything. After increasing the search distance from 0 to 2 it was still listing all the previous images. I moved to 4 and it still lists the previous images from distances 0 and 2 since the filter didn't apply the commits I guess. It worked previously so v355 broke it.

>>12940 Same here. I made the mistake of importing multiple CG sets, then scratching my head wondering if I've already sorted their duplicates halfway through.

Artstation gallery downloader seems to be completely broken much like pixv still is, what a disappointment. Spitting out a 403 before saying Done! and importing zero files. You can see the artist does exist so it should function like any other parser but doesnt. https://www.artstation.com/yuzq

>>12895 Hey, I have extended that FFMPEG debug info dump to say more in today's release. If you like, you can try doing help->about again in 356 with your system FFMPEG, but it is not a huge priority–I am working with another user on this same bug. If you work and are happy now, I am fine to leave it alone for the time being. I am not totally sure what that dll issue is, but I think it may be related to VCRedist. Ever since I moved to Python 3, I think you need VCRedist 2015 on a pre-Win10 Win environment: https://www.microsoft.com/en-us/download/details.aspx?id=48145 I think those api-ms-win-… dlls are related to that.

>>12903 Thank you for this report. It seems like the client did a gigantic repository processing job there, with the 5M rows in one go over about twelve hours. Are you by any chance on a slightly older version of the client? New clients should only ever do 1 hour of repo processing at once and make checkpoint transactions to keep cancel lag low, and they should keep big transactions out of memory. If you are on an older version, can you try updating to, say, at least 354? As it happens, today's 356 will have improved cancel tech here, including a cancel button on the exit splash screen. If it is convenient, I would recommend today's release.

>>12906 That's interesting, well done! If you have a github account, there is a big repository of parsers here: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System I have previously made commitments not to personally work significantly on new Sankaku parsers (one of the owners contacted me to let me know they are always short on bandwidth, for which I sympathised), but the owner of that repo would be interested in this neat discovery, I am certain.

>>12910 >>12915 >>12940 >>12941 Hey, I am sorry for the current difficulties here. The duplicates system is in slight limbo as I move each component over. Some features such as undoing/dissolving groups are not yet available, and certain logic behind figuring out which files are alternate inside an alternate group are still inefficient. Today's release will move better/worse/same quality dupes over to the new system, which will improve the situation but not complete it. Now I am in the weeds of this code, I am only discovering how bad the old system was. Even though some 'reset and requeue' actions appeared to work, they often left invalid relationships behind. The new system I am moving to is comprehensive and reduces potential pairs much faster due to correct transitivity, but it will be a few more weeks until everything is working on the same data. You shouldn't be able to set anything 'invalid' on the new system, but in situations where it doesn't know what to do, like manually overruling a better/worse with an alternate, it makes no changes. Please bear with me for a bit. >>12940 When you apply an action to some duplicates, is this through the duplicate filter or the 'show some random dupes' button? Are the 'potential' pair counts decrementing at all? I believe the duplicate filter will always decrement the potential pairs count, but some actions may not decrement the count completely yet (I think in some cases, like setting alternates, it may not decrement it at all), but I am not certain. The situation will improve in today's 356, particularly for setting groups of alternates, and moreso in future weeks.

>>12949 Yes, I am afraid they set CloudFlare javascript protection on that API request URL. The hydrus downloader cannot pass through that yet, so the downloader is currently broken.

Encountered a strange bug with client version 355 where after the client had been open for over 24 hours, Ctrl+V was not working for pasting links. A restart of the client fixed it, but I thought it should be made aware of.

>>12956 Sure - 356 reports the same thing in-UI, but the log now gives the below:
2019/06/20 01:36:53: FFMPEG was recently contacted to fetch version information. While FFMPEG could be found, the response could not be understood. Significant debug information has been printed to the log, which hydrus_dev would be interested in.
2019/06/20 01:36:53: FFMPEG was recently contacted to fetch version information. While FFMPEG could be found, the response could not be understood. Significant debug information has been printed to the log, which hydrus_dev would be interested in.

{'env': None, 'universal_newlines': True, 'startupinfo': None}

environ({'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'DESKTOP_SESSION': 'xubuntu', 'SSH_AUTH_SOCK': '/run/user/1000/keyring/ssh', 'SHELL': '/bin/bash', 'LOGNAME': 'localuser', 'SESSION_MANAGER': 'local/xubuntu-laptop:@/tmp/.ICE-unix/3665,unix/xubuntu-laptop:/tmp/.ICE-unix/3665', 'XDG_GREETER_DATA_DIR': '/var/lib/lightdm-data/localuser', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session0', 'LD_LIBRARY_PATH': '/home/localuser/toplevel/Hydrus', 'QT_QPA_PLATFORMTHEME': 'gtk2', 'QT_ACCESSIBILITY': '1', 'XDG_SEAT': 'seat0', 'GDM_LANG': 'en_GB', 'DEFAULTS_PATH': '/usr/share/gconf/xubuntu.default.path', 'DISPLAY': ':0.0', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'MPLCONFIGDIR': '/tmp/tmpzqtgebog', 'XDG_CONFIG_DIRS': '/etc/xdg/xdg-xubuntu:/etc/xdg:/etc/xdg', 'GTK_OVERLAY_SCROLLING': '0', 'XDG_SESSION_DESKTOP': 'xubuntu', 'GLADE_CATALOG_PATH': ':', 'XDG_CURRENT_DESKTOP': 'XFCE', 'XDG_RUNTIME_DIR': '/run/user/1000', 'GLADE_MODULE_PATH': ':', 'GPG_AGENT_INFO': '/run/user/1000/gnupg/S.gpg-agent:0:1', 'GLADE_PIXMAP_PATH': ':', 'XDG_SESSION_ID': 'c2', 'XDG_DATA_DIRS': '/usr/share/xubuntu:/usr/share/xfce4:/home/localuser/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share:/var/lib/snapd/desktop:/usr/share', 'GDMSESSION': 'xubuntu', 'MATPLOTLIBDATA': '/home/localuser/toplevel/Hydrus/mpl-data', 'USER': 'localuser', 'XAUTHORITY': '/home/localuser/.Xauthority', 'SSH_AGENT_PID': '3647', 'LANG': 'en_GB.UTF-8', 'MANDATORY_PATH': '/usr/share/gconf/xubuntu.mandatory.path', 'CLUTTER_BACKEND': 'x11', 'LANGUAGE': 'en_GB:en', 'SHLVL': '0', 'XDG_SESSION_TYPE': 'x11', 'HOME': '/home/localuser', 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'XDG_VTNR': '7', 'PWD': '/home/localuser', 'XDG_MENU_PREFIX': 'xfce-'})

STDOUT Response:

STDERR Response: ffmpeg: /home/localuser/toplevel/Hydrus/libz.so.1: version `ZLIB_1.2.9' not found (required by /usr/lib/x86_64-linux-gnu/libpng16.so.16)
This is LTS, but it is up-to-date. Looking at my packages tells me roughly that zlib is at 1.2.11. I'll get back to you about the dll issues.

Encountered this bug in 355, updated to 356 and it persists. Tried to separate a Hentai Foundry artist lookup subscription in the 'manage subscriptions' window. TypeError '<' not supported between instances of 'SubscriptionQuery' and 'SubscriptionQuery' File "include\ClientGUICommon.py", line 781, in EventButton self._func( *self._args, **self._kwargs ) File "include\ClientGUIScrolledPanelsEdit.py", line 4952, in Separate panel = EditChooseMultiple( dlg, choice_tuples ) File "include\ClientGUIScrolledPanelsEdit.py", line 197, in init choice_tuples.sort() Same thing happens with the rule34xxx tag lookup when trying to separate the subscription. Bug didn't happen when separating a danbooru+gelbooru tag lookup subscription.

>>12959 >is this through the duplicate filter or the 'show some random dupes' button? Through the duplicate filter. >Are the 'potential' pair counts decrementing at all? Yes when I'm finished the potential pairs count goes back to 0 but it really hasn't applied anything. I updated to v356 today. I imported a few new images and rescanned for duplicates. I tried doing the same thing and it's still broken. Hopefully in the next release or two you can figure it out.

>>12960 e621's parser is now broken as of today. ServerException
The server's error text was too long to display. The first part follows, while a larger chunk has been written to the log.
No error trace--here is the stack:
File "threading.py", line 884, in _bootstrap
File "threading.py", line 916, in _bootstrap_inner
File "include\HydrusThreading.py", line 342, in run
callable( *args, **kwargs )
File "include\ClientNetworkingJobs.py", line 974, in Start
self._SetError( e, error_text )
File "include\ClientNetworkingJobs.py", line 438, in _SetError
HydrusData.ShowException( e )
File "include\ClientData.py", line 434, in ShowExceptionClient
trace = 'No error trace--here is the stack:' + os.linesep + ''.join( traceback.format_stack() )

<!DOCTYPE HTML>
<html lang="en-US">
<head>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollo

>>12991 parser is working again, didn't do anything just left it idle. not really sure what caused it or why it happened. something on e6's side?

>>12993 They turned on pretty aggressive ddos protection for a few hours. Everything that was not a legitimate browser received access forbidden errors.

>>12979 Thank you for this report. Do you have the new 'clipboard watcher' on? Is it possible the clipboard crashed from, say, trying to pull an in-clipboard screenshot as text or anything?

>>12981 Thank you, this was exactly what I was looking for. It looks like the env being passed still has some hydrus-specific path somewhere so ffmpeg is trying to load .so files from the hydrus dir. LD_LIBRARY_PATH, looks like, which I thought I was already redirecting–looks like the LD_LIBRARY_PATH_ORIG, which I am supposed to have, isn't there for your situation. What a headache. I will give this another go.

>>12984 Shit, thank you for this report. I am 99.7% certain I can have this fixed for 357.

>>12987 Are you clearing your duplicate search progress ('reset potential duplicates' from the cog icon under the duplicates page) and researching everything? If so, you do not need to do this, and I would not recommend this action for any normal operation. You do not need to 'rescan' for dupes in this way to keep up with new files. It would requeue many intra-alternate comparisons with the current system. If you are just 'filling up' the 'finding potential duplicates' search bar and getting the same alternate pairs come up over and over, this would point to a different problem. Either way, this whole situation with comparing alternates is on my mind, as here also >12999 . As I switch potentials over, I will write an alternates decisions cache to make sure you don't ever see the same thing twice, even if you were to flush and re-search everything.

>>12993 >>12991 >>12996 Thank you for this report. I'm going to try to write a 'This looks like a Cloudflare error' detector to try to give better summary info in these error cases, since this keeps happening.

>>13007 >If you are just 'filling up' the 'finding potential duplicates' search bar and getting the same alternate pairs come up over and over, this would point to a different problem Yes, that's what I've been doing. I can give you logs or something if it'll help debug the issue. If you're re-writing it I guess you won't need them though.

For whatever reason, the pixiv single file page parser stopped working for me - every url I plug in tells me it found nothing in the document. In looking at the raw data from the example urls I noticed that any nsfw pages tell me I'm not logged in but I've checked the login credentials outside of hydrus and they're correct, and regardless sfw submissions don't download properly either. It had been working as of yesterday and the tag gallery parser still picks up urls, albeit only non-nsfw. Did something change on pixiv's end or what? It chucks a few include.HydrusExceptions.ParseException: Unable to parse that JSON: Expecting value: line 1 column 1 (char 0) exceptions at me for the various tags when testing a url but I'm assuming that's due to the pages not showing up.

>>13005 Looks like v357 is detecting my system ffmpeg properly, such that it can import webms without the static one now. Great stuff. Thanks for your hard work.

So I found an interesting yiff.party issue. So I've noticed one subscription keeps failing and gives me the message 'connection broke mid-request' and retries over and over to no success. So I decided to look into it, the particular json it is trying to download is here: https://yiff.party/7242260.json If you actually look at it in a browser you'll see that it literally cut off part of the way through, or at least it does for me off my VPN, on several different VPN servers, and on TOR. I suppose the error message it gave me was correct. Any idea on what's causing that? I'd guess a server error >>13038 Also similiar issue here, for example the parser finds nothing for https://www.pixiv.net/member_illust.php?mode=medium&illust_id=64809850

>>13011 Thanks. Yes, I will wait a bit for this. The new potentials system will work better all-around and plug into the new alternates and duplicates storage logically correctly, so a bunk of jank will clear out. There will be a couple of weekly follow-ups later as well that will ensure certain alternate decisions are remembered better. Please keep me updated on how this goes for you as I roll more of this out.

>>13055 >>13038 Thank you for these reports. Yeah, it looks like Pixiv changed up their internal API again. A user has lined up a good parser that I will fold into 358. As for yiff.party, I am no expert with it, but in some testing I have been doing with another user's new parser, I have found the server very unreliable. It is always under heavy load and often just cuts off connections, including for media content. For some reason my network engine is not picking some of these errors up and is delivering broken json and jpegs–I assume Yiff is not delivering Content-Length header or something, so I don't know final expected content length. I will keep looking at this a bit and likely fold in this newer better parser in the coming weeks. My engine deals with connection errors better in recent weeks, but there aren't excellent ways to detect half-done content. I will keep thinking about it. >>13047 Fantastic, thanks for letting me know.

If I have both the media viewer and file tag editing windows open, I can no longer scroll down on the media viewer to advance to the next image until I click on the media viewer to give it focus.

>>13074 Thank you for this report. The new unified shortcut processing test, which fixes various focus issues, accidentally does not have an exception for scroll events. I am 99.7% confident I can have this fixed for 358. In the meantime, one way of browsing from manage tags is to hit page up/down on the tag autocomplete input box–it should pass a previous/next event up to the media viewer to which it is attached.

Small bug: If you open the manage tags window in the dupe filter, when you close it, focus is set to the main hydrus window instead of the dupe filter. The main window pops up over. Windows 10.

Hey there, my client (v357) seems to reliably stall when I try to import https://e621.net/post/show/1915926 It downloads the image but never gets done with the processing. Not sure on what step exactly it gets stuck. But the client.exe will show around 30% CPU load. (Windows 10, 4 cores/threads system). It makes no difference if the url is hit by a subscription or a gallery downloader. I tried the same with various debug>reporters enabled and the log looked like this:
2019/07/02 14:05:08: Query "id:1915911..1915940" pre-work file login test. Login ok: True.
2019/07/02 14:05:08: Running read url_statuses
2019/07/02 14:05:08: CallToThread doing a job.
2019/07/02 14:05:09: Running read url_statuses
2019/07/02 14:05:09: Running read hash_status
2019/07/02 14:05:09: Running read in_inbox
2019/07/02 14:05:09: Running write content_updates
2019/07/02 14:05:09: Running read nums_pending
2019/07/02 14:05:09: Query "id:1915911..1915940" pre-work bandwidth test. Bandwidth ok: True.
2019/07/02 14:05:09: Query "id:1915911..1915940" pre-work file login test. Login ok: True.
2019/07/02 14:05:09: Running read url_statuses
2019/07/02 14:05:09: CallToThread doing a job.
2019/07/02 14:05:10: Running read url_statuses
2019/07/02 14:05:10: Running read url_statuses
2019/07/02 14:05:10: Running read url_statuses
2019/07/02 14:05:10: Running read hash_status
2019/07/02 14:05:10: CallToThread doing a job.
2019/07/02 14:05:12: Running read hash_status
2019/07/02 14:05:31: CallToThread doing a job.
2019/07/02 14:05:31: Running write serialisable
2019/07/02 14:05:31: Running write serialisable
2019/07/02 14:06:00: CallToThread doing a job.
2019/07/02 14:06:00: CallToThread doing a job.
2019/07/02 14:06:01: CallToThread doing a job.
2019/07/02 14:06:22: CallToThread doing a job.
2019/07/02 14:06:31: CallToThread doing a job.
2019/07/02 14:06:59: CallToThread doing a job.
2019/07/02 14:07:11: CallToThread doing a job.
2019/07/02 14:07:11: CallToThread doing a job.
2019/07/02 14:07:11: CallToThread doing a job.
2019/07/02 14:07:11: CallToThread doing a job.
2019/07/02 14:07:11: CallToThread doing a job.
2019/07/02 14:07:12: CallToThread doing a job.
2019/07/02 14:07:12: Running read file_system_predicates
2019/07/02 14:07:13: CallToThread doing a job.
2019/07/02 14:07:13: CallToThread doing a job.
2019/07/02 14:07:13: Running read serialisable_names
2019/07/02 14:07:13: Running read serialisable_names
2019/07/02 14:07:41: CallToThread doing a job.
2019/07/02 14:08:11: CallToThread doing a job.
2019/07/02 14:08:11: CallToThread doing a job.
2019/07/02 14:08:11: CallToThread doing a job.
2019/07/02 14:08:22: CallToThread doing a job.
2019/07/02 14:08:41: CallToThread doing a job.
2019/07/02 14:09:11: CallToThread doing a job.
2019/07/02 14:09:11: CallToThread doing a job.
2019/07/02 14:09:11: CallToThread doing a job.
2019/07/02 14:09:41: CallToThread doing a job.
2019/07/02 14:10:11: CallToThread doing a job.
2019/07/02 14:10:11: CallToThread doing a job.
2019/07/02 14:10:11: CallToThread doing a job.
2019/07/02 14:10:13: CallToThread doing a job.
2019/07/02 14:10:13: Running read serialisable_names
2019/07/02 14:10:13: CallToThread doing a job.
2019/07/02 14:10:13: Running read serialisable_names
2019/07/02 14:10:22: CallToThread doing a job.
2019/07/02 14:10:41: CallToThread doing a job.
2019/07/02 14:11:11: CallToThread doing a job.
2019/07/02 14:11:11: CallToThread doing a job.
2019/07/02 14:11:11: CallToThread doing a job.
2019/07/02 14:11:41: CallToThread doing a job.
Cheers.

>>13089 Thank you for this report. There's some funky focus rules with this stuff–I think when the always-on-top-of-parent manage tags window exits, it reverts focus on like the initial run to the most top-level window, rather than its parent. I think I have some catches for this in the main media viewer, but I presume the duplicate filter isn't handling it right. I will have a look at this today.

>>13091 Thank you for this report. I am afraid I do not get this problem either on my dev machine or my irl client. Do you sync with my PTR? Have you personally uploaded a number of new tag siblings or parents to it before? Can you try downloading the file manually, should be this link: https://static1.e621.net/data/01/9e/019e3fd020b4ea13e578576b128e5986.jpg And importing it just from your hard drive? If it imports, does it come in as a new file or 'already in db'? Do you see it having tags? If it imports ok as just a bare file, what happens if you queue up the URL again, now the file is imported? If the URL freezes up, what happens if you open a new url downloader and set its tag import options to parse no tags and try again–does it still freeze? If that still fails, what happens if you open network->downloader definitions->manage parsers and edit the e621 file page parser and put in the URL in the top-right of that edit panel to pull the data and then click 'test parse'? I get this: *** 1 RESULTS BEGIN ***

tag: character:mario
tag: character:tanooki mario
tag: creator:ultimatesol
downloadable/pursuable url (priority 75): https://static1.e621.net/data/01/9e/019e3fd020b4ea13e578576b128e5986.jpg
tag: 1up mushroom
tag: black eyes
tag: blue eyes
tag: brown skin
tag: cloud
tag: coin
tag: fungus
tag: hi res
tag: mushroom
tag: red skin
tag: star
tag: super leaf
tag: super mushroom
tag: super star
tag: video games
tag: white skin
md5 hash: 019e3fd020b4ea13e578576b128e5986
source time: 2019/06/26 19:00:00
tag: series:mario bros
tag: series:nintendo
associable/source url (priority 0): https://www.deviantart.com/ultimatesol/art/Super-Mario-3D-Land-272390217
tag: species:canid
tag: species:canine
tag: species:goomba
tag: species:human
tag: species:lakitu's cloud
tag: species:mammal
tag: species:raccoon dog
tag: species:tail goomba
tag: species:tanuki
tag: species:toad (mario)

*** RESULTS END ***

>>13096 > Do you sync with my PTR? Have you personally uploaded a number of new tag siblings or parents to it before? Yes, I do sync and I have uploaded new tags before. > Can you try downloading the file manually, should be this link: https://static1.e621.net/data/01/9e/019e3fd020b4ea13e578576b128e5986.jpg And importing it just from your hard drive? I used - file->import files->add files - selected the file - 1/1 files parsed successfully - import now and the page "import (0, 0/1) does not appear to do anything. Closing the page asks if I really want to close while it is still importing. > If that still fails, what happens if you open network->downloader definitions->manage parsers and edit the e621 file page parser and put in the URL in the top-right of that edit panel to pull the data and then click 'test parse'? I get this: I tried this just in case and the result at first glance looks like your's.

Deviantart login appears to have broken for me. I have been gone for about a month, so it is likely cookies have expired. However, attempting to refresh the login does not work, and returns: "invalid - The temporary variable 'validate_key' was not found!" I also tried turning the login off, and downloading some content that did not require login, then turning it on again, no luck.

>>13097 Thank you. Looks like it is definitely the file itself, then. If you are still up to chase this error down, can you try extracting a new fresh copy of hydrus to your desktop or something and then trying to import the file from your hard drive to that new, empty client? This will help us figure out if there is a hardware problem with your computer, like some weird bug in your graphics driver that means hydrus can't do some processing on file import on that file, or if there is an issue with your specific IRL client install db (which I strongly suspect). My best guess here is that on import the file has some similar-files metadata or something that is not inserting correctly into your db. Maybe something got janked up and your similar files search tree has a loop. I am gearing up now to put today's release out, so I don't have time to add it for 358, but I will try to fit in a 'file import report mode' or similar for 359 that will expose what is going on here so we can see better where your client is running into trouble.

>>13099 Thank you for this report. Sounds like they have changed their process. I will try to have a look at this next week.

>>13104 I'm totally up to chase the error down. I tried a fresh copy of 357 and the file imported fine. Using my existing db with the fresh copy showed the broken behaviour again. Updating to 358 (because the change log said something about db loops) made no difference. So as you suspected, it has to do with my db. If there's anything I can query on the sqlite db directly that'll help, let me know. I'm not afraid to get my hands dirty with the command line client.

>>13119 Great. Shut down the client and let's open up the sqlite3 in install_dir/db and then do this: (you should be able to copy/paste each line) .open client.db
ATTACH "client.master.db" AS cm;
ATTACH "client.caches.db" AS cc;
SELECT hash_id FROM hashes WHERE hash = x'8fdab765d788fb987fceee739bcfe3abdc0af747e45ef0e204b6e930739ff1aa';

(note this hash_id number. if it does not give anything, then .exit now and let me know)

SELECT * FROM shape_search_cache WHERE hash_id = (your_hash_id);
SELECT * FROM shape_perceptual_hashes NATURAL JOIN shape_perceptual_hash_map WHERE hash_id = (your_hash_id);
SELECT * FROM shape_perceptual_hashes WHERE phash = x'fb33c17a84e0966c';
SELECT phash_id FROM shape_perceptual_hash_map WHERE hash_id = (your_hash_id);

(note any phash_ids here, should be either one or none)

SELECT * FROM shape_vptree WHERE phash_id = (your_phash_id);
.exit And let me know what you get. It is possible the shape search stuff will just give you no results. But this will let us know if the file has been seen by your db–likely through PTR sync–and if it somehow already has shape data, or if the shape data is colliding for some odd reason with an existing file. Note if the error is located here, maybe due to a malformed search tree, there is an outside chance that database->regenerate->similar file search tree will fix this. It should take a couple of minutes or less. If the operation hangs indefinitely, then we have definitely found our problem.

>>13131
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open client.db
sqlite> ATTACH "client.master.db" AS cm;
sqlite> ATTACH "client.caches.db" AS cc;
sqlite> SELECT hash_id FROM hashes WHERE hash = x'8fdab765d788fb987fceee739bcfe3abdc0af747e45ef0e204b6e930739ff1aa';
30364954
sqlite> SELECT * FROM shape_search_cache WHERE hash_id = 30364954;
sqlite> SELECT * FROM shape_perceptual_hashes NATURAL JOIN shape_perceptual_hash_map WHERE hash_id = 30364954;
sqlite> SELECT * FROM shape_perceptual_hashes WHERE phash = x'fb33c17a84e0966c';
sqlite> SELECT phash_id FROM shape_perceptual_hash_map WHERE hash_id = 30364954;
sqlite>
I guess this means no results and there's no need to regenerate the similar file search tree? What's next?

>>13060 Just letting you know the deduplicate filtering works normally again for me in v358.

I've just updated from 354 to 358, and I can't use file lookup scripts anymore. I get this error every time: UnboundLocalError local variable 'f' referenced before assignment Traceback (most recent call last): File "include\HydrusThreading.py", line 342, in run callable( *args, **kwargs ) File "include\ClientGUITagSuggestions.py", line 483, in THREADFetchTags parse_results = script.DoQuery( job_key, file_identifier ) File "include\ClientParsing.py", line 2722, in DoQuery parsing_text = self.FetchParsingText( job_key, file_identifier ) File "include\ClientParsing.py", line 2688, in FetchParsingText if f is not None: UnboundLocalError: local variable 'f' referenced before assignment

>>13105 Unfortunately still not working for me. I tried just clicking "do login now", no dice. "The temporary variable 'csrf_token' was not found!" I also reset login (clear cookies), then "do login now", still no dice. Version 359. I'm playing with the login script now, I'll let you know if I figure anything out….

>>13105 >>13161 OK….. So after waiting a while, trying some more with no success, and toying with the login script to no success, it suddenly started working in the login script testing area sometimes, but wouldn't work for the real logins….. I then Reset the login completely, and tried login several times, no dice. After testing it probably 15 more times, I can get it to work now…. about 20% of the time, and I have absolutely no idea why. So it's working for me now…… I'm just left confused. Thanks!

Any idea what these two errors mean?

>>13143 So I enabled the new file import report and got this
2019/07/13 01:03:12: File import job starting work.
2019/07/13 01:03:12: File import job hash: 8fdab765d788fb987fceee739bcfe3abdc0af747e45ef0e204b6e930739ff1aa
2019/07/13 01:03:12: File import job pre-import status: ,
2019/07/13 01:03:12: File import job mime: image/jpg
2019/07/13 01:03:12: File import job testing for decompression bomb
2019/07/13 01:03:12: File import job file info: (655563, 1, 918, 1279, None, None, None)
2019/07/13 01:03:12: File import job generating thumbnail
2019/07/13 01:03:12: File import job generating phashes
And then it sits there.

>>13147 The deduplicate filtering isn't working properly anymore in v359. Distance 0 is giving me very dissimilar matches.

>>13172 >>13143 Hey, I am sorry I am getting back to this late. By that 'file import job generating phashes' being the last statement, it is definitely having trouble here simply using CPU/GPU time to generating similar search data for that file. The db appears fine. This is double-odd, as none of this code stands out as something that could hang. It basically loads the image and does some transforms and math on it–normally, if there were a probeem, it would error out with some decent text. It can seemingly load your image ok, since it figures out the resolution and generates the thumbnail ok. The image looks like a normal RGB jpeg, not like a 14,000x22,000 png or something that might blow up ram doing this. So, my guess here is it is something like your graphics driver is unhappy with some particular part of the shape of that file so that when it does DCT to it, or tries to greyscale it, it is hanging on that call. The image libraries I use send difficult image math to the GPU using OpenCL, and this sort of this has come up once or twice before. I can think of two things to try: 1) This sounds stupid, but update your graphics driver. If this happens to be a bug in your current version, it may be cleared up in the latest. 2) See if you can disable OpenCL acceleration for client.exe under your graphics driver settings, at least for this one file. You do this just like you can override FXAA and Vsync etc… for a game exe. For Nvidia, it is under the normal Nvidia Settings panel and then 'Manage 3D Settings', and then you make a new profile for hydrus's client.exe. Otherwise, I'll make a new 'similar files data generation report mode' so we can drill down to exactly what is hanging here for you.

>>13147 >>13186 Thank you for this report. Can you post an example pair of dissimilar images that are being matched for you, so I can check on my end? I don't mind nsfw. Also, if you load up one of these images in a search page and right-click->file relationships->find similar->exact match/speculative, does it give the same bad results in those cases, or does it give just the original file? Do either of the files in the example bad pair have duplicates or alternates already set? Could they be accidental false-positives?

>>13150 Hey, I am sorry for this. It should be fixed in the latest 359. Thank you for the report. >>13161 >>13163 Thank you for the feedback. As far as I can tell, DA is rolling out the new layout/login system/whatever on a pseudo-random A/B testing basis or something. It seems like once you are logged in, it will 'fix' you to the new layout for the duration of that login, but I have also seen some 500 Server Errors. If they do it like Pixiv did, I assume they'll slowly roll it all out to all users 100% of the time and ultimately retire the old layout. Phoneshit compatibility and dynamic website drawing seems to be the way of the future.

>>13167 I am not sure. I have seen this seemingly false 'db is locked' error on boot a couple of times before but was unable to pin it down. Is there any chance there is/was an old 'client.exe' process hanging around in Task Manager, unable to close itself (and tidy up the db locks) from a previous session? Or was this on a freshly booted computer? My best recommendation is to see if there is an old ui-hidden client.exe in Task Manager, and kill the process if so, and if not, try rebooting the computer. If you still have trouble, try going into install_dir/db and running the sqlite3 executable there. It will open up a new console that you can paste commands into. Do this process four times for the four 'client' .db files: .open client.db
SELECT * FROM sqlite_master LIMIT 1;
.exit That will try to open and read some simple info from each file (i.e. swap in 'client.db' for 'client.caches.db' and so on). That may end up silently 'clearing out' and bad lingering locks so you can boot again, or it may report a better error here. Let me know how you get on.

>>11542 I cant seem to run python scripts from open externally anymore. This is the command I'm using [python "C:\Users\*\My_Documents\_tiff_shortcuts\reader.py" "%path%"]. It opens a blank cmd window with python as its title for a split second. I did some testing and if I just have the python exe open with no other arguments it still doesnt work. I also tried a very simple script with no %path% argument which asked for an input, same thing. I talso tried a program that just copied over sys.argv into a txt file to see whats wrong but the file is unchanged.

>>13202 Thank you for this report. Can you try again with help->debug->report modes->subprocess report mode on? That should dump a heap of debug info into some popups and your log and help us see how the command is failing. Some of that will be private environment info, so don't post it all unredacted, but is there anything in there that pops out?

>>13199 Here's an example of two dissimilar matches at distance 0. In the duplicates tab I did a search and reviewed matches using the filter. >does it give the same bad results in those cases, or does it give just the original file? It has just the original file. >Do either of the files in the example bad pair have duplicates or alternates already set? Neither have relations previously set. >Could they be accidental false-positives? There are a large number of bad matches so this pair isn't an isolated indecent.

>>13203 I'm retarded, my previous script works just fine, the subprocess report showed me that I forgot to install a package which I needed. But something else I'm trying fucks it up - highlighted line in pic related. I think calling another subprocess or capturing the output of one causes issues. I don't really mind if it isn't fixed, ill try a python library to get the exif title instead. If you're wondering what the hell I'm trying to do, I'm adding tiff files to hydrus which have a title tag that I can plug into a dictionary to get the file path of my manga folders (I prefer not having manga archived and manga is stored outside of hydrus).

>>13198 No worries. Everyone's busy and there's no rush. What makes me doubt the graphics driver is the import working fine with a fresh database. Could there be anything else with my database holding back the import? Thousands of other files (yes, I am a hoarder) have imported fine since that particular one. So generally everything works.

>>13204 Thank you. The files do not get a potential pair for me, so this is not some legit false positive accident. There is definitely something going wrong in your db. I recommend you pause automatic idle time potential pair searching by clicking on the cog icon on the duplicates page. You'll probably want to completely reset your potential duplicates as well through the same menu, once we have figured this out. Can you please close your client and go to install_dir/db and run the sqlite3 executable to bring up the sqlite terminal, and then copy/paste these lines in one by one: .open client.db
ATTACH "client.master.db" as cm;
ATTACH "client.caches.db" as cc;

SELECT phash_id, hex( phash ) FROM hashes NATURAL JOIN shape_perceptual_hash_map NATURAL JOIN shape_perceptual_hashes WHERE hash IN ( x'232d97e6126a25b8ec4ae36f65b5e8ae08314e74d06f4fcbc72350a9fd3f5995', x'6ed82640fdd14643660cb4f50f022b5b0dd7a4e1c4ff3f6d430a5ed8e7c82fac' );

SELECT media_id, hash_id FROM duplicate_files, hashes ON (king_hash_id = hash_id ) WHERE hash IN ( x'232d97e6126a25b8ec4ae36f65b5e8ae08314e74d06f4fcbc72350a9fd3f5995', x'6ed82640fdd14643660cb4f50f022b5b0dd7a4e1c4ff3f6d430a5ed8e7c82fac' );

SELECT media_id FROM duplicate_file_members NATURAL JOIN hashes WHERE hash IN ( x'232d97e6126a25b8ec4ae36f65b5e8ae08314e74d06f4fcbc72350a9fd3f5995', x'6ed82640fdd14643660cb4f50f022b5b0dd7a4e1c4ff3f6d430a5ed8e7c82fac' );

(note those media_ids down, and put them in here a couple times):

SELECT * FROM duplicate_file_members WHERE media_id = (media_id here);

.exit Does that first SELECT statement give you: (some number)|EAE648A595946D8B (some other number)|E8EAD2E49595C583 Or does it give very different numbers, or a much bigger list? Does the duplicate_files, hashes SELECT give two lines, or less? Does the duplicate_file_members SELECT where you put the media_ids in give you one line per media_id, or more?

>>13205 Hmm, I am not sure what it doesn't like there. I don't know enough about subprocess details. I have not had luck with anything but subprocess.Popen. If you do something like this: p = subprocess.Popen( ['exiftool.exe', argv[1]], stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE, universal_newlines = True, encoding = 'utf-8' )

( stdout, stderr ) = p.communicate() That may work out better, I am not sure. Maybe doing the PIPE stuff is ok, since I am already doing that to call your process and there is some sys-related std-'chaining' issue going on? I dunno though, because it looks like the .run command just calls Popen anyway deeper into subprocess. Was that traceback you got hydrus-side written to the client.log in install_dir/db? Does it show the full trace there, so we can see the actual meat of what it was having trouble with?

>>13208 Shit, I forgot that it was ok in a fresh db. Unless something very odd is going on that is prohibiting the next message getting out or being flushed to the log or something, after that 'generating phashes' line should be 'File import job generated xxx phashes'. Since that message is not coming up, it suggests the hang is inside the pure-CPU graphics processing code that generates similar files phashes. This all happens before the db is opened for the actual file import. Ok, let's see how this new phash generating report mode goes in tomorrow's release.

I tried to import an epub but hydrus somehow turned it into a zip file. This is the file https://files.catbox.moe/d6005p.epub

>>13215 2019/07/18 19:42:36: File import job generating phashes
2019/07/18 19:42:36: phash generation: loading imageThat's all it does, eating up a CPU core.

>>13214 The same thing that I posted was written to the log, nothing more. Alright so I used a python library to get the title and that works just fine. Even the stdout in hydrus will show anything I print from the program. But it seems I cant load my json file. Nevermind, I think the correct solution to all my problems was to use full filepaths in my program. I.e. "C:\\Users\\*\\My_Documents\\_tiff_shortcuts\\data.json" instead of just data.json. Thanks for the help, the debug subprocess report was useful.

What happened to custom shortcuts editor? Seems like I can only see the reserved ones. They are still accessible and still work but seems like I can no longer edit them

When I copied the 360 folder over the 359 one all my files disappeared. I hit F5 and now it's thrashing my hard drive. Is it reconstructing the database? How badly did I fuck up?

>>13227 Ah, it looks like epubs are zip files, a bit like cbr/cbz: https://www.lifewire.com/what-is-an-epub-file-2621084 Most ebook formats are not supported yet in hydrus. Text is difficult for a bunch of reasons, and accurate mime (filetype) detection based on file content (which is how hydrus currently determines filetype) is a big one. You can leave the epub in there if you like, and when support eventually comes in I'll add a file maintenance job to recheck all current zips to see if they are really epubs and fix it automatically for you, or you can keep it out so your ebook reader can recognise it in the meantime.

>>13230 Ah, it looks like PIL (Pillow), a backup image library I use, completely spergs out when it tries to load that file! When I force it to load with PIL, I get the same CPU problem. I looked inside PIL, and I don't completely know what's going on, but it looks like the LOAD_TRUNCATED_IMAGES setting, which is putting a fake EOF in the load file call, is not being recognised, so the virtual file being loaded is just a truncated file plus an infinite number of EOFs. Can you check your options->media->BUGFIX: Load images with PIL setting for me? My guess is this is on, perhaps because you are/were a Linux user at some point? Uncheck it, and you should be good going forward. There was a while that PIL was more stable and reliable than OpenCV, and now I presume the opposite is true.

>>13236 This is now hidden behind help->advanced mode, as a lot of new users were stumbling into it by accident.

>>13237 I am very sorry you are having trouble. Can you talk more about your situation? When you say all your files disappeared, does it still seem like the client thinks you should have them (and the thumbs are not loading), or is it seemingly empty? Do you have 'system:inbox (some files)' or is it like 'system:inbox (0)')? Did it boot with your old session of page tabs, or just a fresh single empty page? Did it give the 'this looks like the first time you are running the client' popup message on boot? Did the 360 folder get run before you copied it over 359, or was it a completely fresh extract? If you check your install_dir_359/db folder, what are the size and created dates of the four client*.db files in there? If you go deeper into the client_files directory and its subdirectories, do you still have media files in there? Or have you migrated your database/files to another location? Do you have a backup of your 359 client?

Clicking on the button to change which site to search in a gallery downloader pops an error '<' not supported between instances of 'GalleryURLGenerator' and 'GalleryURLGenerator'

>>13254 Sweet, that helped! Thank you! It was indeed on, though I have always been on Windows. But I've been using Hydrus for so long, I might have just turned that on myself who knows when and forgotten about it. Speaking of which, is there an option I might have set at some point (that I am unable to find now) which disabled Hydrus from playing sound in videos? I feel like it did that at some point, but it hasn't for ages now. Is that a feature? Or am I just too daft to figure it out?

>>13256 It seems like it was a momentary issue. After killing the client and restarting it all my images were present. Before then it was like they weren't even there, but things are fine now.

>>13201 >>13201 No, it's alright.. The client opened up normally after I clicked OK on both those errors and they never appeared again since that time. I was just curious about the meaning of those errors.

I don't seem to be able to administrate my remote files service at all. I can upload files with the automatically-created administrator account just fine, but attempting to create new accounts for the file service, modify the account types, or ask who uploaded a file causes the client to lock up indefinitely. My first guess was a version mismatch, locally I'm running 361 on windows where my remote server is running 360 on ubuntu 19.04. I downgraded my windows client to 360, then upgraded back to 361 after running into the same issue. I've had no issues administrating the remote tags service running on the same server. As an aside, the server is on 360 because I was unable to get 361 working on the server, even after reimaging the entire box and dropping a fresh copy of hydrus from the github on it. On 361 the server will run, but doesn't seem to listen on any ports. Any ideas for either of these?

>>13262 Thank you for this report. I am sorry for the inconvenience. I am 99.7% sure I can fix this for 362.

>>13264 There's no native audio yet. It proved very popular in the last big job poll, so I expect to give it a proper go this year. Within the next 3-5 weeks I hope to have 'has audio' metadata sorted, which will display in the media viewer as a speaker icon or something.

>>13277 Ok, thank you for the update. Please let me know if it happens again and/or you discover more about this.

>>13291 I fixed a related error here in the past week or two–hitting 'forget it' on the 'a client is already running, do you want to wait or forget it?' boot dialog was accidentally clearing out the 'is running' file, SHIT, so a subsequent attempt to run the client was ploughing ahead regardless of any existing instance and throwing this error. This may be something of what you were seeing here.

>>13309 Thank you for this report. I am sorry you are running into this trouble. The service admin stuff has been caught up in some of the new network error handling recently, where if the network engine wants to wait because of a recent error, the old admin service code is now waiting to retry connections silently in the background. And because admin stuff is shit old code, it often waits on the ui thread, so it is locking up the client for x minutes until the connections time out. I suspect this is what you are seeing. The job of making this workflow nice is bigger than I can quickly do, but I will go through all the admin commands and ensure they cancel snappily on a failed connection. I did change how services start in v361, I had hoped for the better. I assume you have access to the file repository's server.log? Does it print any errors right after starting about being unable to map ports? I have a job this week to improve how this works further, improving error reporting.

>>13344 I went back and checked the logs from the v361 instance. Now that I'm looking more closely, it actually doesn't print "Starting services…" at any point until after I keyboard interrupt it, no matter how long I wait after starting the server on that particular attempt. Every one of the logs looks basically the same as this one with different timestamps: 2019/07/27 22:22:23: Initialising controller…
2019/07/27 22:22:23: Initialising db…
2019/07/27 22:22:24: Initialising daemons…
2019/07/27 22:22:24: Server is running. Press Ctrl+C to quit.
2019/07/27 22:23:03: Received a keyboard interrupt…
2019/07/27 22:23:03: Shutting down controller…
2019/07/27 22:23:03: Shutting down daemons…
2019/07/27 22:23:03: Shutting down db…
2019/07/27 22:23:04: Starting services…
2019/07/27 22:23:04: Services started
2019/07/27 22:23:04: Stopping services…

>>13345 Thank you, this helps.

>>13343 Yeah, that's possibly what caused it but I don't remember for sure as it only happened once for quite some time now. Anyway, I'll update to the latest version and I'll make sure to let you know if that ever happens again.

Hey Hdev, hope you had a Merry Christmas. This isn't a bug on your end because it broke while on the same version 372, It appears that the deripbooru downloaders are totally useless now and ignores all files. As just a test, running a search for "Tutorial" only pulls 15 results, but then ignore's all 15. Meanwhile there's hundreds of files listed under "Tutorial" on derpibooru. This isn't just a one off case, any tag punched into the search wont scrape for the site. I hope it's not unusable forever like the pixiv change, they were working fine just last week. So if you've got time could you look into the parsers?

>>13487 Thanks. Apparently they shifted their back-end around or something. A user sent me an updated downloader, attached, which I will roll into 379. I'll do a couple tests as I add it, but I am not a user of derpibooru, so let me know how it works for you.

First of all thank you, this is great software. I didn't see a mention for this minor error in the thread. Only a few images have this problem (actually all under my "series:umihara kawase" tag), on image open everything works fine, but after a few seconds a popup shows up anyways saying FileMissingException. I have tried all the database options hoping something would iron this problem out. https://pastebin.com/iNqB5G5w There definitely isn't a file of that name at the location stated so the error makes sense but what doesn't is why it still works anyways and obviously has a different filename as in pic related

>>13503 I will update on this and say that it seems like my entire database is corrupting. It cant find a considerable amount of my files anymore, they no longer open despite errors. I do have a backup but I dont know how bad it is itself. I tell hydrus to vacuum the database and it instantly completes saying its done but i just have a hard time believing it does it so fast and nothing changes. what can I do?

>>13503 >>13524 nevermind sorry i think this is unrelated, there are quite a bit of 0 byte files on my hard disk now. its going down, down to china town, REAL TIME

>>13495 so just an update on this, the scraper works but the one the user gave you is set to download with default filters on, as if you didnt make an account. there's no difference between derpi scraping and derpi no-filter scraping right now. Not sure why it even picks up 3 results since none of them are tagged right but that's on derpi's end and none are explicit. There's no workaround for this with a login that i could find, like on deviantart

I'm not certain where exactly it was introduced (probably in the past 3 or so releases), but there's a bit of a regression: The trash display, when using "system:inbox"-selection displays ALL inbox entries (that is, both those in "trash" AND the ones in "my files"). It used to display just the inbox entries that were trashed (which is the desireable behaviour).

>>13529 Derpi scraping is working here, Though I can't vouch for filters working or not. HOWEVER, I can vouch that I'm not getting any tags at all for anything downloaded from derpi currently.

The duplicate filter doesn't zoom in/out to the center of the screen, more like lower right. At 2560x1440 resolution.

>>13526 Hey, I am sorry, I am just coming back to this thread after a period of not keeping up correctly with my messages. I have done significant work with other users in these sorts of situations, so please feel free to email me or hit me on discord or just post here on any further issues you are having. I'm happy to work one-on-one. There is a help file in hydrus under install_dir/db/help my db is broke.txt that will walk you through various hard drive error recovery situations, particularly if you are not familiar with chkdsk and other tools.

>>13551 >>13529 Thank you. I am afraid I am not a user of derpibooru and did not write the new parser, so I am a bit ignorant. I think I saw some conversation somewhere that the new site doesn't actually support the new filterless search stuff? Or that the format for that has changed somehow? I will hit up the guys on the discord who made this to see if there is an update.

>>13532 Thank you for this report. This should be fixed now. >>13554 I have noticed something like this in the normal media viewer too. I've been pulling all that code to pieces to get mpv integrated, so I think I messed some mouse reinitialisation coordinates up or something. Thank you for the confirmation report, I will look into it.

>>13559 Unfortunately, I am told the 'no filter' search strategy no longer works, so I think your best bet now is to use Hydrus Companion to copy your web browser's logged-in derpibooru cookies across to Hydrus, and then it will (presumably?) inherit your account's filtering preferences: https://gitgud.io/prkc/hydrus-companion >>13551 I just did a test download, and I am afraid I get all tags ok–this sounds stupid, but can you double-check your tag import options? If it is still bad, can you do a derpibooru parser test under network->downloader definitions->manage parsers?

Apologies if this is a duplicate bug report. Shift select is broken sometimes. Select the first file -> select the last, only last gets selected. Then select the first, only first selected. Select the last again and then finally all the files are selected. Linux, v380

Not sure whether to put this as a bug or feature request. Collecting files while under 'my tags' collects them based on all tags (my files + public tag repo).

Apologies if this has already been reported. Trying to load a .gif/.webm gives the following error on linux: Non-C locale detected. This is not supported. Call 'setlocale(LC_NUMERIC, "C");' in your code. zsh: segmentation fault (core dumped) hydrus-client going into the settings and switching to the default hydrus viewer solves this issue for me, but I thought it best to mention it. Linux v381 but I'm pretty sure this has been going on since at least v380 or maybe even earlier

Does nijie work for anyone? I tried with a new install of the latest version and still get no successful connection. Knowing if it's just me or not would help.

When I try to add a tag that's the old sibling in a sibling pair, instead of adding the tag I entered in and saying that it will be displayed as the new sibling, it just adds the new sibling, even when I manually select the old sibling in the autocomplete panel, and won't let me add the old sibling at all. This is a huge problem if the sibling pair was ever removed, and I don't know what's causing it. It happens with both "my tags" and "public tag repository" sibling pairs, and I believe this is a recently introduced bug.

>>13573 I get tags on derpi booru downloads but the hydrus companion does nothing from what I see for any website, not just derpi. How would you plug the cookies into hydrus when it doesnt have a login manager?

>>13575 Thank you for this report. I have noticed this myself. I can't figure out if it is my code or my mouse driver (I have shift mapped on a Logitech mouse) or Qt in general–I have another Qt program that also does the same thing, again only sometimes. I can't obviously think of any place where I say 'ok, stop accepting shift for a bit', although I may be mistaken. When this happens again, if you have another program in Qt with a list, can you check that at the same time? Does this only happen for you on thumbnail select, or can it also happen on column-ed lists, like the list of queries in a download page? That is where I see it most, when I am trying to shift-select six completed watchers or similar so I can delete them.

>>13577 Thank you. Unfortunately it is all merged at the moment. I had hoped to be putting some more time into per-service tag presentation and handling options in the past couple of weeks, but Qt boshed my whole schedule. I am not sad at all and am really pleased with Qt, but a bunch of those tag plans are now on the back burner. Ideally, I want a whole new set of UI for saying which services apply and take precedence in different places (like thumbnail banners and tag siblings, as well). I am not sure when I will get to this.

>>13581 I am sorry for the trouble. This should be fixed in v382 now. It is a non-Windows problem with loading MPV, triggered accidentally for you because your system happens to have MPV access when I did not expect it to. If 382 still give you trouble, please let me know.

>>13583 I do not know much about the site, but I feel like it has 'locked up' a bit since I last visited it. Do you have to have a login now to see anything? Pixiv went through an extensive round of updates over the past six months, perhaps nijie have as well, and this has broken the downloader in hydrus. I am afraid I do not see a new downloader either in the shared repository: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System/All-in-Ones/Single-Sites What do you get when you try to download something? No results at all, or a parsing error, or a connection error?

>>13584 Is this in the 'manage tags' dialog? Some time ago I moved some automatic sibling/parent action options to the 'cog' icon button there–is your dialog set up to auto-replace tag siblings on a pend/add? If it is turned off and still auto-replacing, can you say exactly the series of button-presses you are using to add the tag? Is it an enter press on an arrow up/down selection, or a mouse double-click, or anything else?

>>13587 Were you able to set up the Hydrus Client API for the Companion to talk to? Was it ever able to see the Client API? Under the Companion page here: https://gitgud.io/prkc/hydrus-companion There is a section 'connecting the extension to Hydrus', which goes through the steps. You'll have to turn the Client API on and then grant the Companion permission to access your client.

>>13593 You're right. I wasn't aware that there was an option for that. I must've accidentally enabled it sometime and didn't notice. My bad, although I often forget that not all options are in the options menu, and that some are hidden in gears in different parts of Hydrus. However, I recently encountered another bug. This one's been bothering me since I started using Hydrus at around V370, but it happened again just yesterday, and I'm more confident that this bug is legitimate because of how strange it is, but I'm welcome to being proven wrong again. It helps me learn. After adding a child-tag to "male" or "female" in the local tag service, the tag "gender:male" or "gender:female", respectively, will be added to all files that I have tagged with the child-tag at the time of the relationship's creation. That's what I've gathered after doing a bit of experimenting. gender:male and gender:female aren't even tags that are in my local tag service at all. They're the new-siblings of "male" and "female" respectively in the PTR, but the PTR shouldn't be interfering with the local tag service like this. This doesn't really seem like something that would be an option either, but after looking around for a little bit just in case (and keeping my eyes out for gears this time) the closest options I've found to what's happening here are "suggest all parents for all services" and "apply all siblings to all services", both of which I've never had enabled.

>>13595 It is possible that setting was reset at some point when a system was rewritten, I am not sure. The whole siblings/parents system is a bit of a mess behind the scenes, so I am sorry for the inconvenience. For your multi-service parents situation, please uncheck the 'suggest all parents for all services' checkbox under options->tags. At the moment, the only parents options you have are 'each service by itself' or 'all merged together', and this controls it. I notice that 'suggest' is the now wrong word here–it should be 'apply'. I suspect the option got written into a logic overhaul at once point, as it used to always be 'all merged together' all the time. I very much want to improve the settings for which services provide siblings and parents to other services. I want to give you full control, letting you turn them completely on or off or override or whatever on a per-service-to-service basis. I had planned to do this in November/December, and then unfortunately Qt boshed my big tag plans. I am not sure when I will get to it, but the once-prototype logic here is creaking at the seams and I really would like to overhaul it.

>>13600 That option already is disabled and it always has been. But this isn't exactly working how that option works anyway, because it only applies the parents when I create a new relationship, and only with files that are tagged with the child-tag. Also, it's treating the new-siblings from the PTR as if it's a parent in the local tag service, which is why "male" is getting gender:male added to it, instead of being replaced by it. That's why I'm so confused. After doing a bit more experimenting, I discovered that this appears to happen with all tags, not just "male" and "female", so I'll try to clearly explain what's going on: The public tag repository has old-sibling tags that are replaced by new-sibling tags. Some of those tags exist in my local tag service as tags that I use and are not part of a sibling pair. For example, I use the tag "japanese language" in my local tag service. In the PTR "japanese language" is an old sibling that is replaced by its new sibling "language:japanese" but in my local tag service, "language:japanese" doesn't exist at all, and "japanese language" is not an old-sibling of it. That's the tag I use. a few days ago, I decided to make this tag the parent tag of "japanese text" (a tag that I've been using already) in my local tag service. When I did that, the bug took effect. Every file that I had tagged with "japanese text" (the new child-tag) gets the tag "language:japanese" (the new-sibling of its parent tag in the PTR but NOT in my local tag service) added to it. In other words, when I create a new parent-child relationship in my local tag service, everything that was tagged with the child-tag when I created that relationship gets tagged with the parent-tag's new-sibling in the PTR if it has one. the "suggest all parents for all services" option is not enabled when this happens, and I have never had this option enabled. The bug treats the new-sibling of a tag in the PTR as if it is the parent the tag in my local tag service, but it only applies them when I create a new relationship between this tag, and another, and it only adds this tag from the PTR to files that I have tagged with the child-tag (not just the parent tag) in my local tag service at the time of the relationship's creation. But again. Both cross-service relationship options are disabled when this happens, so those options should not be the cause of what's happening. I don't think it would make sense for that to be the cause anyway, because the bug is treating new-sibling tags as if they are parent-tags, which I don't think there's any option for in Hydrus. Sorry if this still isn't a clear enough explanation. I wish I could just diagnose the problem myself, but I don't know Python, and I've looked at every option I could find to make sure that there isn't some weird "make siblings in remote tag services temporarily pretend to be parents in local tag services when you make a new parent-child relationship in a local tag service" option somewhere that I missed, but I couldn't find any, so I'm pretty sure that it's actually a bug this time.

>>13601 I just remembered that I should include steps to reproduce, since they'll help you see what I'm talking about. 1. open a client that's connected the the Public tag Repository and also has a local tag service as well. Make sure that "suggest all parents for all services" is disabled, just in case. 2. In the local tag service (I think by default it's called "my tags") add the tag "japanese text" to a file 3. Go to "manage tag parents" and in the local tag service, make "japanese language" a new parent tag of "japanese text" 4. Go back to the file you added "japanese text" to and you'll see the results of the bug in that file's tags: This file will have the tag "japanese language" added to it, as it should, because you just made this tag a parent of "japanese text". However it will also add the tag "language:japanese" to the file. This tag is not supposed to be added. You didn't enter it manually, and you didn't make "language:japanese" a parent of "japanese text". "language:japanese" is the new-sibling of "japanese language" in the PTR, but this isn't the public tag repository, and "suggest all parents for all services" is disabled, so there SHOULDN'T be any interference from there anyway, but there is. Even if that option was enabled, "language:japanese" is the new-sibling of "japanese language" in the PTR, NOT the parent of it, like it's acting like here.

>>13589 >I have another Qt program that also does the same thing, again only sometimes. It's probably QT then, heh. I tried searching through their bug tracker but there's so many selection bugs that it would take an hour to find a report. https://bugreports.qt.io/browse/QDS-526?jql=text%20~%20%22shift%20select%22

Found a bug on 381 that's still on 383. I censored all tags but one to make it easier to sort that tag, which caused all tags to simply disappear from the selection tags display. After I set the tags to show again the selection display is still empty in all pages, even the ones from before the change. When I actually open a picture the tags display properly on the left, so there's nothing wrong with the censorship options, just the tag display itself.

>>13594 I get this. copying the API numbers into the companion didnt do anything helpful, following the guide in the github. >API access test results <A network error occurred while querying API version. <A network error occurred while querying API permissions.

>>13592 I'm sorry for not answering earlier, this has been a very busy week. I get a connection error when trying to download anything. It used to work flawlessly up to 2 week ago at least. (I mostly check during week-end).

BUG: Open a video in media viewer. Zoom out (make it smaller) to 20%, zoom in to 100% again, repeat this many times. Video will move lower on screen every time. It does not center properly to the middle of the screen. Doesn't happen with images. BUG: The option "Remove files from view when they are sent to the trash" does not work on downloader pages or import pages.

>>11542 two write holes so far resulting in partially corrupted DBs on linux. haven't figured out the issue yet, my intuition tells me btrfs. I'm in the process of migrating to ext4 (not sure why I was on btrfs, heh, maybe for reflink support). I got a checksum error the first time, then I used quicksync for a fresh db and got another checksum problem while updating the public tag repo. I know how to recover an sql db so, w/e. nvme drive that is only a 3-4 months old so likely not a bad drive, and a pci-e card so likely not a bad controller or cable.

>>13659 uh oh. don't want to speak too soon, but I switched over to ext4. ran into the db corruption bug again while syncing the repo. happened on v383 and v384. hasn't happened yet on v382

>>13659 >>13660 Ignore these, the quicksync full database has an b-tree error in client.master.db. going to rebuild a new db from scratch and see if my issue really was a btrfs issue (I had multiple power outages which I'm pretty sure caused the write hole)

>>13602 >>13601 Thank you, I am sorry for the late reply. I have copied all this information down and will see if I can fix this in the near future.

>>13604 Damn. Thank you, and let me know if you discover anything new here. If it is triggered by an earlier ctrl+click or something, maybe we can figure it out. >>13613 Thank you for this report. I will look into this bug. Does this fix itself after a client restart?

>>13614 Thank you for this, I am sorry for the late reply. Could this be a firewall issue? Do you have a third-party firewall that is set to automatically block any non-whitelisted executable? It should be client.exe in the hydrus install dir trying to host here, if you need to add permissions. A good way to test the visibility of the client api is to open, in your case, http://127.0.0.1:45869 in your web browser. This should present a simple welcome page letting you know it is running ok, or perhaps a more helpful error page. Also, you may want to check your install_dir/db/client…log file and scroll to the timestamps where you did tests. There may be some logged info, or again perhaps some errors about trying to bind the port. If you only mean to connect to the client api from the same computer, I recommend you uncheck 'allow non-local connections' so other computers on your network cannot see it.

>>13621 No worries, I am sorry to say I am in the same situation, ha ha. If you check the gallery/file icon buttons (they launch a list of URLs attempted and the results) on your downloader that gets the error, can you find the URL row that errored? It is probably top of the list. Right-click->copy notes to copy all the error info, and can you then paste it here or email it to me, so I can have a look? If this is a legit network connection error, then I hesitantly guess this may be something other than hydrus–could you have a firewall anywhere stopping access? I assume it works in your browser. Maybe the website has a rule to disconnect a hydrus client, but usually you do not get an error in that case, but something more like a successful 503.

>>13649 Thank you for these reports. I will check out the zooming issue. For the remove files issue, I had thought I had fixed this some weeks ago, and it works ok on my dev machine here. Is there any chance the downloader page here is very old? Pages secretly keep a file domain that they use to judge how to make various display decisions, and older downloader pages may still be using the 'all local files' domain rather than 'my files'. Does it still happen for you on a fresh downloader page?

>>13662 Thank you for these reports. I am sorry for the inconvenience. I will let the QuickSync lad know his client.master.db has a problem. If I haven't told you before, there is some background reading on this topic under "install_dir/db/help my db is broke.txt" if that is helpful. Let me know if you discover any more here.

>>13676 >I will look into this bug. Well, I'm retarded. I did some tests and stumbled upon pic related.

I added a third tag repository to stage my changes to the PTR, but when I go to use the "siblings" feature from tag management, every time I go in the order switches for "local" and the one I added, and it always selects the wrong repository. Also, speaking of the siblings and parents feature, it would be nice to load them from the database based on context. Like if I am on the "local" repository, don't load everything from PTR because it dramatically increases the load time.

I'm going to count the storing credentials in plain text a bug. For the love of all things holy, please. https://pypi.org/project/keyring/ I found this, but I haven't used it. If anyone else has suggestions. I would try to do it myself, but I am finding it nigh impossible to navigate the source code.

>>13679 >For the remove files issue, I had thought I had fixed this some weeks ago, and it works ok on my dev machine here. Is there any chance the downloader page here is very old? Pages secretly keep a file domain that they use to judge how to make various display decisions, and older downloader pages may still be using the 'all local files' domain rather than 'my files'. Does it still happen for you on a fresh downloader page? It happens on the import page (for example when you drag and drop images to the client).

>>13688 Great, thank you for letting me know. >>13692 Thank you for this report. I will make sure those tabs always comes in the same order. Yeah, I am sorry about the lag here. Siblings and Parents are way overdue a major data overhaul. I was hoping to fit some work in Q4 2019, but then Qt interrupted. I hope to get back to it in the near future, at least just to boost some speeds here and there. >>13696 Thank you for this report. I generally agree. I would like to do something better, but it got pushed back by other priorities. For now, as I think the UI says, I recommend you use throwaway accounts with hydrus (and I still will, once this is more secure, in case any websites start frowning on user accounts that use hydrus). I will look into that keyring library, thank you. If I can offload the difficult part of this, all the better. Or perhaps something like this, which will let me store it locally and support portable clients that will move from one OS to another: https://pypi.org/project/password_manager/ >>13697 Great, thank you! I'll check it out.

Hydrus has stopped opening, nothing happens straight up when I try opening it. Looking in the db client log files, I find: 2020/02/25 22:30:44: hydrus client started 2020/02/25 22:30:44: hydrus client failed 2020/02/25 22:30:44: Traceback (most recent call last): File "client.py", line 183, in <module> from include import ClientController File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module File "include\ClientController.py", line 9, in <module> File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module File "include\ClientCaches.py", line 1, in <module> File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module File "include\ClientFiles.py", line 2, in <module> File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module File "include\ClientImageHandling.py", line 4, in <module> File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module File "site-packages\cv2\init.py", line 3, in <module> ImportError: DLL load failed: The specified module could not be found. As the first instance of the error, I updated hydrus to the latest as of this post and… 2020/02/29 23:49:31: hydrus client shut down 2020/02/29 23:53:31: hydrus client started 2020/02/29 23:53:31: hydrus client failed 2020/02/29 23:53:31: Traceback (most recent call last): File "client.py", line 185, in <module> from include import ClientController File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module File "include\ClientController.py", line 10, in <module> from . import ClientCaches File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module File "include\ClientCaches.py", line 1, in <module> from . import ClientFiles File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module File "include\ClientFiles.py", line 2, in <module> from . import ClientImageHandling File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module File "include\ClientImageHandling.py", line 4, in <module> import cv2 File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module File "site-packages\cv2\init.py", line 3, in <module> ImportError: DLL load failed: The specified module could not be found. 2020/02/29 23:53:31: hydrus client shut down I still get this. Not sure if a bug, or I just did something dumb.

>>13718 Thank you for this report. That error is complaining about a missing dll related to OpenCV, which does most of the image loading and handling in the program. Since you are using Windows, in your hydrus install, you should have a 'cv' directory with a 'cv2.cp36-win_amd64.pyd' file in it, about 60MB. That's probably the one it is complaining about, but it could also be a different dll or pyd in the base install directory. Is there any chance you either deleted a file, did not fully extract/install, or maybe had a anti-virus program get a false positive and remove a file? The best fix here is probably do simply re-extract/install, just as if you were updating. This should fill in the missing file, and/or perhaps show a popup you missed before from anti-virus if that was the problem. If that does not seem to do it, can you extract a fresh release to your desktop and run that, just a completely new database? If that is ok, is there a file in the working db that isn't in your existing release? If you have updated from a very old version of the program, it is possible there is a conflicting dll from an old version in your install directory. The solution to this problem is to do a 'clean' install, which is to: Make a backup, just in case it goes wrong! Delete everything in your install directory except for the 'db' folder and all its contents. Re-extract/install. Let me know how you get on.

>>13721 I don't see anything funny with my antivirus. I tried doing a fresh install with a new database, I get the same error. 2020/03/01 19:30:44: hydrus client started 2020/03/01 19:30:44: hydrus client failed 2020/03/01 19:30:44: Traceback (most recent call last): File "client.py", line 185, in <module> from include import ClientController File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module File "include\ClientController.py", line 10, in <module> from . import ClientCaches File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module File "include\ClientCaches.py", line 1, in <module> from . import ClientFiles File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module File "include\ClientFiles.py", line 2, in <module> from . import ClientImageHandling File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module File "include\ClientImageHandling.py", line 4, in <module> import cv2 File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module File "site-packages\cv2\init.py", line 3, in <module> ImportError: DLL load failed: The specified module could not be found. 2020/03/01 19:30:44: hydrus client shut down I'm running Windows 10 if that helps. I did try updating my python to the latest version, didn't help.

When processing duplicates, when you mark two things as identical, it only copies information between the two and not between all duplicates. If A and B are identical duplicates, and A and C are identical duplicates, it follows that B and C are as well. I noticed this while cleaning up.

>>11542 Didn't see the bugs thread last time I posted. The latest e621 update doesn't work with hydrus. I don't know exactly how to explain it, but someone else seemed to know what they were talking about in the last release thread.

e621 changed their site to be more in line with others like gelbooru. Unfortunately, this breaks the Hydrus downloader as an unintended consequence. Note that there is a new default tag blacklist for anons, so a login script is needed for e621 to view all content now.

>>11542 v387 hydrus uses too much vram when not in use, probably need to flush mpv. I'll be at 10-15% (with pages open, but no selections or image viewer) vram usage until I restart hydrus. This is problematic for me because I need that vram when gpgpu training, or for others they might need it when gaming. It would be nice to be able to open an empty page and get 0% gpu usage despite active pages that aren't focused. there's also lag when going backwards sometimes in the image viewer + archive/select filter. probably need to keep images in the cache for backwards navigation.

If something with a duration comes up during a slideshow that has less duration than the slideshow time, it won't loop until the slideshow time expires. A 4 second gif will only play once during a 10 second slideshow and pause until the next file comes up, rather than looping if it's shorter than the slideshow time and has time before the slideshow needs to move on.

>>13754 I should add that it hasn't happened since, and maybe it didn't happen at all. It's possible I had another app taking up gpu resources and didn't notice at the time. I'll report back if I notice it happening again.

>>11542 semi-important bug: new pages don't remember whether you are in the trash or my files. if you duplicate a page while in the trash, or right click on a tag and "open a new search page for" it will open the new page in your files. this could be problematic, say you have the tag "processing files" and you've deleted files. you went over them to make sure you want to delete them, but somewhere along the line while navigating the client forgot you were in trash, you go to purge those files from your trash and but you end up deleting all files tagged under that tag. other bugs: the autocomplete freaks out sometimes and won't give me back the system options despite clearing the input box. I can't manually type in system options either. system:everything is missing from the list. namespace:* does a wildcard search while namespace: converts to namespace:*anything* I use a tiling wm so whenever I open the media viewer it resizes the main hydrus client to make space. hydrus doesn't remember the page vertical position so it ends up scrolling down whenever I open/close an image and I'm constantly losing my place. An option to reuse the opened media viewer when opening files would also be helpful. >>13754 >>13765 I think I found the culprit here. Pretty sure it was firefox leaking vram. Sorry about the false report.

Back with the late replies, I am sorry! >>13728 Are you on Windows 10 N? This I understand is a version of Windows that does not have some Windows Media components. If this is true, then to do some media stuff you need a feature pack, I think this is it: https://support.microsoft.com/en-us/help/3010081/media-feature-pack-for-windows-10-n-and-windows-10-kn-editions

>>13742 Unfortunately, this version of the duplicates system does not do chained or synchronous content merge. The next big iteration of this system will fill in all the gaps here and hopefully allow you set to set up synchronous merge, so that future changes will also propagate to all dupes. >>13744 Thank you, should be fixed in the latest version. >>13758 Thank you, I had not considered this! I will fix this.

>>13754 >>13765 >>13770 No worries about the vram, please let me know if you discover anything more like this, mpv is new for me, and this stuff is always useful. Thank you for these reports. I will make sure that duplicate page and 'open a new search page for' propagate the file domain. I hate this autocomplete bug, I haven't had time to look at it properly. And yeah, unfortunately you cannot type in system tags for now–they are secretly a completely different thing that don't work in the text autocomplete yet. I'll check out the namespace:* thing–it should be functionally equivalent to *anything*, just way less efficient, so if it is appearing, I'll make sure it is hidden. That's an interesting problem with the tiling WM. I'll see if I can anchor a resizing scrollbar to the current top position or something. That code is still a bit jank right now from the wx->Qt switchover, so this may have to wait a short while. I will also make a job to add a checkbox for 'only allow one media viewer, if I launch a file when a media viewer is already open, re-use that media viewer'.

>>13804 >I will also make a job to add a checkbox for 'only allow one media viewer, if I launch a file when a media viewer is already open, re-use that media viewer'. would be nice to have a right click firefox "open in new viewer" and "open in current viewer", then have the shortcut system to decide what your double click action does

I'm using the Linux version of Hydrus. I'm unable to upload any parent/child tag pairs to the PTR. Attempting to do so results in the client starting to upload, but them stopping before it actually uploads anything, then locking up completely and forcing me to kill the client process. Sometimes the client won't end up freezing completely but instead, the upload just stays there making no progress forever until I close Hydrus. When I close Hydrus after it does this, the client process doesn't close properly which results in needing to go into the system monitor and kill the process directly (it won't respond to "end" requests). Oddly enough, this doesn't apply to adding tags to posts. Those upload just fine. It only applies to adding parent/child relationships, and it happens every time. I don't know if it also happens with siblings. >pic related

>>13820 btw in the pic it looks like it's uploading at 12B/s but that actually goes away after a few more seconds and then it just stops uploading entirely.

Whilst making a login script, I ran in to a few errors, first, while doing a test login, hitting "Review" failed and turned up this:
AttributeError
'PySide2.QtWidgets.QPlainTextEdit' object has no attribute 'SetEditable'
File "include\ClientGUICommon.py", line 209, in EventButton
self._func( *self._args, **self._kwargs )
File "include\ClientGUILogin.py", line 1826, in _ReviewTestResult
panel = ReviewTestResultPanel( frame, test_result )
File "include\ClientGUILogin.py", line 1197, in __init__
self._cookies.SetEditable( False )
Second, when I tried to edit content parsers in a login step, it failed with
AttributeError
'PySide2.QtWidgets.QLineEdit' object has no attribute 'SetValue'
File "include\ClientGUICommon.py", line 209, in EventButton
self._func( *self._args, **self._kwargs )
File "include\ClientGUIParsing.py", line 2301, in _Edit
panel = EditContentParserPanel( dlg, content_parser, test_context, self._permitted_content_types )
File "include\ClientGUIParsing.py", line 1946, in __init__
self._temp_variable_name.SetValue( temp_variable_name )


>>13806 That's a much better answer, thank you! >>13820 >>13821 Thank you for this report. Does the parent apply to millions of tags, say something like '1 female'->'gender:female'? When you upload a parent to the PTR, it is considered 'pre-accepted' to your local client, and the client bakes in that relationship by adding all the missing parents (and grandparents etc..). If this might mean three million new mappings, and you typically process PTR rows at 2,000 rows/s, this could take a very long time. Unfortunately, there is no good solution for this at the moment, it is all in one atomic transaction that will freeze your client's UI after a few seconds. Everything will unfreeze once it is finished. The system was designed for thousands of rows and was simply outgrown by the PTR. It is something I need to address the next time I visit parents and siblings. If you do this and your client is still doing CPU and HDD work in your OS's process reviewer program, the best answer for now is just to wait and let it finish. It doesn't happen nearly as bad with siblings. That said, as this is happening, I am pretty sure that the server has accepted your upload here so the janitors can already see it. If they approve it, you will get this parent relationship in a regular update in the coming days, so you might end up just processing it and doing all this big work in idle time and see the pending (1) count disappear naturally when the 'pending' record is vacated.

>>13823 Thank you for this report! Those are wx->Qt artifacts, a couple of typos in rarely used UI that slipped through the big update. You could be the first person to hit those panels since then! These are fixed for 390, please let me know if you discover any more.

>>13828 thinking about it more, when you single click on a link in web browsers it changes the page in the current tag. middle mouse click opens in a new tab. it would be nice to replicate this in hydrus, maybe even as a default. popup hovering would also be much nicer than the small preview box in the bottom left corner. if you hover a collection or video the scroll wheel can be used to scrobble.

>>13828 After your suggestion, I left hydrus running even after it said it was frozen, and after several hours (about 3-4 I'd say) it actually did finish, however when it was finished, the ui was still frozen so I had no way of knowing and just had to hope that it was done when I finally force-closed it and restarted Hydrus. That could be some other bug at work. You already mentioned how it's bad that it works this slowly and freezes the system, but I think another smaller issue here (and I don't know how much you can do about this) is that when it was working and applying the parent/child relationship, it didn't look like it was doing anything. It just looked like it froze, so I kinda just had to trust that it was still doing stuff. Even in the system monitor, it said that hydrus was only using 1% CPU. You'd figure that if it's working hard enough to be perpetually frozen for hours, it'd be using more CPU than that to speed things up and to justify the freezing, but it never went above 3%. The only reason I knew that it was still working, was because you told me that it probably was. Maybe there should be some sort of spinning icon or something else, to let you know that Hydrus is still doing work, even when the progress bar isn't moving, the system say it's barely processing anything at all, and the ui keeps freezing and won't respond. But I guess that falls more into suggestion territory than bug report. ¯\_(ツ)_/¯ Thanks for the help.

>>13840 >>13820 the PTR is a pain in many ways. personally, I got rid of it and moved to just scraped tags + deepdanbooru (v3) and am much happier. if you've got an amdgpu on linux you can use rocm + docker (rocm/pytorch python3, then install tensorflow-rocm inside it). for scraping I have a script that looks for missing urls and searches for their md5 hash on boorus. for deepdan I have my own python script i've made but there's a hydrus deepdan repo out there

>>13802 No worries, I have a late reply of my own… Anywho, that did it. Bizarre how hydrus was working the entire time until now, I was using w10 since release with hydrus. I suppose I made hydrus import some strange file that required some media component not in the N version. Thanks for the solution, was annoying going for a month or so without it.

>>11542 client api returns video/x-matroska for mkv files, but video/mp4 for mp4 files instead of mpeg4. dealing with mimetypes is confusing, it would be nice for the client api to report the full local path (and hashes other than md5) instead of having to manually write a lookup dict or glob searching via sha2

Whenever I load a saved search sorting by time imported it seems to disregard the timestamps and sorts them randomly instead

This gif breaks on hydrus' preview. It's the only time this has happened, so I have no idea why it's doing this.

>>13840 >>13820 Okay, so this same problem is happening, but now it happens when I'm processing PTR updates as well. I don't know what's causing it but it's making it difficult to use Hydrus at all. Sometimes, Hydrus completely locks up but sometimes, hydrus still acts like it's responding, but never actually loads or processes anything, like I'll type a search in, and Hydrus will just say "loading" forever. I checked system monitor, and it says that hydrus is still writing data, but very slowly (tens of kilobytes per second) and the UI doesn't even show that. Typically, after around 2 minutes of saying that it's still processing rows without actually processing them, the processing drops to 0/s and stays like that. Like before, when I try to quit hydrus, the window closes, but hydrus stays running in the background and the exit dialog never pops up, so I have to kill the process manually (it still doesn't respond to end requests) which is annoying and makes me feel like I'm going to corrupt my database, which actually did end up happening one time, but I just restored from a recent backup. This problem isn't because of me using an HDD, because it just started happening recently. It worked fine before. Part of me feels like the problem might be a corruption in my database from before the backup, but hopefully that's not the case, cuz if it is I'm probably screwed. ¯\_(ツ)_/¯ I started hydrus from the terminal to check for any error logs, but hydrus isn't showing any errors, except:
Gtk-Message: Failed to load module "canberra-gtk-module"
Gtk-Message: Failed to load module "pk-gtk-module"
Gtk-Message: Failed to load module "canberra-gtk-module"
Gtk-Message: Failed to load module "pk-gtk-module"
but even though I'm no programmer, I don't think that's causing the problem. Also sorry for the rambly bug report. I've tried everything I can think of to fix this without reporting it to you again, because almost every time I report a bug, it ends up not actually being a bug and just me not understanding something about hydrus, but I'm out of ideas on how to fix this, and because of the bug's persistence through a backup restore, I think it's actually the software this time.

>>11542 mpv bugs with linux: autohide cursor is buggy, I go to prevent it from hiding and half the time it still hides my cursor. svp (smooth video player, svp-team.com) interopt is wonky. svp works using a vapoursynth build of mpv, piped through a unix socket. it will work on half the files in the media viewer, and those files that don't work in the media viewer sometimes will work in the preview box. it's not an SVP problem, I *think* it's mpv sharing a pipe with the viewer in the preview box due to sharing the same mpv config. is there a way to tell if mpv is being used vs the native viewer? maybe mpv isn't being used at all half the time. also being able to set mpv profiles for certain tags (i.e anime vs real life vs movie etc etc) would be great. finally, there's no option to use mpv for everything (images, animations, videos), and if there were, would it be possible to bind the shortcuts like so: left/right goes prev/next on images, on videos it will scrobble and you need to shift+left/right to go prev/next on videos

>>13871 ok, so with SVP it seems to think the video player was completely dropped but half the time doesn't detect when it starts playing a new file. the below message doesn't happen when pausing/resuming. is hydrus destroying mpv and restarting mpv when changing files instead of forcing it to load a new file in the current instance? 13:17:38.723 [I]: Playback [1fc870c6]: playing at 120 [24 *5/1] 13:17:44.196 [I]: Playback [1fc870c6]: disabled while video is paused 13:17:44.202 [I]: Playback [1fc870c6]: deleted 13:17:45.601 [I]: VideoPlayer: mpv [python3.8] connected, waiting for the video info… as for the autohide, I fixed that with cursor-autohide-fs-only = "yes" in the config, and for telling the difference between native and mpv I just used osd-level = 3

>>13872 >>13871 update on these. I could be wrong, but it looks like hydrus-mpv is actually caching the upscaled video. turning on demonstration mode (shows like a line in the center for a smoothness test) and then changing videos, turning off svp, and finally going back to the video it still shows the vertical comparison line and you can see one half is upscaled. this is actually great, although really really confusing at first.

In the migrate tags window, clicking on "no path set" to choose a destination for HTA files is instantly crashing the client on 391 running on Windows 10. I just updated from 390 and it was crashing there as well.

Pic related gives me an "image truncated" error. image file is truncated… (Copy note to see full error)
Traceback (most recent call last):
File "site-packages\PIL\ImageFile.py", line 234, in load
File "site-packages\PIL\PngImagePlugin.py", line 658, in load_read
File "site-packages\PIL\PngImagePlugin.py", line 122, in read
File "site-packages\PIL\_binary.py", line 81, in i32be
struct.error: unpack_from requires a buffer of at least 4 bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "include\ClientImportFileSeeds.py", line 973, in ImportPath
def IsAPostURL( self ):
File "include\ClientImportFileSeeds.py", line 930, in Import
File "include\ClientImportFileSeeds.py", line 217, in DoWork
else:
File "include\ClientImportFileSeeds.py", line 302, in GenerateInfo
percentage_in = HG.client_controller.new_options.GetInteger( 'video_thumbnail_percentage_in' )
File "include\HydrusFileHandling.py", line 177, in GetFileInfo
( ( width, height ), duration, num_frames ) = HydrusImageHandling.GetImageProperties( path, mime )
File "include\HydrusImageHandling.py", line 505, in GetImageProperties
( width, height ) = GetResolutionNumPy( numpy_image )
File "include\HydrusImageHandling.py", line 196, in GenerateNumPyImage
numpy_image = GenerateNumPyImageFromPILImage( pil_image )
File "include\HydrusImageHandling.py", line 241, in GenerateNumPyImageFromPILImage
s = pil_image.tobytes()
File "site-packages\PIL\Image.py", line 762, in tobytes
File "site-packages\PIL\ImageFile.py", line 240, in load
OSError: image file is truncated

>>13870 To give a bit more info, I've since tried running the windows version of hydrus in Wine (a made a backup, then restored it on the windows version) to see if this is a problem with the linux version of Hydrus specifically. The same exact issues happened on the Windows version. This means that the issue that's keeping me from using hydrus is either a bug that's not specific to an OS version, or a bug in my database specifically. I've tried running a database integrity check to see if it's the latter, but since that's one of the things that causes Hydrus to freeze in the first place, I'm not having much luck there. I've even tried letting it run the integrity check while the UI was frozen for 2 days straight, to see if it's just me not waiting long enough for it to finish, and I still got nothing. It just stayed frozen until I killed the process manually. I've been starting hydrus from the terminal to see if I can catch any errors that might be causing the problem, and finally I think I might've found something that can help. This error-like message popped up in the terminal while it was syncing to the PTR. Hopefully, it can help you figure out the problem I'm having:
2020/04/09 14:04:33: public tag repository processed 33,004 content rows at 9 rows/s
2020/04/09 14:05:53: QXcbConnection: XCB error: 13 (BadGC), sequence: 14934, resource id: 0, major code: 130 (Unknown), minor code: 3
2020/04/09 14:07:09: public tag repository sync: processing updates
2020/04/09 14:15:30: public tag repository processed 33,210 content rows at 66 rows/s


>>13770 I am also missing system:everything

>>13914 check this option

>>13915 oh, haha thanks so much I am very new at this

>>13915 thanks

The cycle of me being unable to check and respond to my messages every day continues, worsened by current events. I apologise. I'll keep trying new ideas here on schedule and stress budget, the ideal is 24-hour turnaround. >>13839 I think I agree with this. I'll see if I can get middle-click to open in the background, with have an option to make it raise. I'm open to the idea of previews opening on hover, but I am not ready to do too clever new UI tech until I have cleaned more old wx code away.

>>13840 >>13870 >>13913 Thanks. Yeah, I agree on the best solution is to have UI indication of work. My current first aim here is to relieve the hangups so that I have UI time to display that sort of thing. I made an experimental anti-hang mode under debug->data actions->db ui-hang relief mode. Since you are being hit by this, I recommend it. It may result in other unusual behaviour. I don't think you have a db error, as usually these errors happen nice and quick, but I think you might have Qt issues where your system is having trouble with my Ubuntu's Qt .so files. The best answer here is to run hydrus from source. This takes a bit of work to set up, but most Linux users who have had persistent 'my hydrus can't deal with anything too weird going on' or 'my console gets X warning/error spam' problems tend to find their situations improve a lot. Here is the page: https://hydrusnetwork.github.io/hydrus/help/running_from_source.html If you want to check for a db error more closely, check the 'help my db is broke.txt' document in install_dir/db, as it has useful background reading on the PRAGMA you want to run etc…

>>13850 Shall I add a simple 'ext' to the json structure, like '.mp4'? That'll be easier to match than my arbitrary mime choices.

>>13853 Thank you, this should be fixed in the latest version. It was some weird conflict, I think, with the current secondary sort, which is by default, and thus for most users, time imported.

>>13866 Thanks. I get this too. I am not sure, but I think this may be an mpv bug. I am parsing its size all correct, and my native viewer (and hence ffmpeg) is ok, but mpv seems to be rendering it with a half width ratio or something. I will hang on to this file as a good 'broken file' example and see if a new version of mpv or a different mpv conf setting handles it differently.

>>13871 >>13872 >>13874 Thanks. I wasn't sure why the cursor was hiding in Linux–it doesn't do the same in Windows, and I couldn't figure out if it was a Qt thing I was doing. As background, my current mpv system is that I cache a pool of available mpv windows off screen, and then any time I need a one for a new media, I ask the pool for one (which either generates a fresh one or returns an unused one, and then I silence, clear, and hide the current one, sending it back to the pool. The good thing about this is it lets me do mpv transitions real quick, without audio blips or a couple frames of the old vid at the wrong aspect ratio. The bad thing is a couple of users have had issues where one mpv window doesn't initialise correct, and so any time that window gets pulled from the pool, it is just a black screen with ok audio. I think a similar thing is happening to you. I believe my pool system is also leading to instability for some Linux users. I want to make a different 'safe mode' that re-uses the same window and tries to destroy it once it is done (although I have had extreme problems with crashes whenever I try to destroy mpv windows), rather than my window re-parenting mickey mouse shit. I think my plan here is to keep pushing on cleaning code for now. The way the main media viewer lays itself out is still wx stuff, and is a cause for the flickering on certain media transitions. I need to put aside time to just clean that out and make it nice and Qt, and then get mpv working stably, and once that is less on fire, I'll be more free to consider extensions like svp. I keep meaning to add shortcuts to scan a video back and forth, and then not getting to it.

>>13899 Damn, thank you for this report. I do not get this problem, but I have an idea on what it may be: some OSes are having trouble with Qt's file selection dialogs, and I need to add a flag option to instead launch native OS ones. Do you get a crash if you hid 'add files' or 'add folder' on file->import files?

>>13903 Thanks, I get the same. This error usually means the image is broken, usually the result of a file upload/download that was silently broken, perhaps when the user originally uploaded that image to a booru or whatever. Some of these errors can be compensated for, and some image libraries can deal with them better or worse. If you turn on help->debug->data actions->allow truncated image loading, this file does import to hydrus, but you will have trouble with it in future. I used to have that debug mode on all the time, but then it load to some crashing for certain files. In general, these problems are rare, and I recommend to open the image in Gimp/Photoshop, which can deal with errors much better than hydrus's libraries can, and then save it back again. That new file should be fine to import to hydrus, although it won't have tags. This has come up before and I don't really like the answer. I'll make a job to explore if my image libraries can do the load/save, in a safe way, and have an option to try to repair broken files.

>>13916 As general background, I do this because you'll likely find you don't use system:everything much as you get more into the program and have a larger collection. If that reason isn't a tooltip on that option, I'll add it.

>>13935 I just noticed it crashing yesterday when i was trying to import some single files. After a restart, i noticed both the buttons made the client crash. I don't know exactly what i did to fix it, but after a couple crashes and client restarts seems like it just fixed itself. But i could not solve the migrate tags crash.

>>13948 Right, I'll see if I can play with that 'native' file dialog flag today and get maybe a debug mode for you to try in 393. Otherwise, likely next week.

>>13934 as an update on this, I managed to get a workaround for SVP support w/o needing sockets. I just copied the vapoursynth script it generates and used -vf=vapoursynth=svp.py

I'm on the Linux version. problem that immediately occurred upon hydrus idling once I updated to v393. This showed up in the log:
2020/04/16 06:04:24: Cannot vacuum "/home/[username]/Hydrus_Network/hydrus network/db/client.mappings.db": I believe you need about 25GB on your temporary path's partition, which I think is /tmp, but you only seem to have 3.9GB.
2020/04/16 06:04:24: Cannot vacuum "/home/[username]/Hydrus_Network/hydrus network/db/client.master.db": I believe you need about 7.1GB on your temporary path's partition, which I think is /tmp, but you only seem to have 3.9GB.
At least, I think this is a problem. I'm not actually noticing any bad effects from it yet, except that the log message constantly popping up in the terminal is annoying, but I don't think I would notice the db not being vacuumed for a while anyway, so it might really not be vacuuming now. Something likely changed between v392 and v393 that's causing this to happen. Nothing on my system changed between updates (since I updated then restarted hydrus right after) and it didn't give any error like this before the update, but immediately got it once I did.

>>13964 If you run hydrus from source, You can make tmp folder in home folder like, mkdir ~/tmp and run hydrus python hydrus-client –temp_dir /home/your_name/tmp. This solves the problem.

>>13955 Ah, great. If you end up with a nice .conf you would recommend for others in your situation, please feel free to pastebin it to me with a readme.txt or whatever, and I'll integrate it into the defaults. >>13964 I think these started appearing because I recently made these errors more verbose. Basically the normal maintenance routine is being safely abandoned, and likely has been for a long time, and now it tells you why. As >>13965 says, you can change your temp dir if you like. Many Linux situations with ramdisks and so on run into this. Just launch the client, even a built release, like so: client --temp_dir="/path/to/dir" And you should be good. You can check you current temp dir under help->about. All temp stuff will go in there, including file imports and some other little stuff.

>>13978 Now I think of it, I'll make those statements only print once a boot or something, if you keep seeing them over and over every time a maintenance check runs.

Small bug in the dupe filter: When you zoom out an image it doesn't center but goes down your screen. I was very zoomed in to compare a png and jpg image (the png was the jpg converted!) and when i zoomed out again fully with numpad minus the image ended up below my bottom screen border. When you zoom in it centers as expected.

Another small bug in the dupe filter: When you click the "go back a pair", "previous", "next" and "show a different pair" on the right side always on top window (doesn't apply to the top mouse over window) your keyboard controls stop working (for example numpad plus to zoom). I assume it's a focus issue.

>>13978 >Ah, great. If you end up with a nice .conf you would recommend for others in your situation the vapoursynth file it generates is specific to each user's hardware, so it wouldn't be too helpful. if I come across any fixes or helpful tweaks I'll post them though. as an aside, you've been working on tag siblings/parents recently. have you thought of using an adblock rule parser? a bit weird I know, but it mostly fits into the problem space. brave browser built an extremely fast rule parser, someone ported it to python for qutebrowser. might have to rip some of it apart, but I think it supports most of what is needed (filtering, globbing/regex, substitution). https://github.com/ArniDagur/python-adblock https://github.com/brave/adblock-rust/

>>13978 >>14006 also, I ask about tag rules because I've been write locked for hours trying to import gelbooru's categories and danbooru's siblings/parents. dump of the import files: https://files.catbox.moe/8cxpp2.7z

>>14007 >7 hours and 12gb ram later It'll finish eventually… this might take a few days

>>14008 >27 hours and 30gb ram later I may end up killing hydrus I have a new gpu arriving later today that I want to install. tl;dr don't try to import 400k rules in one go

>>13995 Thanks. At the moment, it zooms based on the 'center point' of the image, which isn't helpful when you are zoomed in and dragged, so the center point is off-screen. Perhaps it zoom center around the centerpoint of the media viewer instead. There's a job somewhere to add an option to zoom off the current cursor position as well. I'll see about adding options here. For now, I think hitting 'zoom fit', which is 'z' shortcut by default, it'll always recenter. >>13996 Thank you again. This is an old wx->Qt artifact I think, and I need to ensure the hover windows are passing keyboard events up to their parent canvas. There may be a Qt issue for some shortcuts like tab and arrow keys where they get eaten by panel navigation (moving focus to next button etc…) that I can't get around, but I'll see what I can do.

>>14006 Thanks, this sounds like an interesting solution. This may sound weird, but the actual bottleneck right now for siblings load is not the construction of the logic object, which is about 16-50ms I think, but actually pulling the tag sibling strings from sqlite, ha ha. Pulling 150k strings from random locations from a cold db just takes a few seconds even on an ssd. The ideal of the new database level tag sibling tech will be that I can work with tag_ids, the master number definitions, for most operations, and only fetch strings in pieces, as I need them. In reality, it will probably be that I load the tag strings in the background, after boot, as they will be less urgent. Once I have all that working faster, and have a cache so the tags you see in the UI are pre-sibling-computed, I expect the actual logic of working with siblings will be the bottleneck again.

>>14007 >>14008 >>14010 Ah hell, just read this. Yes, this may be a bit much. The PTR is groaning at 150k siblings, so you are likely to have significant load and sibling edit lag. The delay there was probably due to tags being added from the parents. The PTR doesn't have all that many parents, 13k or so, but if those boorus have lots, that may have nuked you for this import operation. You may want to check your tag count under services->review services. Let me know how you get on here. You may discover some new tag parent efficiency problems.

>>14108 >This may sound weird, but the actual bottleneck right now for siblings load is not the construction of the logic object, which is about 16-50ms I think, but actually pulling the tag sibling strings from sqlite, ha ha. have you looked into duckdb? OLAP databases (vs OLTP) are much quicker at querying deep data, and duckdb is a drop in replacement for sqlite. >>14109 importing the parent tags, which was about 50,000 IIRC (I can't check, I'm write locked), only took about an hour, I think. that import I was stuck on was 400k siblings. maybe there's an interaction between parents and siblings that you're talking about but it seems like the bottleneck is just the massive amount of sql thrusting. I've chunked it into sets of 50k and have been importing without memory exploding so far.

The pixiv artist downloader seems to skip NSFW pics even with a login. When I log into the browser it automatically shows them, so it's probably the script ignoring the login or something.

>>14159 The login script has been broken for a while now since they added captcha. You have to import your pixiv cookies into the client manually.

>>14168 It is working though. The captcha only shows up if it's your first login in a while, so as long as it doesn't the script works. It even claims to be a valid login, but when I try to download stuff it just skips the NSFW images. If I try and download something like "38371462", which is all NSFW, it just claims there aren't any pics.

>>14172 >It is working though It's not, the login script is only looking for specific cookies for a "successful" login, PHPSESSID in this case. Pixiv will return this just for visiting the site, so Hydrus will say it's successful no matter what. Pixiv uses ReCaptcha v3 which doesn't always show a visible captcha but it's there.

>>14109 >Let me know how you get on here. You may discover some new tag parent efficiency problems. update on this with 350k siblings and 50k parents, I'm lagging out when importing new files now, even 100kb ones. it takes about a second for each file and it locks my UI up. are you maintaining a cache of the relationships or is it trying to compute the resulting tag output from the database every time it tries to import a file? UI stutter would indicate python is doing a big computation that locks the UI

since this week's update, i've been having this issue where the related tags panel in the manage tags window does not display the most recently-used tag at the top. it seems to add tags in completely random spaces and it's really annoying because i have to search the list for the tags i want each time. does anyone else have this issue?

I notice a weird bug, my gallery downloader always randomly stopped when Hydrus Client window is minimized. Last I remember the gallery was downloading at 38/192 number of images. I minimized it and went to sleep. When I woke up it still stay at 38/192 and then it ran again normally when the window is restored.

The e-hentai.org post page url downloader seems to be broken - it keeps throwing the message "Looks like HTML – maybe the client needs to be taught how to parse this?" even when the exact url is in the parser.

So I tried to mess around with the tag importing stuff for the first time, and added some tags to the file import blacklist and ticked off the "only import tags that already exist" box and it worked just fine for the gallery downloader But then I found the option to use the downloader settings for simple url import, and when I tried to choose the default it gave me this 'too many values to unpack error' as seen here ValueError too many values to unpack (expected 8) File "hydrus\client\gui\ClientGUICommon.py", line 218, in EventButton self._func( *self._args, **self._kwargs ) File "hydrus\client\gui\ClientGUIScrolledPanelsEdit.py", line 5745, in _LoadDefaultOptions self._SetValue( default_tag_import_options ) File "hydrus\client\gui\ClientGUIScrolledPanelsEdit.py", line 5761, in _SetValue panel.SetValue( service_tag_import_options ) File "hydrus\client\gui\ClientGUIScrolledPanelsEdit.py", line 6153, in SetValue ( get_tags, get_tags_filter, self._additional_tags, self._to_new_files, self._to_already_in_inbox, self._to_already_in_archive, self._only_add_existing_tags, self._only_add_existing_tags_filter ) = service_tag_import_options.ToTuple()

>>14120 Thanks. The topic of other db engines has come up before. In aggregate, I will be sticking with sqlite. A user who has participated in these discussions before ended up writing up a document about it, if you are interested, here: https://gitgud.io/prkc/hydrus-why-sqlite/blob/master/README.md I overall agree with him. It may be true that duckdb or another db can solve the tag sibling problem faster than sqlite, but I am sure there would be other tradeoffs, some predictable, some not. The real and easier solution here is for me to change my siblings code so it is not reliant on 150k random drive accesses before it can serve results. Hmm, if it was siblings that was slow to import, it may have been something else. If it was hanging on the UI panel doing the import rather than the 'apply' button actually saving it to the db, that suggests my UI add-code isn't scaling well. Maybe it is doing a 500ms full-scan on every lookup, or there is an order-n-squared in there somewhere. Saving that many siblings to the db should be a few million rows, I'd say at most a hundred seconds on an SSD. I'll look at this.

>>14177 It could be due to the siblings. I'd say that is probably even likely, but I am not sure. Once the siblings caches are loaded/built, they generally run fast, and when the PTR's 150k is ok, I am not sure why double that would be such a problem. If you would like to explore more, please try importing some files with help->debug->profile modes->db/pubsub profile mode on. There is more in this help document: https://hydrusnetwork.github.io/hydrus/help/reducing_lag.html#profiles I am working on improving tag siblings at the UI and db level now, so this would be useful information.

>>14196 Hey, I am sorry for this trouble. This should be fixed in 396. Next week's 397 will sort the 'favourites' column.

>>14197 Thank you for this report. This is not intended. The only way I can think of this normally happening is if the domain was full in bandwidth terms and you were using the 'override bandwidth rules' checkbox on the network control's cog button. This override mode is a hack and currently tied into the UI Page Timer, which only works when the current page is in view. I plan to update these bandwidth override controls in future to work directly in the network engine. If you are not using the bandwidth override mode, does this behaviour happen when you simply switch to another page? Does the gallery pause when it is 'not shown' in this way, or is it only when the client is minimised?

>>14204 Hey, I am afraid that the e-hentai parser that once bundled with the client is not functional. It is no longer included, but older users may still have it in their clients. I do not know how or if this works, but here is the latest e-hentai downloader in the user-run downloader repository: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/blob/master/Download%20System/All-in-Ones/Single-Sites/e-hentai.org%20and%20exhentai.org%20-%20version%202019-08-15.png You can import that under network->downloaders->import downloaders. I don't know what you'll have to do to login, if that is needed, but maybe Hydrus Companion can help you: https://gitgud.io/prkc/hydrus-companion

>>14212 Thank you for this report. I am sorry for the trouble! I will have this fixed for 397.

Looks like Gelbooru downloads are broken. Both gallery downloads and subscriptions get nothing.

>>14304 Okay it looks like it's all downloads, not just Gelbooru. Might just be me; I'll check if they work tomorrow. Tried downgrading to 396, when they definitely were working and the issue remained.

>>14305 >Okay it looks like it's all downloads Nevermind that, it's just Gelbooru.

>>14305 >>14304 >>14309 Gbooru subscriptions are broken for me too

Gelbooru changed the name of the class Hydrus uses to find post links in galleries from "thumb" to "thumbnail-preview". If you know where to edit this yourself it's a trivial fix.

>>14317 This indeed fixed it, thanks!

>>14250 I read over that link and I don't see any mention of OLTP. One bonus for OLTPs is you don't need to maintain read caches every time you want to add a feature, they work by being really really fast at brute forcing queries. I mention these things just out of a technical curiosity, this stuff is cool. https://clemenswinter.com/2018/07/09/how-to-analyze-billions-of-records-per-second-on-a-single-desktop-pc/ It's a shame they are relatively unpopular in the consumer space and you either get a partial implementation or very poor documentation / driver support. >>14251 it seems to be OK for now, maybe it was my file storage being fucky. I have 100TB in raid, sometimes it slows down and needs a restart, probably to flush stale caches. I have hydrus on an nvme. As an aside, can you add md5/sha2/etc as an actual tag namespace, not just a system primitive, so it can be searched via the client api? I want to make a hydrus downloader which uses gallery-dl and youtube-dl, but I'm finding myself having to maintain an md5 lookup table which requires a rehashing of all my files.

Thanks for gelbooru reports lads, should be a fixed gelb parser in the latest release also. >>14321 Thanks, that is interesting. The same user who wrote the advanced 'OR' entry parser recently sent me a parser to convert text like 'system:height>400' to actual system predicate info. When I have some time, I hope to plug this into the new tag autocomplete pipeline and allow text entry of system preds. It supports system:hash. Fingers crossed, I will be able to do the same for the client api and instantly add all basic system preds to it. Let me know if it gives you any trouble when I get to it.

Derpi fag here. It appears Derpibooru retired their old api and are implementing a new method. I'll be digging in again to see if I can't fix it myself. I saw last time someone else needed to fix a duplicate tag problem when I gave the parsers. Anyone mind helping me understand what was wrong?

>>14374 Derpi fag is back. The API rework was not too different in the switch. Mostly just had to change the parser to look for a static path that wasn't there before. Created inside and tested inside virgin v399 copies on windows.

I think Twitter changed something and broke the Hydrus parser. I get the following error when I try to use the Twitter gallery downloader
Page Parser twitter media tweets api parser: Unable to parse that JSON: Expecting value: line 1 column 1 (char 0). JSON sample: <!DOCTYPE html>… (Copy note to see full error)
Traceback (most recent call last):
File "hydrus/hydrus/client/ClientParsing.py", line 1637, in _ParseRawTexts
File "hydrus/hydrus/client/ClientCaches.py", line 388, in GetJSON
File "json/__init__.py", line 354, in loads
File "json/decoder.py", line 339, in decode
File "json/decoder.py", line 357, in raw_decode
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "hydrus/hydrus/client/ClientParsing.py", line 2347, in Parse
File "hydrus/hydrus/client/ClientParsing.py", line 540, in Parse
File "hydrus/hydrus/client/ClientParsing.py", line 1643, in _ParseRawTexts
hydrus.core.HydrusExceptions.ParseException: Unable to parse that JSON: Expecting value: line 1 column 1 (char 0). JSON sample: <!DOCTYPE html>
<html dir="ltr" lang="en">
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=0,viewport-fit=cover" />
<link rel="preconnect" href="//abs.twimg.com" />
<link rel="preconnect" href="//api.twitter.com" />
<link rel="preconnect" href="//pbs.twimg.com" />
<link rel="preconnect" href="//t.co" />
<link rel="preconnect" href="//video.twimg.com" />
<link rel="dns-prefetch" href="//abs.twimg.com" />
<link rel="dns-prefetch" href="//api.twitter.com" />
<link rel="dns-prefetch" href="//pbs.twimg.com" />
<link rel="dns-prefetch" href="//t.co" />
<link rel="dns-prefetch" href="//video.twimg.com" />
<link rel="preload" as="script" crossorigin="anonymous" href="https://abs.twimg.com/responsive-web/web/polyfills.31ad74e4.js" nonce="ODY1NzI0YTctNWRlNi00ZGRkLTg3N2QtNThkZjdlMDBkN2U3" />
<link rel="preload" as="script" crossorigin="anonymous" href="https://abs.twimg.com/responsive-web/web/vendors~main.19622e04.js" nonce="ODY1NzI0YTctNWRlNi00ZGR

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "hydrus/hydrus/client/importing/ClientImportGallerySeeds.py", line 309, in WorkOnURL
File "hydrus/hydrus/client/ClientParsing.py", line 2375, in Parse
hydrus.core.HydrusExceptions.ParseException: Page Parser twitter media tweets api parser: Unable to parse that JSON: Expecting value: line 1 column 1 (char 0). JSON sample: <!DOCTYPE html>
<html dir="ltr" lang="en">
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=0,viewport-fit=cover" />
<link rel="preconnect" href="//abs.twimg.com" />
<link rel="preconnect" href="//api.twitter.com" />
<link rel="preconnect" href="//pbs.twimg.com" />
<link rel="preconnect" href="//t.co" />
<link rel="preconnect" href="//video.twimg.com" />
<link rel="dns-prefetch" href="//abs.twimg.com" />
<link rel="dns-prefetch" href="//api.twitter.com" />
<link rel="dns-prefetch" href="//pbs.twimg.com" />
<link rel="dns-prefetch" href="//t.co" />
<link rel="dns-prefetch" href="//video.twimg.com" />
<link rel="preload" as="script" crossorigin="anonymous" href="https://abs.twimg.com/responsive-web/web/polyfills.31ad74e4.js" nonce="ODY1NzI0YTctNWRlNi00ZGRkLTg3N2QtNThkZjdlMDBkN2U3" />
<link rel="preload" as="script" crossorigin="anonymous" href="https://abs.twimg.com/responsive-web/web/vendors~main.19622e04.js" nonce="ODY1NzI0YTctNWRlNi00ZGR
Running v399 on Linux. Happens on a fresh Hydrus installation as well as my normal one, but I can't comment on Windows. Someone in the Q&A thread says they're having issues with Twitter as well.

>>14381 That was me in the Q&A thread, and I am on Windows. I don't know how to get a detailed log like you did but if it doesn't work, it doesn't work.

that's because twitter recently removed the older twitter ui that twitter parser used to grab tweets and search usernames. it's been replaced with the newer ui that's generated with JS (as well as using randomized class names). until we find a publically-available twitter api that we can use for parsing, a good stopgap measure would be to use the all-in-one parser for nitter, a nice js-less web viewer for twitter tweets. you can easily start searching and parsing twitter tweets again with nitter, provided you have the twitter url class set up to do an api redirect to nitter.net instead of twitter.com. a copy of the all-in-one nitter downloader can be found in the attached image, or via this link: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/blob/master/Download%20System/All-in-Ones/Single-Sites/niter-2020.06.03.png i haven't tested it yet (i already have my own nitter downloader i made myself), but it seems to work just fine. if you want to download twitter urls manually with it (via the url downloader page) like i said you're going to have to edit the twitter url class to have the api url redirect to nitter.net instead via a regex substitute (put 'twitter.com' in the first box, and 'nitter.net' in the second). hope this helps!

>>14385 >>14381 you may want to take a look at >>14387

>>14387 Looks good. I'll give it a shot when I get home.

Getting a weird error trying to boot v390 or higher on Linux Mint (18.3). v389 and earlier work just fine. Same error on a fresh install so I'm assuming it's an issue with my machine, but I can't make heads or tails of it.
Traceback (most recent call last):
File "/home/chives/hydrus network/client.pyw", line 185, in <module>
from hydrus import ClientController
File "/home/chives/hydrus network/hydrus/ClientController.py", line 16, in <module>
from . import ClientGUI
File "/home/chives/hydrus network/hydrus/ClientGUI.py", line 11, in <module>
from . import ClientGUIDialogsQuick
File "/home/chives/hydrus network/hydrus/ClientGUIDialogsQuick.py", line 2, in <module>
from . import ClientGUIScrolledPanelsEdit
File "/home/chives/hydrus network/hydrus/ClientGUIScrolledPanelsEdit.py", line 17, in <module>
from . import ClientGUITags
File "/home/chives/hydrus network/hydrus/ClientGUITags.py", line 17, in <module>
from . import ClientManagers
File "/home/chives/hydrus network/hydrus/ClientManagers.py", line 481
self._keys_to_services: typing.Dict[ bytes, ClientServices.Service ] = {}
^
SyntaxError: invalid syntax


>>14387 This seems to break with any Twitter user that has an underscore in their name. I tried adding regex to this (instead of any alphanumeric character) but that breaks everything and I'm probably doing it wrong. Any idea why this wouldn't work with underscores, or what could be done to fix it so it works?

>>14406 I'm not sure if this counts as a solution or if this is what you're already doing, but I haven't had any problems with just directly using the Nitter parser without any use of regex I don't even know where to edit regex in Hydrus. I just finished running a Nitter gallery downloader for a user that has an underscore in their name and it completed without incident.

If you are having troubles with the twitter parser or are using the parser from this post >>14387 update to the latest nitter parser on the presets github.

>>14408 That worked! Thank you! For anyone else stumbling onto this, the link above doesn't work anymore, but this leads to a huge list of parsers (including the updated nitter): https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Download%20System/All-in-Ones/Single-Sites For anyone vaguely curious, it does look like it switched from "alphanumeric" to "any number of characters", thus matching the underscores now.

>>14396 Damn, I am sorry for the trouble here. Since the Qt switch, I have started decorating hydrus's code with 'typing' hints, which were introduced in I think Python 3.5 or so. I release on 3.7 now. As you are running from source, if you open up a terminal and type 'python3', it should give you some summary info when you boot it, including version. Type 'exit()' to quit out. I don't know anything about Mint 18.3, but I would have thought it would have a newer python. Maybe those variable decorators need 3.6 or something? As it is, I don't really like those variable hints, so if that is what your python has a problem with, I've taken out the offending line in your traceback, and all the other obvious ones I could find. I likely have more, so let me know how tomorrow's 401 works for you.

>>14452 me, forgot my BO trip General reply about recent nitter/twitter/derpi/e621/other problems: there were some problems in the code I use to update the default downloaders for the client. Sometimes, when a new default came in, if the user already had a different custom downloader imported to handle the relevant domain, the update was not applying correctly. Some sites got set back to the defaults again, and some complicated situations (e.g. 100+ x.booru.org definitions) got tangled up url class links. I have tightened this code up for tomorrow. Please let me know how this continues to go in future.

I have to right click the search field, wait for the menu, and then click the search field twice to be able to use it in Linux Mint. As soon as it loses focus (switching windows, selecting a file, etc.), I have to do it again. The whole window will be out of focus until I select something, which is probably a related problem. This has been the case for a long time. It's still usable, but this is a really annoying problem. I've also tried to run the Windows version in Wine with no luck. It crashes before the window even opens.


2020/06/17 20:59:11: hydrus client started
2020/06/17 20:59:11: Could not import lz4--nbd.
2020/06/17 20:59:12: booting controller…
Fontconfig error: Cannot load default config file
2020/06/17 20:59:12: There was an error trying to start the splash screen!
2020/06/17 20:59:12: Traceback (most recent call last):
File "/opt/hydrus/hydrus/client/ClientController.py", line 501, in CreateSplash
self._splash = ClientGUI.FrameSplash( self, title )
File "/opt/hydrus/hydrus/client/gui/ClientGUI.py", line 6461, in __init__
screen = ClientGUIFunctions.GetMouseScreen()
File "/opt/hydrus/hydrus/client/gui/ClientGUIFunctions.py", line 400, in GetMouseScreen
return QW.QApplication.screenAt( QG.QCursor.pos() )
AttributeError: type object 'QApplication' has no attribute 'screenAt'

2020/06/17 20:59:12: hydrus client failed
2020/06/17 20:59:12: Traceback (most recent call last):
File "/opt/hydrus/client.pyw", line 180, in <module>
controller.Run()
File "/opt/hydrus/hydrus/client/ClientController.py", line 1377, in Run
self.CreateSplash( 'hydrus client booting' )
File "/opt/hydrus/hydrus/client/ClientController.py", line 501, in CreateSplash
self._splash = ClientGUI.FrameSplash( self, title )
File "/opt/hydrus/hydrus/client/gui/ClientGUI.py", line 6461, in __init__
screen = ClientGUIFunctions.GetMouseScreen()
File "/opt/hydrus/hydrus/client/gui/ClientGUIFunctions.py", line 400, in GetMouseScreen
return QW.QApplication.screenAt( QG.QCursor.pos() )
AttributeError: type object 'QApplication' has no attribute 'screenAt'

2020/06/17 20:59:12: hydrus client shut down
This started after installing v400 and has continued with v401. I'm using the AUR package and there was a previous error message about cv2 which I fixed by installing python-opencv.

After updating to v400 my subscriptions all broke in a way. They seem to still import files and tags properly, but they only publish the files to a page sometimes(maybe only the first page, then it ignores the others, but I'm not sure). I ran all my subscriptions and got a few pages with less than 20 files in total, but using my tagging system to find new files showed almost 200 new files. Sorting by time confirmed that at least some files that didn't show up in pages were imported by subscriptions.

>>14470 I figured it out. It publishes files to a page after the subscription is done, but for some reason if hydrus is not in focus it just skips creating the page. I updated and the bug is still present in v401.

>>14461 This is the search where you type tags, right? Can you try unchecking the options->gui->autocomplete results float in main gui option? Some Linux window managers don't like my floating dropdown, and that'll embed it once you restart the client. If you hit help->debug->gui actions->make some popups, does that show correctly? And how about the left/top/top-right 'hover' windows over a normal media viewer?

>>14464 Ah, I am not totally sure as I do not maintain the AUR package, but I think the normal opencv has its own Qt in it that conflicts with the one hydrus wants to use. Can you un-pip that and instead try: pip3 install opencv-python-headless If you already did that, or it still doesn't work, which version of Qt do you have? And is it PySide2 or PtQt5? Should be doable with 'pip3 show PySide2' command.

>>14470 >>14473 Thank you for this report. I am sorry for the trouble. The new subscription pipeline must be failing here, and I will check it out.

>>14477 It's PyQt5, version 5.15.0.

Gelbooru subs seem to be broken again? They're not finding anything.

>>14486 Okay the update didn't fix it and no-one else seems to have the same issue, so it's probably just me. Could this >>14479 be part of the issue? I think the problems started quite close to importing that, but that's just a hunch.

The page warning opens on a per-page basis, so if I leave my subscriptions running when there's already a big number of pages open I get dozens of the same message. >>14496 Works on my machine. Maybe your downloader is fucked.

Vacuum fails. My Hydrus is getting laggy,so I tried vacuum then I got this error message. 2020/06/28 11:38:21: Vacuumed /home/myname/ssd/hydrus/db/client.db in 1 minute 28 seconds 2020/06/28 12:08:48: Vacuumed /home/myname/ssd/hydrus/db/client.caches.db in 30 minutes 26 seconds 2020/06/28 12:08:48: Cannot vacuum "/home/myname/ssd/hydrus/db/client.mappings.db": I believe you need about 28GB on your temporary path's partition, which I think is /tmp, but you only seem to have 10.0GB. 2020/06/28 12:11:19: vacuum failed: 2020/06/28 12:11:19: Exception: 2020/06/28 12:11:19: OperationalError database or disk is full Traceback (most recent call last): File "/home/erdos/ssd/hydrus/hydrus/client/ClientDB.py", line 15345, in _Vacuum HydrusDB.VacuumDB( db_path ) File "/home/erdos/ssd/hydrus/hydrus/core/HydrusDB.py", line 107, in VacuumDB c.execute( 'VACUUM;' ) sqlite3.OperationalError: database or disk is full 2020/06/28 12:11:20: An attempt to vacuum the database failed. I thought this caused by insufficient memory so I changed –tmp-dir option to more than 500GB partition. And I vacuumed once again then I got this message. 2020/06/28 12:20:59: Vacuumed /home/myname/ssd/hydrus/db/client.db in 1 minute 18 seconds 2020/06/28 12:51:35: Vacuumed /home/myname/ssd/hydrus/db/client.caches.db in 30 minutes 35 seconds 2020/06/28 12:58:31: vacuum failed: 2020/06/28 12:58:31: Exception: 2020/06/28 12:58:31: OperationalError database or disk is full Traceback (most recent call last): File "/home/myname/ssd/hydrus/hydrus/client/ClientDB.py", line 15345, in _Vacuum HydrusDB.VacuumDB( db_path ) File "/home/myname/ssd/hydrus/hydrus/core/HydrusDB.py", line 107, in VacuumDB c.execute( 'VACUUM;' ) sqlite3.OperationalError: database or disk is full 2020/06/28 12:58:32: An attempt to vacuum the database failed.

>>14507 Tanks anon, it works now. Now I just have to catch up over a week's worth of subs…

>>14486 >>14496 >>14510 Sorry for the trouble here. The downloader system is really lacking a versioning system to better track and upgrade for changes. This will be a big job, so perhaps in the meantime I can do some better UI to navigate re-linking when you import via Lain. I still have to think about this, but I hate when the auto-import fails.

>>14507 Thanks. This is an old wx limit. I don't know what Qt's limit will be, but I'll increase the max value for the option under options->gui pages up to 500. I can't promise the program won't crash at very high numbers. Nonetheless, if you are at this many subs, I strongly recommend you try to combine the 'publishing label' of your subs a bit, so instead of publishing a new page for every query, instead your subs publish all their queries to merged 'blahbooru' or 'artist' pages. If you really have a ton of subs and often have many pages open that you rarely visit, you might like to even go the next step and add 'my tags' filtering tags like 'blahbooru subscription import' to your subs through their 'tag import options' and then not publish to a button or page at all. You can then have a single search page and load up these 'my tags' filter tags whenever you like and keep a lightweight session.

>>14508 Thank you, that is odd. As you can see, there's a pre-check that tries to estimate available space, and it skips the attempt if it looks like there isn't enough. It'll only print that message once per boot, by the way. The actual OperationalError there is a sqlite error, so it looks like my check was ok, perhaps on a subsequent db file, but it still failed. Did you use --tmp-dir
or
--temp_dir ? You can check this sticks by checking help->about, where the current temp dir should be listed. Even then, if the temp dir was still /tmp, the 'canvacuum' check should have failed with the nicer error again. Can you tell me the size of the client.master.db file in install_dir/db? I wonder if my calculation is off for that one (but not client.mappings.db, which is the 28GB estimate). I'll say pause vacuums for now, although I think the fail will have paused them automatically for you. If you like, you can try a manual vacuum. Run the sqlite3 executable in the db dir, or any other sqlite interface, and type this: .open client.master.db
VACUUM;
.exit But it may just end up running out of space again unless you jigger-jagger about with environment variables or PRAGMA commands.

>>14508 Hi,I'm >>14528 Thanks for reply,dev. >If you like, you can try a manual vacuum. Run the sqlite3 executable in the db dir, or any other sqlite interface, and type this: It suceeded! But I Don't know why. I made this through Try and error: set TMPDIR and SQLITE_TMPDIR environmental variables and PRAGMA tmp_store etc… But I can't filter out what is the cause. >Did you use > > >–tmp-dir >or >–temp_dir >? You can check this sticks by checking >help->about, where the current temp dir >should be listed. Even then, if the temp >dir was still /tmp, the 'canvacuum' >check should have failed with the nicer >error again. –temp_dir Sorry for incorrect description. I checked it.pic is related. >Can you tell me the size of the >client.master.db file in install_dir/db? >I wonder if my calculation is off for >that one (but not client.mappings.db, >which is the 28GB estimate). 7.2GB. Now it shrank to 6.9GB. This error has been persisted for one or two months so I'm so happy. Thanks dev!

Using an unmodded 403 build, some tags won't download from derpibooru. For example if I try to download this image https://derpibooru.org/images/232093 the tags in brown arn't downloaded. Any Fixes?

>>14535 Thanks, I am glad it got sorted. It's so weird that sqlite wasn't inheriting the temp dir from the environment correct. I'll keep this in mind. >>14548 Thank you, that looks like a (new?) 'species' namespace. I'll make sure a fixed parser gets rolled into v405.


2020/07/20 17:29:15: hydrus client started
2020/07/20 17:29:15: booting controller…
2020/07/20 17:29:19: booting db…
2020/07/20 17:29:11: hydrus client started
2020/07/20 17:29:14: booting controller…
2020/07/20 17:29:19: booting db…
2020/07/20 17:29:19: preparing disk cache
2020/07/20 17:29:24: preparing db caches
2020/07/20 17:29:24: initialising managers
2020/07/20 17:29:26: booting gui…
2020/07/20 17:29:19: preparing disk cache
2020/07/20 17:29:29: The db encountered a serious error! This is going to be written to the log as well, but here it is for a screenshot:



Traceback (most recent call last):
File "hydrus\core\HydrusDB.py", line 555, in _InitDBCursor
self._BeginImmediate()
File "hydrus\core\HydrusDB.py", line 297, in _BeginImmediate
self._c.execute( 'BEGIN IMMEDIATE;' )
sqlite3.OperationalError: database is locked

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "hydrus\core\HydrusDB.py", line 838, in MainLoop
self._InitDiskCache()
File "hydrus\client\ClientDB.py", line 9059, in _InitDiskCache
self._LoadIntoDiskCache( stop_time = stop_time )
File "hydrus\client\ClientDB.py", line 9229, in _LoadIntoDiskCache
self._InitDBCursor()
File "hydrus\core\HydrusDB.py", line 559, in _InitDBCursor
raise HydrusExceptions.DBAccessException( str( e ) )
hydrus.core.HydrusExceptions.DBAccessException: database is locked

2020/07/20 17:29:29: hydrus db failed
2020/07/20 17:29:29: The db encountered a serious error! This is going to be written to the log as well, but here it is for a screenshot:



Traceback (most recent call last):
File "hydrus\core\HydrusDB.py", line 555, in _InitDBCursor
self._BeginImmediate()
File "hydrus\core\HydrusDB.py", line 297, in _BeginImmediate
self._c.execute( 'BEGIN IMMEDIATE;' )
sqlite3.OperationalError: database is locked

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "hydrus\core\HydrusDB.py", line 838, in MainLoop
self._InitDiskCache()
File "hydrus\client\ClientDB.py", line 9059, in _InitDiskCache
self._LoadIntoDiskCache( stop_time = stop_time )
File "hydrus\client\ClientDB.py", line 9229, in _LoadIntoDiskCache
self._InitDBCursor()
File "hydrus\core\HydrusDB.py", line 559, in _InitDBCursor
raise HydrusExceptions.DBAccessException( str( e ) )
hydrus.core.HydrusExceptions.DBAccessException: database is locked

2020/07/20 17:29:27: starting services…
2020/07/20 17:29:27: services started
2020/07/20 17:29:48: If the db crashed, another error may be written just above ^.
2020/07/20 17:29:48: A serious error occurred while trying to start the program. The error will be shown next in a window. More information may have been written to client.log.
2020/07/20 17:29:48: Traceback (most recent call last):
File "hydrus\client\ClientController.py", line 1805, in THREADBootEverything
self.InitModel()
File "hydrus\client\ClientController.py", line 772, in InitModel
HydrusController.HydrusController.InitModel( self )
File "hydrus\core\HydrusController.py", line 515, in InitModel
self.db = self._InitDB()
File "hydrus\client\ClientController.py", line 189, in _InitDB
return ClientDB.DB( self, self.db_dir, 'client' )
File "hydrus\client\ClientDB.py", line 309, in __init__
HydrusDB.HydrusDB.__init__( self, controller, db_dir, db_name )
File "hydrus\core\HydrusDB.py", line 261, in __init__
raise Exception( 'Could not initialise the db! Error written to the log!' )
Exception: Could not initialise the db! Error written to the log!

2020/07/20 17:29:48: boot error
2020/07/20 17:29:48: A serious error occurred while trying to start the program. The error will be shown next in a window. More information may have been written to client.log.
2020/07/20 17:29:50: boot error
2020/07/20 17:29:50: Traceback (most recent call last):
File "hydrus\client\ClientController.py", line 1805, in THREADBootEverything
self.InitModel()
File "hydrus\client\ClientController.py", line 772, in InitModel
HydrusController.HydrusController.InitModel( self )
File "hydrus\core\HydrusController.py", line 515, in InitModel
self.db = self._InitDB()
File "hydrus\client\ClientController.py", line 189, in _InitDB
return ClientDB.DB( self, self.db_dir, 'client' )
File "hydrus\client\ClientDB.py", line 309, in __init__
HydrusDB.HydrusDB.__init__( self, controller, db_dir, db_name )
File "hydrus\core\HydrusDB.py", line 261, in __init__
raise Exception( 'Could not initialise the db! Error written to the log!' )
Exception: Could not initialise the db! Error written to the log!
Should I be worried? Version 402. Only happened this one time and I don't get any errors after I restarted Hydrus.