Thank you for these good ideas. I have incorporated them into my to-do.

Drag and drop windows for tag rules is something I had never thought of, but I like it a lot. It could also do file uploading/exporting and other such duties. I want to add thumbnail dragging and dropping in future, so this could be a great extension of that code.

Internal tag conversion is something that has been on my radar for a while– particularly for large but finicky jobs like adjusting page:n to page:n+1 for two hundred files. I haven't thought about what the dialog would look like for such a complicated and powerful tool, though. Maybe split it into two halves, like LEFT: IF FILE MATCHES THESE PROPERTIES, MIDDLE: DO THESE THINGS. RIGHT: AFTER RUNNING THE SEARCH, THESE FILES MATCHED. THIS IS WHAT THEY WILL LOOK LIKE AFTERWARDS. YOU SURE YOU WANT TO DO IT?. Perhaps this is just an powerful formalisation of the drag and drop idea above. Perhaps I need to write a scripting logic for the program first, so people can do these mass edits a bit easier?

Allowing more complicated hotkeys could also use this scripting/macro language, and would be the easiest of the three things to do. I'll make a note to do that first.

Making the media viewer's elements float above the image on mouse-move is high on my to-do list.

Most of the video and audio stuff was cobbled together, and I need to revisit it. Adding default action options for different mimes is a good idea.

Find similar is another thing I threw together. I've been meaning to expand it for a long time, but never got around to it. I would still like to add a 'find me all the dupes in my collection', but there are several big new things I need to create first. For now, I can definitely add a custom default distance for the menu entry.

Not OP, but I have some suggestions as well. I'm new to this program- just found it several hours ago, so I might have some misunderstandings. I have been waiting to find this type of software for a decade. Thank you.

The installer defaults to c:\hydrus, saying that "program files requires admin". Not necessarily true- give the user write access to the database folder, or simply leave it writable by the 'Users' group. Done. This is what steam does.
But hydrus only has one database, correct? Therefore, in a multi-user environment each user must have their own installation of hydrus. The convention for a program without multi-user support is to default install to "c:\users\[username]\appdata\roaming\hydrus". Use the following variable, don't hard-code the path: "%APPDATA%\hydrus". If you point to appdata by default, be sure that you do not have the installer request UAC elevation, as it currently does. I'm not intimately familiar with install scripts, but I believe it should be possible to only request elevation if the installer needs it (eg should the user manually enter 'program files' as the destination). Just remember to manually grant database write access to the initial user (if not the Users group), NOT the elevated user, and you should be able to install to any directory without issue.

I'd like to suggest a random sort order where multi-page images are kept together. pixiv often has two or more images that are meant to be viewed in sequence, and I'd like to have a sort order preserve this in an otherwise random slideshow.

Thumbnails larger than 200x200 and/or display larger thumbnail on mouse-hover.

I'm not entirely sold on the *complete* abandonment of file names. Correct me if I'm wrong, but there are no file names in hydrus, everything must be a tag. Tags are for navigation, and file names can contain information that would just clutter my tag list with useless bloat. Example: MPCHC video snapshot creates a file with name "[filename][timestamp].png. The file name usually contains the episode name and number. There is no benefit to marking the episode name, number, and timestamp as tags, but this is also information I don't want to discard. If hydrus simply kept the file name… Also, the way you have hydrus store images sort of prevents operating on a picture library with external tools, such as a duplicate image scanner. Without human readable file names or a coherent, modifiable folder structure, I would have to export my entire database, delete all images in the database, do whatever needs to be done to the files, and re-import them. Why does hydrus simply not maintain a database of files as they are? Why must it completely take over, move and rename everything? It seems unnecessary and overbearing.

The internal image viewer navigates through pictures within the currently defined filter. If you open an image with an external viewer it will navigate effectively randomly; whatever pictures happen to be stored next to the one you initially viewed. Perhaps you could implement the option to hardlink all images in the currently defined filter to a temporary folder. External viewers would become more useable.

Different slideshow schemes. Let's say I have a set of tags 1-9. I want to view a slideshow of tags 1-3, then after a set time (and/or triggered by a keybinding), I want the slideshow to display tags 4-6, and after that, 7-9. Basically: just the ability to queue up different slideshows in advance.

Thanks for your time.

A few more things:

tag censorship of namespaces appears to be buggy or underdocumented. Inputting

page:

into the blacklist does not get rid of pages. If I literally define page:1, that explicit page will be censored.
The input textbox doesn't autocomplete tags like every other tag searching text box in the program does.
There is only a blacklist OR whitelist- why not both?

Censoring all namespaces with : works, and that allowed me to see that the censorship doesn't work in the way I had thought it would. Maybe I'm not in the correct mindset yet, but I navigate my picture library by choosing tags to filter by. By default, hydrus is blank, so I always throw in a system:everything. This displays all the tags in my library in the "selection tags" box, and it is from here that I choose tags to filter by: it's much easier to scroll through a list of tags than it is to try and remember what words I may or may not have used to describe my pictures. In no way, ever, is the page: namespace a useful sorting tool- HOWEVER it is necessary data to have for sorting, and I do appreciate that the page number is shown on both the thumbnail and in the internal picture viewer. Censoring page: removes it from the thumbnail and internal picture viewer. All I want is to hide tags from the "selection tags" list! Could you implement a "hide these tags from selection tags" list please?

>>514
>In no way, ever, is the page: namespace a useful sorting tool-

derp, I meant it's useless for filtering. It's useful for sorting

>>513
Glad to see more people interested in hydrus.

> Databases and multi-user
While hydrus by default only uses a single database, you can extend it by using some special databases with hash and tag information ripped from websites that collect such things.
In addition, hydrus uses the mobile installation method, with the program and its data closely coupled, which I find a much better way to do it.
I find your interest in this aspect of application development interesting. Could you expand upon why this should be a feature?

> Filenames and location of files
Filenames in and of themselves can be cluttering, agreed, but by making pretty much everything a tag, you guarantee much easier forward compatibility. Using the "title:" namespace should do what you want.
The reason to "take over, move and rename everything" comes from its mobile nature. This is one area where I would like some more options though.

> Slide show
With that much power you might as well just plug a scripting language into hydrus.

> hide these tags from selection tags
This would help with a lot of other namespaces too, for example "title:". I would very much appreciate this feature.

>>516
>Could you expand upon why this should be a feature?
I'm not sure I follow- I'm suggesting he follow the conventions of the OS. For example, a convention on linux is that you don't design your program to run as root. If a program does run as root, then changing that is more of a bug fix than a feature request. This context is not so dire, but the analogy is clear: I'm offering solutions to a problem, not asking for a feature.

It occurred to me that another way of doing the above would be to install to program files and have the database be in %APPDATA%\hydrus, that's probably the best way to do it, unless you want to keep the mobile approach and just install to %APPDATA%\hydrus. As for "mobile approach", well isn't that what the mobile, zipped version is for? There's no reason the installable version should defy host OS convention so there can be two redundant distribution methods.

>Filenames & 'taking over' folders/files
I would be more amenable to the way things are currently done if I could automatically tag everything with a filename: and folder: namespace upon import and hide it from the 'tag selection' as per my other feature request. If done right, exports could restore the files and folders exactly as they were. While this is indeed possible to do at the moment, you have to manually define the regex/tag pattern every time you import or export- which brings me to a new feature request: import/export profiles. Save a predefined set of conditions for future imports and exports.

>slideshow, might as well scripting language
Perhaps. I see hydrus as very similar to foobar2000: tags and the power to manipulate them. foobar2000 does not have a scripting language but it is possible to generate a set of playlists using tags, and bind keys to change the currently playing playlist. My feature request is best described as a set of playlists, which I don't see as a very tall or complex demand. I think a scripting language would be extreme overkill for such a feature.

>>513

>I'd like to suggest a random sort order where multi-page images are kept together. pixiv often has two or more images that are meant to be viewed in sequence, and I'd like to have a sort order preserve this in an otherwise random slideshow.

As long as the order of the images is part of some kind of tag namespace it wouldn't be hard to add that. An option to randomize one namespace and keep all the others' orders would do what you want.

>If hydrus simply kept the file name…

I have a namespace called "filename" that I add to every file I import. If I ever want to get the original file back all I have to do is export it and use [filename]. This is the regex I use:
[^\\/:*?"<>|\r\n]+(?=\..*$)
If you want to add it to files you've already imported, just import them again and it will tag their existing database entries.

>Perhaps you could implement the option to hardlink all images in the currently defined filter to a temporary folder. External viewers would become more useable.

Oh I like this one, though I think read only symlinks would be safer. Otherwise it would be easy, create a folder named something meaningful from the search, and symlink every file into that folder.

>>518
>An option to randomize one namespace and keep all the others' orders would do what you want.
I don't have any formal manga/comic collections where I would use series, volume, title, chapter AND page, so I don't want to presume how this should be implemented, but I don't think that accurately describes what I want. Perhaps this would require more than one sort rule to fully flesh out the idea for all use cases, but all I want is a given collection of pages to be presented in order. Perhaps we should define four sort rules:

random + preserve page order for title
random + preserve page order for chapters
random + preserve page and chapter order for volumes
random + preserve page and chapter and volume order for series

That doesn't even cover all possibilities. Optimally, sort orders should just be user definable.

>I have a namespace called "filename" that I add to every file I import
Yes, I saw the thread for "better regex" here, and I have been using the filename: where necessary, but it's a band-aid solution for a problem that hydrus goes out of its way to create! Hydrus hashes every image and renames the image files to their hash, correct? Then when importing a picture, if the file names match no duplicate is added. How about hydrus has the option to NOT move/rername files, and hashes are stored as a hidden tag (or otherwise in some form in the database)? This approach would actually be more efficient too, since it involves less read/write operations to the disk and relies on the database instead of the files. I get that you can live with tags and tags alone, but throwing out file names altogether is just needlessly limiting options.

Moreover the band-aid solution of creating a filename: tag doesn't address the issue that, once pictures are inside hydrus, access is relatively exclusive to hydrus, which is a problem. See next paragraph for an example.

>just import them again and it will tag their existing database entries.
Yes, I have noticed this and I appreciate it deeply. I have made use of it several times already. However, if I'm exporting the entire database to perform operations on the files with some other program, the import will not work unless I delete the database first. Changed files will be seen as new files: duplicates everywhere. Deleted files will still exist in hydrus' database… Not starting from scratch would just be a mess. Using some other program to operate directly on the database images \could\ work if hydrus is smart about handling unexpected changes, but from that program there will no identifying information on the pictures; this would prevent me from making an informed decision for a process such as batch removing duplicate images. If two+ pictures are near identical, I'd prefer to remove the one with less/no tags, but I will have no way of knowing which picture that is. Similarly, if I ever want to open ANY picture with ANY program, I can't use file -> open in said program anymore- I'd need to open hydrus, export the picture, THEN open it with any given program. If hydrus allowed having coherent file names and structures, I could process my library so that tags are listed in the file name. Do this, and hydrus will go from being a roadblock, to a lubricant.

>though I think read only symlinks would be safer.
symlinks require administrative powers under Windows, hardlinks do not, which is why I suggested hardlinks. If symlinks prove superior for whatever reason, then by all means it should be default in the linux version, and optional non-default for windows. I have to wonder how efficient the process of creating thousands of links is… we might run into a problem where it takes so long it just isn't feasible.

>create a folder named something meaningful from the search
I was thinking that it would be linked to a system temp folder- no need for a meaningful name. If you want to save the folder for later you can rescue it from the temp folder manually, rename it to whatever you want. Well, hydrus could automate this process for you too, if the dev feels so inclined to write it.

Instead of starting a new thread I'll post my suggestions here too.

A "suggested tag" feature. When you're adding tags it should show tags most commonly used with the ones already added, as if you're doing a search for images with the incidence (desc) option. For example when I tag an image with character:alice margatroid it should suggest the tags blond hair, blue eyes, and hairband.

I don't mind that Hydrus renames and moves files into its own directory, but I would like an option to have the subdirs of client_files and client_thumbnails folders be stored as compressed archives (zips) instead of loose files. I currently store images this way and use a cbz reader to view them in order to reduce wear on hard drives. When you get larger collections (literally millions of tiny files) doing things this way almost becomes a necessity. Plus all that file slack adds up.

Lastly some minor changes that I think will make life easier.
An option to change the default tag repository when opening the manage tags window to a remote tag repo instead of local tags.
When filtering with the manage tags window open the left and right arrow keys should change images and apply tags on image change.
The hotkey to open file externally should open the current image when filtering.
Right click on an image -> favorite/dislike along with system:favorites and system:disliked or some other form of bookmarking images.
Some way to filter images into safe, questionable, and explicit categories. Perhaps this can be addressed via tags instead of adding a new feature?

>>520

>Then when importing a picture, if the file names match no duplicate is added.

I'm not sure if you're saying that this is a problem or not. Yes that's how it works, since a matched hash is necessarily the same picture.

Hydrus makes the rather interesting assumption that an image is immutable, in the same vein as Tineye where there is One True Image representation of a particular artwork. Resized and watermarked images are indeed duplicates in the sense that they show the same artwork, but Hydrus isn't that smart, to it they're just different files.

As more people use Hydrus, we could end up with the interesting problem of lower quality images not getting all the tags of the higher quality ones in the tag repositories. Eventually we could need to start adding reference tags to images like "deviantart_resized_version_of:*hash*"

It would be nice if Hydrus had a simple way to replace files and keep their tags. Eventually it may end up using GUIDs for the files that reference the hashes. That would allow more than one file to be the same "artwork" or whatever the GUID represents.

But then you have the political problem of being the curator of the One True Artwork GUID Database. What if someone makes a competing program with their own GUIDs for artworks? Having hashes for the image primary keys avoids that potential drama at the very least.

>>530
GUIDs would be step backwards to be honest. The whole reason hydrus uses hashing in the first place is to be as decentralized as possible, while still communicating with other clients about the same things.
I think this would be solved with the idea of 'hash siblings'. If a user find that one image is indistinguishable from another, but not entirely (resize, watermark), the user should be able to say that the images are siblings. Every tag one image receives should also appear on the other.
Assuming there are enough resized/watermarked image mappings in a repository, this might reduce its sized, in addition to time required to sync.
This is not a perfect solution though: You cannot add a tag exclusively to one image. If such a system could be devised, would it be applicable for use on different versions of images? (Inheritance would be too complex, and I think we need a flat structure for this.)

>>530
>I'm not sure if you're saying that this is a problem or not. Yes that's how it works, since a matched hash is necessarily the same picture.

I know duplicates are identified by hash, but I was assuming (having not looked at the source code) that the actual comparison is done by checking identical file names during the import process. As in, "x file name already exists, therefore this image is a dupilcate". I proposed storing the hashes in tags (or in some form in the db) instead, and finding duplicates through that as a faster, less disk intense, doesn't require renaming and moving files, alternative.

>As more people use Hydrus, we could end up with the interesting problem of lower quality images not getting all the tags of the higher quality ones in the tag repositories. Eventually we could need to start adding reference tags to images like "deviantart_resized_version_of:*hash*"

I'm merely using hydrus for organizing my personal collection, so I don't comprehend the logic behind this. Why would you want a resized version of the original? Unless you mean an artwork that has been converted to conventional wallpaper dimensions. Otherwise, I don't see randomly cropped / lesser quality / watermarked / etc images as anything but garbage- which is why I occasionally need to process the *files* in my library by a program that finds visually similar images… but hydrus makes this a pain, for no adequately explained reason.

To be clear, I'm not entirely opposed to the move & rename the files in a non human readable structure- if it is done for a good technical reason. At the moment, there doesn't appear to be a reason other than trying to shape the way the user thinks about organizing pictures (and perhaps, prevent the user from interfering with the database in a way hydrus isn't yet prepared to handle).

You know what might work? If instead of copying files to the database, they're hardlinked to it. The database could rename the hardlink duplicate however it wants without issue. Hydrus would have to monitor the link status of files to keep the folder and database synced. For example: if a file were deleted from the user's Pictures folder, the corresponding file in the database would not be a hardlink and become a normal file. Hydrus could detect that and ask you if you want to remove it from the database (or move it to a trash folder for later review, similar to the inbox folder).

Oh and I have another suggestion. This has been bugging the hell out of me: I need to be able to select & delete multiple tags in the tag editor. When batch importing pictures, let's say 30/100 pictures have a meaningful file name. I generate a filename: tag for each, and I want to remove the 70 filename: tags with garbage strings. I have to manually remove each of the 70 tags, I can't select multiple and remove them in chunks. This needs to be addressed

>>536

I also have not read the source code but my experience with importing an asston of duplicates suggests that filename is discarded immediately and duplicates are identified via hash. I think one of the first things Hydrus does when it imports an image is generate hashes, and if it sees a match in client.db it knows not to reimport the file (and instead to add any import tags to the existing record).

>>537
>I think one of the first things Hydrus does when it imports an image is generate hashes

Undoubtedly; each image is renamed to its hash and stored in a folder named after the first two characters in the hash, so even without cross referencing hashes, a duplicate cannot be imported.

Man I'm full of feature suggestions. Drag and drop the thumbnails in hydrus to interact with other elements from Windows.

For example, foobar2000 let's you drag a song out of foobar into a windows explorer folder. It will create a copy of the song you dragged in the folder you dragged it to. If you drag a song into another program (and the program supports opening files by drag and drop, which a surprising majority of programs do), the file itself (not a copy of the file) is opened in that program.

I see drag and drop import already works :)

Wow, this thread got big! Thank you all for your comments. I can't really marshall my thoughts into anything that sounds coherent right now, but I've not replied to this thread for too long, so I'll apologise and just say a mix of things. If you would like a specific response to something I miss, please say so or send me an email.

I mean for hydrus to be a store for files that are 'complete'. Such files can have their metadata changed, but there is no reason to alter their content. They are read-only.

Curating a thousand files in explorer is easy, but a hundred thousand is impossible. I see no point at all in trying to manage that many files in a brittle directory structure.

Throughout development, there has been a temptation to permit links, to allow hydrus to manage files at a remove, in their original locations, but this betrays my two former points. External files are subject to renaming, editing, or deleting, and they remain in an impossible-to-manage structure.

Filenames and folders are useful for things you are working on right now, where you might have several versions, or the program you are using to edit relies on filenames to manage library imports or whatever. The only file that should be imported to hydrus here is the final .png or whatever that comes out of the whole process.

Filenames and folders are also useful for a few hundred files, perhaps to be wrapped up in a zip and sent to a friend. Hydrus tries to support this with its export dialog.

Filenames and folders are not useful for storage or search, which is mostly what hydrus aims to do.

Any useful filename data, like in "artist_name-series_name-chapter_24-page_10.jpg", is better parsed and stored in an intelligent database that can regurgitate that information at will. Adding filename: namespaces is fine, but I think that data is better slowly converted to more granular and powerful tags. You can go tags->filename easily, but filename->tags takes more effort.

>>537 Has it right on hashes; the database stores a decent amount of metadata about files, including hash. Those hash indices are cross referenced whenever needed–the actual files are only ever loaded when the whole file is needed for something (display, uploading etc…). When a file is imported, its hash is calculated and then checked against the db–if it is already in the db, it tells the import controller 'redundant' so it can report that and move on to the next file. There's a fair amount of similar checking for image URLs as well, and md5 hash for APIs that report that.

>>513

I don't really want to mess around with %APPDATA%, because I want to keep the program together in one place. I want the program to be portable, easy to backup, and able to run from multiple locations at once without conflict. The Windows installer violates this, but I make that release to be a simple solution for people who don't want to mess around with extracting archives.

Having said that, if there is a good way to set the proper user access control to the db directory under program files with ISTool, which is what I put the Windows Installer release together with, then I am interested to know.

Keeping short sequences of images together is a continuing problem. page namespace works well for large groups, but not small. I may create a new file format, like .cbr is used in online comics, to represent these collections as single multi-page units.

I'll have a look at tag censorship for namespaces; thank you for reporting it. I'll also have a think about options for namespace hide on the 'tags in selection' box.

>>528

Suggested tags is something I definitely want to do.

You can change the default tag repo in file->options->gui.

Page up and page down can do that for the manage tags window right now. I can add this to options if you want, so you can change it to whatever you want (although binding left and right to a text control can be a problem, because then you can't use those buttons to move the caret around).

I haven't touched like/dislike and numerical ratings in a long time. I want to get back to them.

Doing safe/questionable/explicit has been tough for me to think through. I think you are right, saying do it with tags, probably with a 'safety' namespace or whatever. It is really subjective though–maybe it'll be better done with a numerical rating? If people want to filter safe vs explicit, I now suggest they just run two clients–one for sfw, the other for nsfw and anything else private. That way, they can browse their sfw client with someone looking over their shoulder and have nothing to worry about.

>>530

At some point, I want to make a service to accept and sync that sort of metadata to clients. I'll be writing a better dupe checker as well, and add a gui that'll show two very similar images to the user and ask questions like 'which is better?' and 'should tags be merged?'. The complicated parts of that data will be made automatic for users, so images will be replaced/rotated/whatever a bit like tag siblings are now. I'll run a public service, and anyone else who disagrees with my ruleset can run their own. I hope this is a decent compromise between the current mess and the impossible pursuit of 'file purity'.

>>536

I will see about selecting multiple tags–thank you for the suggestion.

>>540

Export dragging is unfortunately a massive pain to code. I have tried several times to figure a way to do it, but I can't find a wx-compatible multiplat solution for virtual file drag and drop. Dropping real writable files with nice filenames is easy, but I want to insert an interim step that'll copy and rename, which I can't figure out in wxPython.

>544
>I mean for hydrus to be a store for files that are 'complete'. Such files can have their metadata changed, but there is no reason to alter their content. They are read-only.

I think you've mentioned this is a feature you'd like to expand upon, but my problem is largely duplicate images that don't have exact hash matches. Until hydrus can process the entire library for duplicates, instead of just finding duplicates of just one selected image.. It will be necessary to periodically scan the library with an external program to find clutter.

>Curating a thousand files in explorer is easy, but a hundred thousand is impossible. I see no point at all in trying to manage that many files in a brittle directory structure.

I have fifty thousand pictures, but I see the value in having some easy access folders of pictures. I have a reaction images folder that is manageably small and is faster to access via Windows' indexed search from any given program's file>open prompt, than by starting up hydrus, copying the file location. I have a separate pornographic collection of images which I keep stored on a separate, encrypted drive, and I'd like to NOT mix those in with normal pictures. Regardless, I do feel you might be discriminating against potential users with smaller libraries that still want to benefit from hydrus? Well, I'll drop this because I think I'm starting to sound a bit entitled to have my free software exactly the way I want it. I suppose I could run a separate instance of hydrus on the virtual drive, and keep the reactions folder in synchronized export.

>Keeping short sequences of images together is a continuing problem. page namespace works well for large groups, but not small. I may create a new file format, like .cbr is used in online comics, to represent these collections as single multi-page units.
That sounds like a fantastic solution. Maybe add support for downloading multipage images from pixiv while you're on that line of coding.

>Export dragging is unfortunately a massive pain to code. I have tried several times to figure a way to do it, but I can't find a wx-compatible multiplat solution for virtual file drag and drop. Dropping real writable files with nice filenames is easy, but I want to insert an interim step that'll copy and rename, which I can't figure out in wxPython.

I'm not sure what you mean by copy and rename, but if I'm dragging pictures out of hydrus I do not expect them to have nice file names. I'd be perfectly happy with just the {hash}.{ext} file being delivered on drag and drop.

>>544

>Keeping short sequences of images together is a continuing problem. page namespace works well for large groups, but not small. I may create a new file format, like .cbr is used in online comics, to represent these collections as single multi-page units.

For the moment, I've made a sort order of "creator-title-page-filename" that will sort by the page namespace if I've made one, but if not, the filenames likely have page numbers in them and will keep the files in order.

>>544
>You can change the default tag repo in file->options->gui.
>Page up and page down can do that for the manage tags window right now.
Thanks. I managed to overlook these.
>I can add this to options if you want, so you can change it to whatever you want (although binding left and right to a text control can be a problem, because then you can't use those buttons to move the caret around).
That won't be necessary. The problem I had was that I was using the arrow keys to change image and opening the manage tag window prevents those keys from doing that. Adding new tags will go much smoother now that I know page up and page down also change the image and that they do so while the manage tag window is open.

As for safe and explicit using a new tag namespace seemed to work well on my local tags, but there was no one to disagree with my opinion. Something else to consider would be having status decided by the image's tags. Images tagged with explicit tags 'sex, blowjob, handjob, etc' also get explicit rating automatically. Not that I'm opposed to such things. It's just nice to be able to see only safe content when looking for things like image macros, guides, wallpapers, and other times it's nice to be able to search for only explicit content because reasons.

I'm trying to download a tag of 800 images right now, and I just got 15 failed downloads. And I'm right in the middle. They were mostly 503 errors.

So now my options are to suck it up and give up, or restart the entire download and wait 2 weeks till the slow-ass downloader gets to that part. Sometimes you get a link to the image (that you have to re-type out yourself if you want to retry) but not this time.

Can we PLEASE finally get some proper handling of download errors? Because what hydus has right now is a joke. On one hand I remember you talking about being nice to the various booru sites when scraping tags, which I understand perfectly, but then there's this shit.

>>562
Downloading is complicated. There are a lot of unknown factors when communicating with other server, especially when those servers are hostile to automated downloads. This is properly not something hydrus_dev wants to spend their time on.
There is a solution though, and this is more a feature request than anything else:
hydrus_dev, please expose some sort of interface for us to create downloaders. This would allow us that care about this to quickly respond to site changes or add new sites, and create robust interaction code, while you can ignore all of this entirely.

The other problem with using hydrus for personal use and sync from a laptop is that hydrus cannot view images remotely. Things that would be perfectly viewable over samba are downloaded, and the download process, as mentioned above, is annoyingly prone to failure.

Unless hydrus implements some local network conveniences within the client/server, or unless hydrus stops taking over files… I don't think hydrus will be an effective tool for a personal collection of images.

Like I keep saying, the insistence on taking over the files creates problems…. As is, hydrus is only good for personal collections if you don't use the hydrus server and exclusively access pictures from that one computer (eg not over LAN).

Maybe there needs to be a fork of hydrus with a focus on home users, rather than massive collaborative libraries.

>>573

you could put your Hydrus folder in Dropbox or set up a cross-network XXcopy script that runs in the middle of the night every night or use one of a million other generalized solutions for synchronizing directories between multiple devices that you own.

>>574
>>573

Hell, I primarily use Hydrus on my desktop computer over a Splashtop remote connection. That'd work just as well, better even, on a laptop over LAN than it does on a tablet over LTE.

>>574
Dropbox/cloud in general: No.

Syncing hydrus itself: Not a solution. That would mandate that I only ever make changes from one computer, because when syncing, the database file will be replaced, not merged.
Also, this method requires that I have the space for it on every laptop I use it on, and pictures can only be accessed on the laptops/PCs I specifically prepare before hand.

Using RemoteDesktop is the best solution so far, but… Really? That's just a pathetic work around for, again, a flaw that doesn't actually need to exist.

I'm just going to be super blunt: I have a collection of pictures I fap to regularly. Hydrus' tagging is a blessing, but the accessibility implementations are a curse. I had previously put pictures in folders and subfolders, which effectively means that any given picture can have one tag and infinite parent tags. For example, a folder of ahegao, with subfolders 'exhausted' 'shocked'. I use any given image viewer to create a slideshow of any number of folders (eg tags) and fap away. Hydrus lets me apply multiple tags without having to define a picture entirely by one trait, but then denies me simple, functional access.

It would probably be easy to implement an option, "remote files are on LAN" for the files service. If this option is checked, the viewer, "right click > share", etc operations would know the files are located remotely (\\pc\share\db\server\00\hash.jpg)
The user would have to set up the share and there'd had to be another option "database location" so you could type in the exact location of the db folder…

>>576
>I'm just going to be super blunt
I forgot to mention, these pictures are stored on an encrypted virtual drive on a remote PC. Hydrus interferes with the encrypted part (is it possible to run two servers on one PC?) and the remote part. Yeah, I get that tags are super flexible, but hydrus itself is incredibly inflexible.

At this point I'm really just considering using hydrus to help tag everything, but once I'm done I'll export pictures with the tags as the file name. Maybe I should take this opportunity to learn python and make a fork, if my requests are not agreeable with the dev.

>>577

You can definitely run more than one server on the same PC as long as they are using different ports.

I can see wanting to be able to edit stuff remotely, but that's what remote desktop is for; for faptimes, a synchronization of the "master" DB on your main computer is fine because presumably you are not editing while fapping.

I seem to recall the dev saying he wants to majorly improve the server, but we'll see what he says to all this. I agree with you that it'd be hypothetically useful to have an instant-sync master DB on the server that all the clients seamlessly access remotely, and my understanding (though I don't use the server) is that this is not how it currently works.

>>579
>I can see wanting to be able to edit stuff remotely, but that's what remote desktop is for; for faptimes, a synchronization of the "master" DB on your main computer is fine because presumably you are not editing while fapping.

Why are you adding unnecessary complication? Syncing takes time and disk space, and if you have as many pictures as I do, you won't be seeing even 5% of it all before finishing.

View the pictures over samba shares. Done. No wasted sync time, no wasted disk space on the laptop. Nothing to do with editing.

>>580

I'm a brute force solution kind of guy. They often work better, in my experience.

>>548

About drag and drop, some things about the database have changed since I last looked at it (files are stored with extensions, for instance), so that now I think of it, it won't be so difficult to do an ugly drag and drop. I will plan a prototype.

>>562

How about if I add an expandable control (like advanced tag options) that will list all the attempted urls and their successful/redundant/failed state, and then provide a right-click menu to retry failed ones?

>>573
>>576
>>579
>>580

I am planning to add multiple storage folder support for the client i.e. 'Hey, hydrus, you can store 200GB in this folder and 300GB in this folder'. This came from a convo with a guy who has nine million files, which obviously won't fit on one drive. The client would just balance file storage intelligently through those folders and occasionally 'defrag' itself to rebalance things. That should work over Samba. The client.db will still be stored under the main install dir to keep the CPU-intensive stuff quick.

Does that sound like a fix to this problem?

>>597
>That should work over Samba. The client.db will still be stored under the main install dir to keep the CPU-intensive stuff quick.
>Does that sound like a fix to this problem?

If you add this feature to the server, it will fix things. I want to be able to access the library from multiple locations, and if I understand this correctly, one database = one client = I won't be able to manage the library from the computer hosting the samba shares.

>>597
Yeah, having url-by-url control sounds great.

OK, I have a number of trivial suggestions Home/End keys don't work in certain/most text fields, but they don't seem to be used for anything else in that context. Could we get those working? I love sharing files through the booru, but I hate the incredibly long obnoxious URL. Might it be possible to make the booru URLs more manageable? The 'share to booru' feature gives me a 'copy local url' and 'copy external url'. I'd like the ability to specify an IP/hostname for the external url. I have a vLAN/VPN with friends, so I have to modify the URL every time. Others might appreciate being able to use no-ip or something rather than a raw IP. The 'paste tags' feature doesn't seem to work, at all. Maybe I'm using it wrong. I copy tags from one picture and try to paste them onto another, but the result is garbage (a lot of one letter tags). Alternative views for 'tag parents' page. I'm thinking a tree view: show each parent tag once, and all the children underneath, indented. This will better illustrate multiple levels of parents (like you have already done in the tagging text field). On the subject of parent tags, there doesn't seem to be a way to back those up. How about the ability to export tag relationships? (parent, sibling, censorship) The image viewer's share menu is less complete than the normal share menu. Since the image you last viewed is automatically selected, this isn't a big deal, but why is this? thanks

An option that if you tag one file, all the files that have the same tag in a namespace (e.g. the same title) get that tag also.

Some more flexibility with scheduling subscriptions would be good. Take these examples: >update every Saturday at 2am >update on the seventh of each month >update on the first Tuesday of each month

>>615 If you point two databases at the same file store, you need to be careful with files orphaned in one database getting deleted out from under the other database when the client runs a cleanup. Otherwise, technically there's nothing keeping you from doing what you want.

It'd be nice to have a way to quickly add or remove tags to files without having to open up a new dialog each time.

>>679 Press F3 to open the tagging box Press pagedown when in the tagging box to move to the next image, without closing the tag box.

>>650 Thank you for all these. I think home/end is eaten up by the taglist beneath an autocomplete tag entry if there are results. When there are tag results, home/end scrolls to the beginning/end of the list of tags. Same with page up/down and cursor up/down. I'm not sure if that is helpful–maybe this behaviour should be optional? I want to revisit boorus, including adding aliases. I don't have time now, but it is on my to-do list. Adding the better external url options is a great idea. The way tags were copy/pasted in the tag dialog was an old method. I have updated it to use the same data protocol as clicking 'copy all tags' on a tags list, so this should work in future (although there are some graphical bugs that I'll clean up later). Thank you especially for this. Better views for tag parents and siblings is something I really want to do (and maybe ways of filtering, like you might on a web browser's shortcut editing page), but I can't find time. It is on the list, but low priority. Same thing for exporting them. I'll get around to it someday! The share menu on the media viewer is old. I've made a note to update it. >>657 I think that tag parents can do what you want. Let me know if I've misunderstood, or if you would like help getting into tag parents. >>666 Good suggestion, Satan. A good time to do this would be when I move subs over to JSON, which should be in the next month or two.

>>680 it doesn't seem to be working for me. But I was thinking something like being able to manipulate tags from the selection tags box.

>>685 >I think that tag parents can do what you want. Not really. The idea would be to give tags to all members of a set at once. Most mangas share one tag for all images in them in many repositories, simply because even if you search for a tag, you typically want to read the entire manga. So if I want to add tags to the entire set, it would be nice to add them quickly instead of making a search page for the set and control-A F3. Now that I think about it, tag parents could work if I made e.g. "title:this comic" a parent of all the tags the comic is known to have. However I would have to maintain a huge ass list of tag parents, one for every title I have.

>>689 I'ma go ahead and say your use case is too niche, and also misguided. The change you propose would be silly. All this, for the ability to find manga by tags? Just tag the first (or any) page, then. It's not that hard to search for a tag, filter to the desired title:blah namespace that pops up in the list, and remove the initial tag filters, so that the only filter remaining is title:blah. Done. No need to do screwy things to hydrus namespaces.

Is there full boolean search? What would i have to do if i want to view all images with at least one of a number of tags at the same time.

>>690 You seriously need to chill out. What would be changed in the namespaces that would constitute "do screwy things"? My suggestion wasn't to change the namespace handling at all, it was to give the option to find the namespace and add the same tag to all the files with the namespace in one step. Seriously, you presumed that I didn't know that you could refilter on a file once you found it? You have serious issues if your first reaction to something that doesn't make sense in your mind is to lash out with words like "silly" and "screwy."

Can we please get a way to add tags with a hotkey or a right click menu? How about a modified version of the archive/delete queue where the left and right mouse buttons can be assigned to give different tags?

>>698 That exists- select multiple images, right click > filter > custom filter >>697 Wow you're mad. My statement was not made with any vehemence or emotion, I'm sorry you interpreted it that way. I stated my case in what I thought was simply a 'matter of fact' tone. I stand by what I said- your request does not solve any particular problem and only serves to potentially screw up the database with its wide reaching implications.

>>544 >>597 Looking at this: http://wiki.wxpython.org/DragAndDrop#CA-cc495b3a21b78daea18c76017d6b2e6ae2c1b3c1_133 I've read up on DoDragDrop and apparently it blocks, so you could use that to your advantage here. I am thinking something like this (maybe you've tried this already, I don't know): def OnDragInit(self, event):
original_filename = 'somehow figure out the filename'
dest_filename = 'the new filename'

temp_dir = tempfile.mkdtemp()
try:
dest_path = os.path.join(tempdir, dest_filename)
shutil.copy(original_filename, dest_path)

file_object = wx.FileDataObject()
file_object.AddFile(dest_path)

tds = wx.DropSource(self.text)
tds.SetData(file_object)
tds.DoDragDrop(True)
finally:
shutil.rmtree(temp_dir, true)
On this page, http://wxpython.org/Phoenix/docs/html/FileDataObject.html , it mentions that FileDataObject isn't fully cross platform yet (it contradicts itself at one point too…). To be honest I would not worry about that, I doubt it's possible to get anything better and at this point you're only limited by what wxPython can do.

And I forgot to capitalize True which shows how little Python I've been doing recently

>>700 >Wow you're mad. My statement was not made with any vehemence or emotion, I'm sorry you interpreted it that way. I stated my case in what I thought was simply a 'matter of fact' tone. I stand by what I said- your request does not solve any particular problem and only serves to potentially screw up the database with its wide reaching implications. You bet I'm mad. This is exactly what I have an issue with: your attitude. You have some idea of what a non screwed up database is but you don't say anything about what the implications are in terms of my suggestion. You're being condescending and dismissive, and to top it off you cop out your comments by saying you were just trying to be matter of fact, which is a failure in and of itself because it is not possible for something to objectively be "silly" or "screwy". I'll grant you one thing: my suggestion does not solve any particular problem. Let's recap: I want to be able to select a couple images that may be part of a title namespace, and with one right click or similar step I want to apply a tag that the selected file has to every other file that shares the title namespace. I envision right clicking "creator:kikuchi hideyuki" and a menu pops up saying "Apply tag to -> title:vampire hunter d". What it would do is change a process that takes about six steps now and make it take two steps instead. And I feel it's worth it, because I'm getting tired of tagging everything in a manga by opening new search windows for the namespace tag and hitting select all. You may not feel it's worth doing, and you obviously cared enough to post. Explain yourself then. Twice now you've said that my suggestion would damage some ideal you seem to hold about the database. What is it? Do you feel that tags belong to images only and not collections? If so, then why is it that every manga collection site tags based on collections and the images have nothing beyond the page number to identify them? Or is there an easier way to do what I'm trying to accomplish? Keep in mind I'm probably coming from an already filtered result page where I've found one image out of the hundred or so in the collection and I've hit upon someone else's tagging from the public tag repository which I want to add to the rest of that title.

>>691 Not yet. I'm all for adding it. How would you like the workflow to go? What series of clicks and button presses would say: TagA OR TagB OR TagC Would making shift+enter start/continue an OR chain work? Would such a predicate be editable when it is active? (when it is above the autocomplete entry) How would it be editable? Where would an incomplete chain be displayed? Any thoughts you have would be useful! >>698 You can do this with a custom filter, although the interface is not ideal. I want to make shortcuts custom across the program so that eventually you'll be able to set specific tags or ratings from anywhere. It'll just take time. >>701 I planned something similar to that when I was storing files without extensions, but I hated the overall idea of having to copy as an interim step. It also means I have to copy all the files in the gui thread (which blocks) before the DnD finishes. If the user accidentally dragged, that means ui delay. What I really want is: Create a DnD object, throw it at windows. Windows treats it like a file DnD, modifies the cursor appropriately If the user releases on a droppable location, the DnD object is informed, including being told that path. Windows doesn't do anything else. I can then copy the files to that location at my leisure, in a background thread, with complicated renaming happening dynamically. But I think that would require a shell extension, which is both beyond my experience and non-multiplat. I tried looking through the Filezilla Client code to see how they do it, and I think they drop a file with a weirdass tmp name and then have a ShellEx that locates/tracks that file and immediately deletes it. They then know the dest_path and can stream ftp stuff at it as they like. Since I now store files with extensions, throwing "{hash}.jpg" at a copy FileDataObject sounds like an ok compromise. I want to add thumbnail dragging to change thumbnail order as well, and also dragging thumbnails between search pages. >>703 This is an interesting thought. I see that parents don't quite fix your problem. Collections are proving a lot more complicated than I originally thought, and I think the ultimate solution I'll move towards is to bundle them up into single media objects–.cbrs or whatever. There are very few times you want to deal with page 62 of a comic alone, and many many times you want to deal with a whole chapter/volume. Tagging should probably work the same way. I can add what you want, but I am not sure how to avoid false positives. What is to stop an over-eager user tagging creator:whatever onto series:whatever? Parents and siblings all go through a janitor process for this exact reason–they are powerful enough that a mistake can be ten thousand files tagged wrong. Perhaps this is something that could only work for title tags?

can you make it so we can control what types of files are imported automatically by hydrus? I've got it set to import from my downloads folder, but I only want it to try to import images, not videos, archives or other weird files.

>>837 Yes, I can do this. I have added it to my list. I will need to create a new control to handle selection of which mimetypes you want–I suppose a list of checkboxes works, right?

>>839 that would be perfect, thank you.

Sure, maybe in a tree of "image/animation/audio/video" While you're in there can you add a way to move imported images somewhere instead of just deleting them?

>>846 I think I'll change it to be something like: successful files->do this with them failed files->do this with them redundant files->do this with them already deleted files->do this with them with the options being: do nothing move to location x delete Or something like that. Hence I'll be somewhat merging the 'synchronise' and 'delete' modes into a generalised and more granular solution. And maybe I'll add a checkbox for 'don't retry paths you already tried once', to optionally preserve the synchronise mode's current efficient behaviour.

When I want to share multiple images at once (such as when dumping several on an imageboard), it's a bit of a pain to have to right-click->share->copy->copy path each time. Having to go manage->files' tags is also rather redundant, especially since manage only has one option in it (although I usually end up using f3 anyway). Can we have a way to customize the right-click menu so that favorite actions would be more easily reached? Or maybe a better option would be to add more possible actions to shortcuts. Being able to assign ctrl-c to "copy path" of the currently selected image, for instance.

>>858 I plan to improve shortcut customisation in the near future. That said, at the moment hitting Ctrl+C copies the currently selected files to the clipboard, and if you only copy one, Windows should paste that as the path in a normal file selection dialog's path text box. I don't know about other OSes. This week, I'll change manage->tags to be just 'manage tags' if you don't have any ratings services.

>>859 oh yeah, you're right, that does work. nevermind then

Can you make the danbooru downloader bypass the loli block? https://greasyfork.org/en/scripts/3575-better-better-booru This userscript does it well, I can give a simple explanation of what it does if you plan to implement this.

When downloading from a booru/thread/etc., could it made so you can block files from importing that have certain tags? Would a flow like this >define tag x as blacklisted tag >do a booru search for tag y >find 10 results with tag y >booru or tag database says 3 of those 10 results also have tag x >download only the 7 files that don't have tag x >add the hashes from the 3 barred files to a list of files not to be downloaded again, similar to when deleting a file be possible to implement?

Couple things I had trouble with trying to archive an artist: >got random failed images with a database error implying that it tried to access an object that didn't exist when pulling from dA; I suspect these are mature images, add an option to feed dA a user account similar to pixiv >add a Blogger downloader

I'd like to be able to drag images from the client like with Windows Explorer. So for example I can drag an image from Hydrus to 8chan to upload it.

>>876 Hentai Foundry does something like this–I have to push some search options before parsing gallery pages to get all the super-rude images to display. Does danbooru work similarly? A simple summary of that script would be great–just the POST calls I need to make or GET args I need to add to unfilter the gallery results, or whatever! >>877 This is slightly tricky, because hydrus can usually only link a file to a url after the file has been downloaded (it needs the file to generate the hash). I can add a censorship taglist fairly easily, but to be comprehensive, I would probably add the filter check to after the import. I am thinking of adding a 'recycle bin' service soon, so I would probably move a censored file to that immediately after import and inform the import page that that had occured, so where it says '3 successfuil, 1 failed', it might also say '2 censored'. Is this what you are looking for, or were you specifically looking to reduce bandwidth by censoring before the download? >>878 Do you have any of those errors still, that you can post here/email to me? They were probably written to your client.log, which is in install_dir/logs. I am not sure what you mean about feeding dA a user account like pixiv. I thought my dA download page already searched by artist name? Is there a way of searching with some numerical user_id? I can definitely add a blogger downloader. Do you have an example blog of the sort you want to parse? >>882 This has been a problem in the past because I wanted to do it in a 'beautiful' way, but I think I will have to bite the bullet and do it in an ugly way that works for Windows, Linux and OS X all the same. I will make a job in my to-do list to prototype this. For now, you can do very similar just by hitting Ctrl+C on your selected thumbnail and then Ctrl+V on your browser's file selection dialog.

>>877 >>884 I was hoping not to have to deal with content that I'm not interested in by avoiding downloading it in the first place, and as you say, to save bandwidth. I get what you're saying though. I didn't know generating a file's hash was needed to get the remaining tags. Thanks for the response and keep up the good work m8

>>884 For danbooru you need to use the API for the searches http://danbooru.donmai.us/posts.json?tags=<tags> or http://danbooru.donmai.us/posts.xml?tags=<tags> whichever you like From there get the id and go to the post page http://danbooru.donmai.us/posts/<id> On this page you can either get data-file-url from #image-container or get the meta tags og:image or twitter:image:src these can contain a sample url, so you may need to brute force the extension

>>884 Not sure what you mean by "ugly" method but it works for me. The main thing I'm looking for is that it works for multiple files at once though, unlike the copy-paste method. Thanks though!

>>887 For multiple files, you may just want to export them all to a folder and delete the folder afterwards. A bit of a pain, but at least you can select them all at once.

>>877 Most boorus let you search for files without a tag. If you want tag x but not y, then you would just search for "x -y" A more permanent blacklist might be a neat idea, though. It could let me avoid loli without having to remember each time, for instance.

>>891 I know this and use this method where possible, especially if I strongly wish to avoid some content. However, like you said, it would be better not to have to make a new blacklist each time you do any kind of search. Another thought, some boorus have a limited number of tags allowed per search, and having a built-in option to filter unwanted content could partially get around that I think. Now I'm thinking about it, what about a system with a custom blacklist where you can place tags in order of severity? Here tags at the top of the blacklist get added to each search as a "-tag" if there's space in your search, and tags lower down the list, if there's no space left in the search to put tags, the files are downloaded and then censored automatically. mite b cool.

The Help mentions a hidden wildcard at the end of a search for tags. This is a good feature but it seems to make slower computers stutter especially when synchronisation is happening. Any chance for an option to turn this off / a button to autocomplete a search?

>>892 I think ideally, you wouldn't want to download a blacklisted tag at all if possible. Rather than just trying to censor what's being downloaded, I think the best option might be a client wide blacklist, so you can avoid blacklisted tags already in your database and from incoming images unless you explicitly search for those tags. Maybe an option to delete blacklisted images as well.

>>894 I agree with you there, I was just taking into consideration some things hydrusdev mentioned when first replying to me here >>884 >This is slightly tricky, because hydrus can usually only link a file to a url after the file has been downloaded (it needs the file to generate the hash). I can add a censorship taglist fairly easily, but to be comprehensive, I would probably add the filter check to after the import. Having the option to create a custom blacklist that appends itself to each search in "-tag" form should be enough for some boorus, but for others, like for example e621, which limits searches to 6 tags, and Danbooru, which limits searches to a mere 2 tags, some more flexibility is needed.

>>894 >>895 I have thought about this a bit and decided that I'll add it when I rewrite the downloader engine (which should be in the next month or two). I'll check for tags before I download a file, and if the tags include any censored tags, it'll skip the file. >>893 Do you want it only to return counts for the exact match for what you put in? Like if you type 'tank', you would only get: tank (210) or whatever? If so, you can probably do something this by changing the various thresholds of the autocomplete dropdown searches in file->options->maintenance and memory. The 'autocomplete' options are what you are looking for–hover over the number control, and you'll get a tooltip explaining what each one does. If you can't figure it out, just ask and I can explain more.

>>903 Once again, doing God's work.

>>884 >Do you have any of those errors still, that you can post here/email to me? They were probably written to your client.log, which is in install_dir/logs. > >I am not sure what you mean about feeding dA a user account like pixiv. I thought my dA download page already searched by artist name? Is there a way of searching with some numerical user_id?
AttributeError

'NoneType' object has no attribute 'parent'

Traceback (most recent call last):
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDownloading", line 1172, in __call__
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDownloading", line 1255, in _GetArgs
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDownloading", line 394, in GetFileAndTags
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDownloading", line 382, in _GetFileURLAndTags
File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDownloading", line 309, in _ParseImagePage
AttributeError: 'NoneType' object has no attribute 'parent' By "feeding dA a user account" I mean something like the "manage pixiv account" option for viewing mature images. https://github.com/xofred/deviantart-gallery-downloader does the same thing, but that uses Mechanize to crawl galleries manually so it may not be applicable to you. >I can definitely add a blogger downloader. Do you have an example blog of the sort you want to parse? http://fupoo.blogspot.com/ Right now using a URL downloader on that blog while fucking with the URL search paramaters sorta works but is clunky and doesn't grab images after the jump. And now that I'm thinking of it, I also got shit like this trying to crawl Paheal:
Database Error!

<include.ClientData.Booru object at 0x07617EF0> is not JSON serializable

Unknown Caller, probably GUI.

Traceback (most recent call last):


File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.HydrusDB", line 172, in _ProcessJob


File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDB", line 6007, in _Write


File "C:\code\Hydrus\build\client\out00-PYZ.pyz\include.ClientDB", line 4698, in _SetJSONDump


File "C:\code\Hydrus\build\client\out00-PYZ.pyz\json", line 243, in dumps


File "C:\code\Hydrus\build\client\out00-PYZ.pyz\json.encoder", line 207, in encode


File "C:\code\Hydrus\build\client\out00-PYZ.pyz\json.encoder", line 270, in iterencode


File "C:\code\Hydrus\build\client\out00-PYZ.pyz\json.encoder", line 184, in default


TypeError: <include.ClientData.Booru object at 0x07617EF0> is not JSON serializable


>>913 Thank you for this information. That first error is an unexpected failure of my parser to find a download link for a file. I have improved the error, and it should also print for which url it failed. If you see it again, please let me know the url so I can see what is going on. I expect it is a webm or something that uses a slightly different link-text than 'Save this file' or 'Save this video'. I didn't even know that DeviantArt allowed adult images! I can add a pseudo-login, but I would rather keep it simple if I can. I just had a poke around their html, and while they are quite careful to hide mature images and thumbnails (and even identifying information from which I could extrapolate a link) from non-logged in users, they acidentally leak the encoded image link in a tumblr share button. I don't think I'll have time today, but I'll put some time into it next week. Blogger shouldn't be too difficult–thank you for the example. The jump through the 'are you an adult' splash page will be an extra step to code, but that's it. There is probably an API to skip it entirely. I will look into it next week. That Paheal JSON error is a mistake from the gui session rewrite from a couple versions ago. Booru pages aren't being saved to the database properly, which also happens every few minutes to maintain the 'last session' gui session. It is just some annoying error spam while a booru page is open, but the downloader otherwise works. It is fixed in tomorrow's release.

Is it possible to implement a command in the context menu that sends a picture to IQDB? It would be great to open iqdb.org in the default browser, sending the current image to look for the source. I don't know if this could ever work though.

Would support for .zip or other archives be possible? i.e. given an archive file it attempts to import any images it finds. Exporting as a .zip might be nice, too

>>921 I am sorry, it looks like I missed this post. I'm planning to do something like this, but I've since added the thumbnail drag and drop export, which should speed up doing this the traditional way. >>1003 Ironically, I just removed this support a week or so ago! It was making my other code complicated, so when I went to rewrite it, I didn't bother to update the zip stuff. I can bring it back, but I'll probably create separate dialogs for it. I'll add support for more formats than just zip, as well, as here: https://docs.python.org/2/library/archiving.html

Could we get a feature to rename individual tabs in the session? Simple right click menu on the tab with a rename tab option opening a simple dialog with one textbox would be enough. Also, when you slim down the leftside panel with all the search and tags fields, the fields themselves stay fixed width. Given that some very long tags can make it scroll sideways (and the sideways scroll bar appears within the thinner panel without being hidden, unlike everything else in it) anyways, can the fields contract with the panel itself? Thanks for your time.

>>1030 Thank you, this is a great idea. Should be simple, so I'll try to fit it in this week. I'll have a look at the management panel minimum widths, as well. I noticed this bad resizing when I was working on the thread watcher the other day.

This may or may not be a sound idea: being able to customize how many files Hydrus downloads from a site before it "takes a break" for a time period before retrieving any more. I find most times when a file fails to download it's after Hydrus has already grabbed a hundred or so images, maybe because the site is trying to limit your access or something. Perhaps be able to make the downloader take a break as soon as a file doesn't download? I don't know, might not necessarily be the best solution.

I would love to see some refactoring done, the way Hydrus seems to be structured now, the server and GUI client seem to be very interlinked with wxWidgets, which I figure makes it hard to make the server run headless. Please note I make this observation from looking at how the client behaves towards the user, haven't had much quality intimate time with the code yet.

>>1053 This is a good idea. I will add options to change the inter-file and inter-gallery-page delay times (currently three and five seconds or something), and adding another option for a break period after n files should be doable. I could also do it based on num_requests and num_mb sent to any particular domain (i.e. after 50MB, wait twenty mins before requesting again from there). >>1054 This is in the works, and I try to do a bit more every week. The server's wx parts aren't necessary, so I plan to replace them all and move to console switch and network POST request control of the server. So far, I have refactored my general file parsing and rendering code away from the wx-entangled client-specific code and cleaned up my import tree so the server doesn't accidentally import any client code. Now I have to rewrite the wx app controller for the server and pull wx out of my pubsub and db maintenance timer code. And then add some switches to control it.

I've been loving Hydrus for awhile now and wanted to start using it to manage a video collection since folders just weren't cutting it anymore. The open externally option works great for this. Only problem is that thumbnails are only generated from the very start of the videos, not part way through. So visually all the added videos look the same; black boxes because they all fade in. Is there any way that thumbnails could be generated part way through the video's duration for mp4s or does that conflict with how the server works?

>>1070 Honestly I think you'd be better served by using software actually intended for managing video collections. Kodi, plex, emby are all good cross platform solutions to take a look at. Imo projects need a scope, and it's fine to use it however you want, but when you start asking the dev to support your unintended uses you put the project at risk for feature creep. Offset thumbnail generation isn't (afaik) a big ask, but there are already so many excellent video cataloging programs that already do what you ask- I really don't see the point in bending hydrus that way.

>>1070 I am considering converting to animated thumbnails for video because of this. Something like grabbing ten frames evenly spaced throughout the video and packaging that into a small gif or just a list of jpegs. I can then add an option for which frame people want to show by default and also do a preview ~10-frame animation when people mouse over the thumbnail, like how some video sites do.

>>1074 that sounds amazing >>1071 I actually use Plex, but that's for shows and movies where metadata scraping is possible. Not the type of video library I'm talking about.

>>1074 Weren't you going to do something similar for ugoira support? Could do the same thing for both. Also do you still plan to give us a way to save our tag import regexes into profiles? I have to copy and paste the same five regexes into the namespace import box with every bleeping folder I import.

>>1080 Both ugoira and saved tag rexes are on my to-do list, but it is taking a while. This week I have almost finished my big wxless server rewrite job, and then I want to move subscriptions and gallery downloads to the new download system, and then I should be a bit more free to add complicated new things. Saved regexes should be the simpler of those two jobs though, so I will bump it up a bit.

I want to be able to skip this and send files directly to the trash on pressing delete.

>>1096 I will add an option for that this week, and for archive/return to inbox multiple files. Thank you for the suggestion.

Is there a way to import ratings like Safe and Explicit from boorus when getting tags? That would be helpful.

>>1105 Not at the moment. Would you like to import it like a namespaced tag, like 'rating:explicit' or whatever? I can't add import to literal like/numerical ratings until I move gallery downloads and subs to the new system. Also, is there a standard for this stuff across boorus? Is it only 'safe' and 'explicit', or can it get more complicated?

>>1106 Most boorus I've used also have "questionable".

>>1106 Currently I am tagging manually with rating:safe rating:questionable and rating:explicit. These are the ratings that sankaku chan, danbooru and gelbooru all use. After some research, konachan and xbooru also use this scheme, so it appears to be a standard.

>>1108 >>1109 Thanks. I will see about adding this to the booru code as a 'rating:' namespace. This might be something that is easier to add after I convert galleries and subs to the new system. I'll think about it a bit, and have a look at the html and css.

If it's not already added: an option to bypass the recycling bin and use the old behavior of deleting from disk. I just noticed the change introduced in version 170 to use the recycling bin broke the option 'delete files after successful import' on my linux install and I can't tell if this is a bug with hydrus or if I'm being retarded. Plus not having to empty the recycling bin of imported files is convenient on both *nix and windows.

>>1125 Sure, I'll add an option. Do you have an error for the 'delete after import' stuff, or do the files just not disappear? It is working for me on Linux and Windows, even on read only files. Do you know if your Linux distro has unusual rules for what can be sent to the recycle bin? Error tracebacks should be written to the bottom of your client.log, which is in install_dir/logs.

>>1129 >Do you have an error for the 'delete after import' stuff, or do the files just not disappear? No error message appears and the files just don't disappear. As far as I can tell nothing is written to client.log about it. I've also tried launching from a terminal window to see if anything would show up there. What I've found is that trashing a file within hydrus and then using 'delete from trash now' on it does send the file to recycling bin as expected, so originally I thought it was a problem with permissions. I tried different combinations of r/w, ownership, and even trying to delete on import a file previously imported by hydrus (identical permissions as one that can be moved to recycling bin via trashing). >Do you know if your Linux distro has unusual rules for what can be sent to the recycle bin? Not that I know of. I'm running Debian stable. Since no one else is talking about this I'm guessing that things are set up wrong on my end.

I don't know why this hasn't been implemented yet. Is it possible to have an option for the client to minimize or close to the tasktray?

>>1130 I looked at the code again, and deletion errors in this case were getting silenced. I have now made them pop up and write to the log. Let me know if you get any continued trouble with this, even after you move to deleting rather than recycling. >>1135 Thank you for the suggestion. I have had trouble getting taskbar icons to work in Linux/OS X, so this might be a Windows-only option to begin with. I'll add a shortcut for it, as well.

>>1140 Fine with me if the tasktray thing only works on windows for now. Also, another suggestion if it isn't already part of the sub rewrite, would it be possible to specify arbitrary tags for each sub, instead of only being able to rip them? It's annoying to manually append them every import, especially when I forget to do so.

>>1148 Yes, adding tags to subs is planned. I'll have it work like for import folders. I can extend it to multiple tags in future, if people want that.

I'd like to know how much disc space a tag or set of tags takes up. Is there a way to do this without searching for the tags and then selecting every image? when you've got tags containing several thousand images, trying to do that can severely slowdown/freeze the client

>>1154 No quick way inside the program. You can make sure your client loads things as fast as possible by making sure your db is vacuumed (database->maintenance->vacuum) and that the hard drive hydrus is installed to is fairly fast and defragged. That said, I would like to work on speeding up media loading times. Some predicates like num_tags work way too slowly on a big db. Do you find the slow period is when the client just says "Loading…" in the status bar, or when it says Loading… x of y? Since you want something simple and low level, if you like, you can query the db directly by going to install_dir/db and opening client.db in sqlite3 or SQLiteStudio. If you have never done this sort of thing before, I recommend SQLiteStudio, as it has a friendly gui. In that, you should be able to open client.db and a SQL Editor page and run a command like this: SELECT SUM( size ) FROM files_info, ( mappings, tags USING ( tag_id ) ) USING ( hash_id ) WHERE tag = 'alphonse mucha'; That will select total size for all namespaces and all tag services. It will result in either a number, meaning total bytes, or NULL, meaning nothing was found. I can generate a more complicated query that'll do namespaces or other pseudo-predicates, if you want.

Manga downloads for Pixiv, pretty please?

>>1155 vacuuming noticeably improved the speeds to where it's bearable, thank you. Most of the loading is "loading x of y"

>>1156 I am planning to work only on the downloader and subs rewrite next week, and likely the one after as well. Once that is done, it will be much easier for me to add things to the gallery downloaders, and I will get right on this. Please keep reminding me if it looks like I have forgotten. >>1157 Great. I will keep working on speeding this up.

so ive noticed webms that have sound, lose their audio track in hydrus. If you could add sound support, thatd be awesome!

>>1172 I definitely want to support audio natively, and I have a rough plan to get there. Once I am happy with this current downloader/subs rewrite, I should be free to work on big new features like this again. I should be able to support sound for mp4s and mp3s and whatever else as well, which will open up the program to do several new things and will also need some new gui elements to handle mute and volume and so on. The audio is still in your webms, btw, I just don't render it. I use 'open externally' on those webms for now.

Might be outside the scope of hydrus, but what about some sort of positional annotation for images. Some popular boorus have this feature; it is normally used to translate an image without having to modify and typeset the it. They normally work by displaying the text when you hover your mouse over a specified area (say a speech bubble). You could set it up by having a companion file for an image with annotations, and in your chosen file format, have position, size, and content information. I.E. there’s the image file “(some SHA hash).png” with annotations. The annotations will be in “(the same SHA hash).yaml,” and hydrus will read that file when the image is previewed and render the annotations on top of the image.

I would like to be able to assign keys to zoom functions, primarily a "zoom full" and "fit to screen" switch, instead of using ctrl-mousewheel. Also, a way to choose default display behaviour for images that are larger than the screen resolution akin to the already present zoom for smaller images would be much appreciated. Something along the lines of "fit larger image to screen width" is what I am missing most

Is there any possible way to speed up the time it takes to import files into the db? Even importing 500 files can take up to 20 minutes, which is a long time considering I have a few hundred thousand files to import. Perhaps it's because I have such a large amount of tags/mappings. I almost wish I could force it to use 80% of my CPU/ram.

>>1178 I like the idea, but this is outside my scope for now. In the future, the network may run user-written app services, which would support this sort of specialised function very nicely, but for now I just don't have the time. >>1181 I have a plan to move towards complete shortcut customisation for all functions. The situation should get better as I continue my overall YAML->JSON rewrite. (some shortcuts are currently stored in YAML, and some in JSON, and some are hardcoded, and there are then two or three ways the gui catches key/mouse events, and it is generally a bit of a mess all around that needs cleaning up and harmonising). I've been thinking of extending the current 'zoom to fit canvas' option to support also saying 'unless the image is >90% normally', as pngs and other sharp images get unnecessarily blurry with the tiny zoomin. Adding 'fit to width/height' and other options is a good idea, and maybe having different options per mime as well. I will make a proper job of it and put it in my to-do, thank you. >>1188 Someone in the v175 release thread mentioned file imports being slow. I am not sure what is causing it. The file importer should attempt to run as quickly as your machine allows, and my dev machine reliably rises to ~25% CPU (using all of one of four cores) and imports about 2-3 files per second, depending on file size, but some people seem to be having some gui-related delays that I haven't pinpointed yet. I just ran a test on my laptop, and I only get 7%, and about 1 file/s, so perhaps having a tag- or file- heavy db is causing the slowdown. I am looking into it. HDD latency is also a common problem. Is your hydrus hard drive particularly fragged or full, or could there be some other disk-intensive program running while hydrus is trying to import? If you like, go help->debug->db profile mode while you are importing some files. That will dump a lot of profile text to the bottom of your log file (install_dir/logs/client.log). You can email/pastebin those profile tables to me and I can have a closer look at your situation. Unfortunately, the language I write hydrus in (Python) won't work on multiple cores simultaneously unless you bend completely over backwards, so I am limited in some CPU areas.

>>1217 >Do you think it's possible to have an option for swapping spaces with underscores instead when searching tags? Why not just put underscores in your tags?

>>1218 I didn't want to manually mess with the tags. On second thought, I realized how minor it is. Spaces are fine, Nevermind.

I'm going to throw this in here so other people can comment on it too. Auto-Loading When I type in a tag, it loads all the files related to that tag. (At the bottom left corner on the status bar)This can take several seconds depending on the amount of files loading. I was thinking instead, you can have users limit the amount of files that is loaded at a time. So for example, when I type in "character:princess peach", it'll only load the first 100 out 1,000 images. (Additionally, Allow users to set how much is loaded) Then, as I scroll down past the 100 image mark, it'll load the next 100, and so on, until it reaches the total of 1,000. Basically, it'll work exactly like Sankaku's Complex Auto-Paging system. https://chan.sankakucomplex.com/ This is extremely useful if someone were to type in a generic tag like "loli", which would cause the client to attempt to load around 120k files at once. Instead with this method, you'll get the results nearly instantly. Right now if I type in loli, it's going to take quite some time for me to even see any images show up. TL;DR, load files in small chunks instead of 1 large chunk. Tag List This may or may not need to be a feature, but I don't see a way to view all of my currently available tags? I realize this will take awhile to populate, so I figured maybe you can have an additional option to limit how many tags are displayed at once(around 20 is good), and have it so scrolling the tag list auto-generates the next 20 tags and so on.(same as method above, except with tags) Gelbooru has trending tags, which shows around 20 tags on the left side. However since we obviously can't have a trending tags, it could be randomized to give the same result.

>>1223 Here is a webm of what I mean when entering a generic tag. As you can see, it takes awhile. With my file streaming suggestion above, it'll stop loading at 100, or whatever the user sets, so that I get my results quickly. (Loading is also slightly slower due to the fact that I was currently importing files + recording)

>>1223 >>1224 Here's what I just emailed you, so people can see what I generally think: I like the idea of auto-loading. I am going to have a look at my media-generation code (the bit that does "Loading 512 of 2000") in the near future, and if I cannot speed it up sufficiently, I will look at doing something like this. Thank you for the suggestion. If you haven't seen it yet, for now you can add a "system:limit" predicate, which will limit the results to a random sample up to the given number. I use it all the time, and it does the first half of what you are talking about here. You can also pause a search by clicking the "searching immediately" button, which is helpful if you want to put in several tags without having the CPU hit of the half-completed search going on in the background while you type the subsequent tags in. There's no way to see the tag list yet. I have thought about creating a tag cloud to display a client's most popular tags, and there is a thought to add this information to the tagging dialog (e.g. anything tagged with 'series:blah' could have a tag cloud that had various common 'character:' tags for that series), but I am not sure how I would really display that data on a normal application gui.

OK, I have a reason to revisit the server again. BARE minimum change I would have you make- if the client is local to the server (eg same directory/installation) have just one shared database. Having the client and server operate on separate databases results in one of two things: the database being twice as large as it needs to be, or reduced client functionality. I have my database, I want to share it while still being able to use the client and without wasting space. I can see two solutions 1) couple the client and server into one database -or- 2) implement LAN/local specific client-server code for the client viewer. That is, keep everything exactly the same except allow the viewer to directly access the server database, as opposed to importing it to the client database and viewing it there. This could be in the form of a special kind of user, instead of writing LAN code. What do you think? My end goal is to have client functionality for my personal uses, while also serving my database.

>>1231 Thank you for the suggestion. I have thought about doing something like this before, but it is actually quite tricky, as the way the server and client store some sorts of similar information is different, and so many tables just can't be combined without hacking things into inefficient shapes. The server stores files and tag mappings with timestamp and account info for every row, for instance, while the client considers everything to be timeless and anonymous and is more concerned with keeping everything quick to search. Combining the two datastores would also need new gui and db code to replicate all their existing http communication on an intra-application level. Furthermore, the db engine I use, SQLite, is a complete pain about multi-threaded access, so trying to get simultaneous server and client access would require some locking magic that would almost certainly slow things down, especially when the server wanted to suddenly generate a big update for 45 seconds straight. Writing insta-access local http requests produces some not dissimilar problems, in that the server is designed to use very little CPU recording new content and occasionally generating update files, not in performing complicated tag queries. I could create a cache of the server's data inside the client and search that, but that's what a file repository already is. My general long-term plan is to move many of the server's responsibilities to the client, file sharing included, once I have some p2p functionality going. I don't really like how the file repo works, so it might be better for the server to be a p2p node-connection facilitator, rather than a big media server. If your concern is more in doubling file storage between the client_files and server_files directories, there is probably a good way to neatly merge them, maybe with an auxiliary db file that tracks which hashes to keep and delete between the two programs, but that makes clean backup and restore a new headache. I will think about it a bit more. I expect I am missing some obvious solution. If you have any ideas on how to cleverly do this (and it needs to be multiplatform, so we can't spam symlinks), I would be very interested.

>>1240 I'm not suggesting symlinks, but on the subject of symlinks & Windows, cygwin invents its own symlink to neatly avoid the permission issues in NTFS style symlink. Food for thought. I like the idea of moving the server's functionality into the client. The client already has the beginnings of a booru server, and imo it would make a lot of sense to a) rename client.exe to hydrus.exe, remove server.exe b) add a headless mode to hydrus.exe (hydrus.exe –daemon) P2P sounds interesting- are you thinking complete redundancy across the entire P2P pool, or…? I can't really tell you how to handle the server, since the massive scale you're designing for is not how I'm used to thinking. I'm also not used to the concept of DB locking- I remember that as a thing of the past, because so many internet services have found ways around it. I don't have personal experience here though.

>>1241 Hydrus.exe is generally where I think I am heading. I would eventually like the network to move towards something like gnutella, but working with hydrus's specifics like sha256 hashes and predicates and optimised for small files like images. I'd like for people to be able to go "show me some x" and get a page of thumbnails for the various media the other nodes have to share. Not necessarily comprehensive over the whole network, but more like "here's some x I found" like Limewire and those other clients used to do. Nodes would share their and others' file and tag caches about, perhaps with differing levels of trust, and offer proxy and signing services as well. Specific always-on supernode services would probably play a part, but only really as certificate authorities and IP caches and that sort of thing. That's a very vague description of a very complicated system, so my current practical plan has been to get some instant messaging going and start tacking on features like real-time IM file transfers. In time, I can add asynchronous (email) messaging and so on and so on. I was messing with this stuff a long time ago, but I've been distracted by the sheer mechanics of showing media in the client (which is what most people request me to work on). I have a lot of ideas and a few splatters of mostly broken code lying around. Nonetheless, I feel that the 'searching and viewing and tagging/rating' part of the client is working fairly well right now, so I am starting to think about this stuff again. I am also planning to add support for many regular client actions (import and tag files, that sort of thing) to the client's local server, through POST requests. Then people will be able to externally script mass imports and tags from various complicated locations. Formalising more client commands into proper serialisable objects is a good step towards client-to-client syncs and so on.

I'm new to hydrus and so far I'm really enjoying it. There's one thing I would really like to see and that's the implementation of sibling posts. I would like to be able to link two or more entries together and preferably have them show up next to each other. I often have multiple versions of the same image and I don't like to see them all over the place. This seems like a good solution to that problem. I could obviously just tag it with different volumes and pages but that just looks bad to me. I don't know if there is already a better way to do it but this seemed like the place to leave my idea. Thanks for your time and keep on doing what you're doing!

>>1258 I am glad you like it! One of my next big plans is to scale my 'looks very similar' duplicate detection code up to the entire database, so you'll be able to say "show me all the images that might be slightly different resolution, or have little watermarks, but are otherwise basically the same thing", and then have those results appear in a way that makes sense. I then aim to add some new concepts for the db to understand like "This file is the same as this file, but it has a slightly higher resolution" or "has fewer jpg artifacts" or "isn't cropped" or whatever that people can enter, perhaps through some filtering system. This information could be shared through a new service, and then people could clean their dbs and merge tags for dupes in an efficient way by setting up commands for each relationship type (like "when you spot dupes, I want you to delete the version that has jpg artifacts and port its tags over to the better version."). It would be easy for me to add "This is an alternate of this. The first image is the 'parent'." to that sort of system. I could add a little button to the media viewer to show the stack of alternates, or include them in normal queries, or whatever. In any case, I am not very happy with how collections work at the moment–I think I could incorporate this sort of rich metadata into better sorting and display of media. But the first step is in gathering that metadata in an efficient way. This stuff will be a lot of work that will take me a long time to do, so please follow along and let me know what you think of the components as I build them and keep me reminded if it looks like I have forgotten about siblings. Thank you for your interest and suggestion.

speaking of p2p, have you looked into ipfs integration at all? tl;dr magnet links for static website content some people over on /tech/ have been playing with it and it looks perfect for boorus

>>1270 beat me to it >>1247 yeah, ipfs >>>/tech/412592 should be the current thread they're discussing it in Seems like a perfect fit, since it takes care of the hard part by trying to be a general purpose protocol for data decentralization and distribution, and it bases everything around hashes similar to your method. It even has python api bindings: https://github.com/ipfs/python-ipfs-api My very rough idea of how it would actually be used: Transfer of all Hydrus hashes to use ipfs' multihash format ( https://github.com/jbenet/multihash ) so as to not lose the millions of existing mappings we already have (likely easier said than done, but still) and then pinning those hashes in Hydrus. For new files we could get the hash instead by simply adding them in ipfs (which automatically pins them as well in the ipfs client, not sure about the python-api) and recording the resulting hash in the database. You could even do the same for file thumbnails as well. When you want to retrieve the file, a simple "ipfs cat <file_hash>" (or however it would be written in python) would let you retrieve the file from ipfs and view it in Hydrus. If you actually did this, then it would immediately come with the added bonus of every hash Hydrus knows about can now be searched for by its tags and retrieved even if the user does not have the file locally as long as at least one other person on the entire ipfs network has that file to share and is accessible. Essentially removing the need for file repos altogether. Of course, the real implementation will be more difficult than just that, and will beg for the need of a few extra features of how the whole thing would work. You would need to be able to handle cases where a hash is requested but not actually able to be found (likely not too difficult) as well as have a proper interface for pinning, and how it works with the inbox/archive/trash paradigm you currently have (my idea would be to add and pin everything, just marking it differently like you do now, and then when you want to actually delete something you just unpin it and let ipfs garbage collecting take care of the rest). Not to mention the many subtleties of Hydrus I don't even know about that would make implementing this more difficult, I'm sure. But, again, if you actually did go about doing this, the benefits wouldn't even end there. The tag repos themselves could be decentralized despite having to be update-able (and thus changing its hash that ipfs relies upon) because of ipns. Now, ipns especially is still being worked and has its problems in its current state, but in the future should be more reliable and a practical alternative to tag repos by, again, just having a specific ipns hash that every Hydrus client is told "This is the public tag repo, download this over ipfs and use it locally" In fact, it even supports DNS, so you could even instead tell it to go to /ipns/brand-new-hydrus-website.biz and it would look at brand-new-hydrus-website.biz for a custom ipns DNS record with the correct ipns hash to then use, letting you change the hash without worry. Hell, there may be a way to optionally not use a local database at all and basically just use an updated, decentralized, shared database directly. There is already talk about how to implement and use databases on ipfs (see https://github.com/ipfs/ipfs/issues/82 ). And then there are ipfs "services" that I admittedly am not too familiar with but I'm sure are very interesting and has a plethora of possible uses with Hydrus. The most relevant to Hydrus I've seen is https://github.com/ipfs/archives/issues/8 which talks about decentralizing searches, where different nodes running this search service have only a fraction of all combined mappings but can be used together to not only decentralize the data it searches but even take part themselves and decentralize the actual process of searching. Sorry for dumping a fuckton of info and autistic ramblings in front of you, but ipfs has given me a massive boner that simply will not recede and my doctor says that talking about it will help. Hope you were able to garner something useful from it.

One of these might already be possible but im not sure, but i have two feature requests A.Dual Monitor Support, I would like hydrus to open fullscreen on the monitor I have hydrus on, not always on my main monitor, or even, a "windowed" fullscreen mode, so I can still browse through images and tag with the hotkeys, but have the images show up in a smaller window, this can even be useful to shrink the window a bit and put the tagging window to the side so they won't overlap (the "fullscreen" window would be locked in size and the image would stretch to fit) My other request, is a tag downloader, instead of downloading all the images off a booru, it would just grab tags. This would be very useful for people who are picky about their collections, but also want the tags that already exist on popular boorus. So instead of downloading several thousand images of a character, they can just set the tag downloader to go and grab the tags off the site without grabbing the image, this might also lower the resource usage on the booru in question (educated guess here)

>>1278 I was thinking that the simplest way to go about doing it would be to add a context menu option to send the file to an existing ipfs daemon and mark it as being shared within hydrus via a new archive/inbox category and maybe an icon in the thumbnail corner. This would maintain a clear division between hydrus and ipfs. Many people manage media with hydrus that they don't want made public or simply wouldn't want to participate in such a scheme for various reasons (such as bandwidth limits). ———- Lots of picture pages on deviant art include a download button to save a higher-res version. Hydrus currently saves the smaller resized picture that is displayed on the page. It'd be awesome if it were changed to get the larger picture whenever one is made available and fall back to the resized version when it isn't. ———- Lastly I'd like to bring up adding g.e-hentai and exhentai gallery support. They also offer full resolution links on some of their pages and it'd have to go slower than usual because they limit users to X pages per hour without using their credit ystem. Ex would obviously need to use login credentials. Gallery language filtering would be similar to hentai-foundry's filtering. A lot of work, but in the end it'd be well worth it imo.

>>1278 PLEASE do this. You have so much good work put in to this thing already, if it could be used for distributed serving of tagged files it could be something truly phenomenal. People wouldn't even have to upload many images to chans anymore after a while, just link to a file in the system and the copies that are already out there will be streamed. All the people who like it will just grab it through the system and help distribute it and manage the tags.

>>1278 I appreciate all this info, thank you. Now I have read a little bit about it, IPFS looks quite interesting and something I can definitely plug into. That python library looks great. I haven't used the network yet, so I do not know how stable or laggy it all is, so I will make the first step very simple and optional. I have created a new job to basically do what >>1280 suggests–I will add an IPFS option page where you can point to your local IPFS daemon, and add an 'add this to IPFS' option to the right click menu for local files. I'll change my download code to allow downloads from unknown locations and let the client search IPFS for any outstanding downloads, essentially like a different kind of file repo. If you happen to know, what's the ipfs cat/get failure state? If the file doesn't exist on the network or just can't be found in say 30 seconds, does it dump out with an error? >>1282 A distributed chan is what hydrus was originally intended to be, funnily enough. I wanted to build a localhost service that would be a p2p node on the internet side, a bit like IPFS, that you would connect to in your browser, and would fetch distributed threads and images from via AJAX. All built through a LAMP stack, originally, until I stumbled over python and the pursuit of "Actually, getting images is fairly easy, let's try to burn some heavy clientside CPU searching and organising them" moved it all towards ACDSee for Anon. >>1279 For multiple monitors, try pressing 'f', or hit the 'fullscreen switch' (tooltip) button in the top hover window of the media viewer, which should convert it to a normal window. The button looks like a couple of windows overlapping. The media viewer should remember its position and maximised/borderless state, but let me know if it doesn't. I would love to build an efficient tag downloader for boorus, but I would need them to supply an API call for me to do it. Their API would have to accept file hashes, either MD5 or SHA-whatever, and then return tags. I could then say to the booru "Hey, you got any tags for [file]?" and they would say "Yeah, [tags]." Or they would need to support something like muh_booru.net/post/[file_hash] in addition to their existing muh_booru.net/post/internal_booru_post_id systems they use. Unfortunately, I currently have to download the file if I want any tags, as I need the file hash to associate those tags with, and the file hash can only be generated by having the file itself. There are ways I can be clever, like if you have downloaded the file before, the client will remember that and will be able to reproduce the hash without having to download the file (that's what "get tags even if file is already in db" is about). Again, if the boorus wanted to provide some xml/json/whatever API that was something like "Hey, booru, can you give me a hash->tags dictionary for this query?" I could do it and cross-reference with the files already in your db, but the API that danbooru and some others offer doesn't give hashes and at the API level doesn't namespace the tags. At the moment, my parser walks through every gallery page, picking up image pages, and then goes through each image page in turn, harvesting tag lists and image links. 4chan and 8chan bundle the MD5 hash in their API and it is great. I can check for redundant and deleted files without having to download, but boorus do not do it. Sorry, I feel I didn't explain that very well. Let me know if you still have questions. TL;DR is that it is currently technically impossible to just download tags from a booru and insert them into the db. They need to offer more information on their end. >>1280 Thank you for the note about the deviant art button. I though the click-zoom (which loads an image with the deceptive classname 'dev-content-full') was the same resolution, but it isn't! It looks like there might be some referral or cookie hoop I have to jump to get the button's link to load, but I will see what I can figure out. I don't know much about g.e-hentai and exhentai. Isn't one like a login for the other, and one hosts loli or something? I am all for writing a plugin, but I want to do per-site bandwidth management in my next network engine iteration, so if there are complicated bandwidth limits then maybe I should put it off until then?

>>1288 >try pressing f HOLY SHIT HOW DID I NOT REALIZE THIS WAS HERE Also thank you, I understand what you mean about the tag downloader, shame about that.

>>1288 8chan can pull md5 without downloading an image with the api? This is where you might lose me in technicality, but, is it possible to use this to pull md5 hashes from the site without downloading something to your hard drive?

>>1290 Yup, it's actually not complicated at all. In fact, you can go to the json endpoint of this very thread just by replacing the .html in the url with .json: https://8ch.net/hydrus/res/471.json . Just do a Ctrl+F for "md5" and you can see what he's talking about. >>1288 I was actually interested in the ipfs python api myself and decided to look into it. For testing I used the hash QmPZ9gcCEpqKTo6aq61g2nXGUhM4iCL3ewB6LDXZCtioEB as that is the standard welcome document that comes with every installation of ipfs and is unlikely to change. Don't quote me on that, though, it might not work in the future. It looks like a call to api.cat() will return the contents of whatever hash you want, and any errors will raise a general ipfsApi.exceptions.ipfsApiError with the message describing what actually went wrong. An invalid hash, for example, with have the message "invalid ipfs ref path", and a timeout should have the message "context deadline exceeded". I'm not sure what the default timeout for it is, but you can set your own by sending {'timeout': '<time>'} as the "opts" parameter in the call to api.cat(), which is the equivalent to "ipfs cat <hash> –timeout <time>" in the go client. <time> has to be both a number and what unit your using ("s" for seconds, "ms" for microseconds, etc.) This is what I used to catch different types of errors. There are probably better ways to do it but it works for now and should answer your question: import ipfsApi
api = ipfsApi.Client('localhost', 5001)

def retrieve_file(hash):
try:
return api.cat(hash, opts={'timeout': '30s'})
except ipfsApi.exceptions.ipfsApiError as e:
msg = str(e)
if msg == 'invalid ipfs ref path': return 1
elif msg == 'context deadline exceeded': return 2
else: return 3

contents = retrieve_file('QmPZ9gcCEpqKTo6aq61g2nXGUhM4iCL3ewB6LDXZCtioEB')
print(contents) Hope that helps! Also, thanks for fucking everything man. You really made my day.

>>1291 *milliseconds Basically the same, right :^)

Oh man oh man if IPFS support gets added to this I'm gonna freak out (in a good way). https://blog.neocities.org/its-time-for-the-permanent-web.html

>>1290 As >>1291 says, I need an API that'll give me the MD5. I just looked at Danbooru's and they actually do offer that in their json API, like this: http://danbooru.donmai.us/posts.json?tags=blue_eyes I guess I forgot they did, because I do remember looking at that before, or that is a new addition. Moebooru boorus offer something similar, but looking at this: http://konachan.com/post.xml?tags=kantai_collection I don't see any hash info. EDIT: Wait, it does, I was searching the wrong thing. Nonethless, in both, the tag list is not namespaced, so in order to get better tags, I would have to pursue the image page. That's not bad (although I don't really want to make a habit of hammering them with 20-100KB page requests just to harvest namespaced tags), but since the API request calls and responses are different across boorus (and some are unreliable, like http://gelbooru.com/index.php?page=dapi&s=post&q=index which currently gives me an error about overuse), I have shied from extending the already-messy services->manage boorus to support different APIs. Perhaps that is a long term project I can do, or it can wait for when downloaders are more complicated plug-ins that users can write themselves, which is a long-term goal. >>1291 Thanks, this info is great. I will have a long recheck period (something like four hours, I guess) and bottleneck the number of files to attempt so hydrus doesn't end up hammering people's IPFS daemons every twenty minutes with 500 spurious hash requests (each of which I presume sends out some request to other nodes across the network), and also to abandon the download if a file can't be found after x requests.

>>1291 by any chance, could a bit of js be written to display md5 hashes next to images as one browses 8ch? Sadly I don't know js and am completely incapable of writing something like that myself, but it could turn out to be incredibly useful if I had something like that

>>1304 I assume Greasemonkey could do something like that, but I don't know anything about coding for it, so I'm afraid I can't personally help.

>>1304 https://greasyfork.org/en/scripts/13395-8chan-md5/code Have fun! If you don't know what a userscript is just use the help on that site. On some posts it doesn't work because they're so old they don't have a recorded md5 on 8chan, at least it appears that way. For some reason the md5 is embedded as an attribute on all 8chan img tags, but not exposed as text to the user? So this was super easy, almost a one-liner with jQuery.

Is it possible to have files which are rated, identified by a color? This would let me see what files I have rated without running a system:rating search/sort.

>>1313 I like this idea and the mockups you have made a lot. I think I will generalise it so users can set their own predicates, like: file is rated in some way on some/all rating services file has some number of tags on some/all tag services And then doing the colouring with semi-transparent tints to mix with the otherwise default colours for any of: thumb border thumb background the namespace text at the top your overlay example. Thank you. I have created a job and a plan for this. It might be a little complicated, so I will first create something simple, maybe just starting with the overlays and specific ratings. I'll be doing a bit of thumbnail code cleaning soon, so that would be good time to start this.

>>1317 Glad to see you're looking into it. thumb border colors is good enough for me, or whichever operation consumes the least amount of memory.

I don't know if it's a feature or a bug, but when introducing tags I can't use neither the button END nor HOME, or use the CTRL or SHIFT modifier to select or move around. If I mispelled something (especially when starting a new database) it can be a little bit annoying to either switch to mouse or mash the backspace key.

>>1312 Thats awesome! Just added question though, if I want to make the md5 appear a line down, where would I put \n, OR is there a way it could a little [-]/[+] be made to hide/display it? Ive already got a save as original filename script lengthening that row of info, so now its REALLY long lol Great work btw

>>1329 I added a break before the MD5 (html uses <br/> for newlines), hopefully that unclutters things for now. I'll add a +/- thing in a little bit.

>>1334 Saw the updates go through, looks awesome! thought it was js, my bad, didn't realize it was html

>>1322 This is difficult because if you type something in, the autocomplete dropdown (the list where all the tags appear) needs to grab the vertical navigation keys (page up/down, home/end, arrow up/down) before they can propagate to the textbox so you can navigate the list of results. Arrow left/right should work, and with ctrl and shift. Does ctrl+arrow left work for you? I could stop the list from grabbing the key events when there is only one result, which would often happen with a bad spelling, but I am not sure if that would be jarring/confusing. I'll change it for next week, and we'll see if it is helpful.

You know what, don't change anything, it was a stupid suggestion. I am used to using home/end, that's all. I can live with pressing a few times the left button. I really didn't think the whole thing through.

>>1268 For the similar detection code, maybe you could use/look up the code of imagemagick. I believe the people that made VisiPics used it and it was really powerful.

Hey dev, I always find it incredibly time consuming that I have to exit the tagging dialogue when using it with the full screen viewer, to archive or rate files. Any chance you could include the rating option and an archive button, or enable the shortcut to work when the tag dialogue box is open? I usually skip ratings all together because of this, and id like to use them more than I do. I usually have to close the tag dialogue and the media viewer then highlight everything ive been tagging to archive, and I get crashes every once in a while, and I usually set my inbox to random order, so I often end up not being able to hit archive on a bunch of files that are now tagged. The crashes are to be expected on my end, my hydrus library has now passed 100,000 images, i had to move it off my ssd because its too big, it gets a bit buggy now, especially when tag selecting ill have to wait a moment. But really at that size, again, expected

Hey, just started looking into this program, and it seems amazing so far, but I've got two suggestions I'd thought I'd post before moving my collection into it. First would be a source field. Keeping the source of an image has always been important to me, and I've found ways of saving the source with pictures via file names and folders since I started my collection. It hasn't been the easiest or cleanest, but it worked. However, Hydrus copies imported images with different file names to it's own directory, so I'd lose all the source information I saved with the images. It'd be awesome if you could add a source field into Hydrus somewhere for images, possibly multiple for artists who post their pictures across multiple websites, so I can import and organize my collection in Hydrus without losing the source of images. Second is more of a personal preference, but I'd suggest adding an option to change how zooming in on a picture looks. Zooming in on Hydrus makes the image become all fuzzy, which makes pixel art look poor, jpg compression a bit harder to spot, and makes the image look poor. I could just open images externally to image preview, but I thought I'd suggest the idea anyway. Sorry if either of these have been suggested before or are already planned, but thanks if you consider looking into them.

>>1384 Why don't you just use a tag with a "source:" namespace?

This has probably been asked a few times already, and there's probably a reason for this, but Why doesn't Hydrus have this?

>>1388 Didn't think of that, I guess it'd work. Still wouldn't be a nice as a proper source field, but putting it as a tag probably wouldn't be any worse than putting it in the file name.

System tag for if image/gif is transparent?

>>1350 The tricky bit is the search algorithm. There are several ways to compare two files, but comparing n files in anything less than n-squared time is tricky. (I can't remember the exact math, but comparing 100,000 files by doing them all in turn would take something like 5-10 billion comparisons!) So instead, you basically have to store the chunks of data that represents your images (I use a perceptual hash based off of a DCT of the thumbnail) in something like a VP-Tree that allows for quick hamming distance comparisons, and then encode that data structure in a database and then write some clever insert/remove/search functions to maintain and use it. At the moment, I store the phash and can do "Is this single file like any of these 100,000 files" in 100,000 inefficient db hits, which usually takes a second or more. Getting this to be n log n or whatever is all doable, but I just need to find the hours to point my brain at the problem.

>>1383 I had a plan to copy everything in the top hover window (the one with archive and prev/next buttons) to the tag dialog, and then it got overtaken by other things. I have put it up to top priority. ACTUALLY: Maybe I should instead move the tags dialog into the media viewer? If a click of the tags hover on the left were to 'pop-out' a fully-fledged manage tags panel, maybe that would be better. Or keep it launching a window, but make the manage tags dialog be a frame instead, that isn't on top? So you can drag it about but still interact with other things. I can make it a frame pretty easy, so I think I will experiment with that. I am not happy with the general multi-filtering-workflow, and I think the clunkiness of dialogs is a part of that. >>1384 For the first thing, as >>1388 suggests, I think tags will generally do what you want. If I add a box for source or any other specific variable, then I have to add ways of adding and removing and searching and efficiently harvesting that info, when from a data standpoint, it is still a bit of text. If your important information is stored in filename, and if it is stored in common patterns, you can capture that information very quickly and powerfully with regexes. If you haven't heard of them, they are a great but 200% autistic tool that can convert one bit of text to another. When you are about to import some files, hit add tags based on filename. Hit add on the quick namespaces panel and do something like source : (?<=\\)[\w\s]*?(?=\..*$) There are also ways to get directory and parts of a filename and so on and so on. When you hit ok on that sub-dialog, you will see a preview, so you can go back and edit the regex if it doesn't do exactly what you want. If you already know regexes, then you are fine, but I am also happy to help, and other people on the board know much more than me. I don't really like them, but I don't know of a better tool. Also, if you download from hentai foundry or whatever inside hydrus, you can set source:hentai foundry as an explicit tag for everything you download in the tag import options panel. Let me know if you would like any help using this. For the second, I agree. I want to add some simple 'zoom to fit?' options for each mime, but I think I should allow people to choose which resizing algorithms are used as well. There are some bugs in the zoom code too.

>>1389 I haven't written it yet! Several controls in the program (like the thumbnail viewer) hold too many objects for OSes to handle with normal gui elements, so I had to rewrite them all with custom code. As a result, I have to rewrite basic stuff like shift-selection and so on from scratch. Funnily enough, I don't think anyone else has suggested drag select yet, although now I think of it, I have wanted it myself a couple of times. It shouldn't be too tricky to add. I will put it on my to-do list, thank you. >>1391 It looks alright for me. Which OS are you using? What colour do your system predicates usually have? Is the text not showing but the predicate still has background a colour when you select it, or have I misunderstood your problem? Can you post a screenshot?

>>1400 I don't know anything about regexes, but I know you touched them briefly on on the help part of your website and linked out to a full guide, guess I'll have to look into them. Just to clarify what I mean though, I'm not talking about adding "source:deviantart" or "source:paheal", I'm talking about saving the actual source URL of the image. So what I meant by source field was a separate thing from tags, where a clickable link would be placed, like on paheal or gelbooru. That'd also mean you probably wouldn't need to worry about adding in ways to search for or by sources. Although, it sounds like I'm more or less the first person to suggest a source field, so I guess there really isn't any demand for the feature. If there isn't enough demand to justify the effort of adding it, that's fine. I managed to found a way to put the source URL into file names, I can probably make it work with a tag. It just would have been nicer to have a proper source field with a clickable link instead.

>>1401 I think >>1391 is suggesting that you add a global system tag for when images have transparency, not complaining that system tags for him are transparent or something. I actually thought the same thing and had to re-read his post, though, so I blame his wording.

You talked about offsetting thumbnail generation "so every video doesn't just have a black thumbnail". Why not take it a step further and generate multiple thumbnails per video? See picture.

>>1402 I see, that's an interesting idea. Anything you download from a website inside hydrus actually has its 'page' url saved to the db (so the client can know to skip a download if that url turns up in a subsequent gallery search), so this particular information is already supported somewhat. It would not be too hard to query this cache and expose it to the gui in a 'known urls/I have got this from' list or something from a thumb's right-click menu. This information could also be shared amongst users, and that might be very much a useful thing to do. Let me think about it a bit, and then I'll prototype something. >>1404 >>1391 Thank you, I read it wrong. Transparency info is something I don't currently cache in the db, so I can't search it. Once I add efficient dupe searching, I will likely be adding additional 'this image is like' information, like 'redness' and 'lots of curves' and 'sharpness'. This system could support 'has transparency in its palette' very well, so please remind me when we get to that point. >>1405 This is the plan. I will move to all animations having gif thumbnails with up to say ten frames, and then you'll be able to set a custom offset for the 'display' frame. Also, I'd like to have these thumbs do their rough animation when you mouse over them.

>>1400 Being able to select either/or would be useful. I typically seperate them now, usually the image viewer in the upper left, tag dialogue on the lower left, and some anime on the right side of my screen. I like having it as a seperate window, but being able to select both, or adding the clickable things in the image viewer would be great options, or what you said. Also as is now, the only thing that changes size is the list of current tags, you should allow the spacing on the tagging dialogue box to be changed for the other windows, so the list of tags based on what you are typing, and the window on the left for which tag repo you are using can be shrunken as well

Is there a way to add explicit tags to files being imported?(not downloaded/subscriptions) I only see add tags based on filename

>>1440 Make sure you have version >=180

>>1444 Or here (i don't quite understand what you mean as both options are obvious for me)

>>1445 Thanks. Didn't realize it had more options in it. I never clicked the "Add tags based on filename" because I assumed it would automatically start trying to add tags from the filename.

Suggestion: Hotkey for tag autocomplete Basically, I want to be able to hit a hotkey and have my tags auto-complete instead of waiting on a character threshold or long/short wait timer.(Manual auto-complete?) For example, if I type 'creator:princess', it will populate quickly. If I had it set to a long wait, then it'd be needlessly waiting on something that can be processed quickly. if I were to accidently type 'creator:' though, i'd be waiting around 20 seconds for it to finish while my gui is frozen. With a hotkey, I can type 'creator:princess', press 'Tab', and it'll automatically start the auto-complete. This way, I can control exactly when it'll search for tags. Of course, you can also allow a hotkey in conjunction with wait timers too.

Would it be possible for the tumblr downloader to be able to pull from the source link of a post if it's an appropriate mime-type and exists? I'm not sure if the behavior should to be to download the source instead of the post (because there can be multiple images and only one source) or just download everything, source & post, perhaps an option? Here's an example blog that uses the source to link to higher res versions of the posted images (as it seems tumblr limits resolution on uploads): http://t-hoodie.tumblr.com/

So today I realized that if a file is tagged with both 'pov' and 'looking at viewer', it should obviously be tagged with 'pov eye contact'. I made a new tag parent that works the other way around, but going from two tags to one new isn't supported. I know that the pov eye contact tag seems a bit useless if you can just search for pov and looking at viewer, but it just feels like it could possibly be a useful feature of some sort.

>>1456 This is a great and simple idea that I can add quickly–thank you. I will add an option for it, and hardcode tab as the shortcut. >>1461 I really like the idea (I didn't know tumblr had that source thing–I assume it appears if you embed an image from a url or something?), but I am not sure how to harvest that url information. I use the old tumblr api to generate links, like so: http://t-hoodie.tumblr.com/api/read/json?start=0&num=50 I can't see that 'source' information or the puu.sh links in there. I looked around and found a per-post api, like this: http://t-hoodie.tumblr.com/post/133375777797/json But I can't see it there either. I could write a new parser to parse each post for a 'meta-item source-link' class tag and harvest the href from that, but I won't have time to do that for a little while. Do you happen to know of another API I can plug into, or anywhere else I can get that puu.sh link? Or even a way to get an rss or json or whatever of the artist's puu.sh links directly?

>>1475 I love the thought of adding these complicated sort of rules for parents and siblings, but it will have to be something for the future. For the moment, please add those sorts of tags manually.

>>1479 The source link is manually input on each post, I'm not sure how widespread it's use is on tumblr. I had assumed you were scraping tumblr pages like booru pages and could have just scraped the <a> tag, which thinking about that now doesn't seem like the best solution as tumblr blogs are theme-able and the source link class could differ per blog. Looks like the v2 api exposes the source url. https://api.tumblr.com/v2/blog/t-hoodie.tumblr.com/posts?api_key=fuiKNFp9vQFvjLNvx4sUwti4Yb5yGutBN4Xh10LXZhhRKjWlV4 (I don't know who's api key that is. It shows up in the api docs https://www.tumblr.com/docs/en/api/v2#posts) Also, there's apparently an official client library for their api https://github.com/tumblr/pytumblr not sure how useful that is to you, also not sure how viable it is using the v2 api, as hydrus would need a consumer key for tumblr, perhaps have the user request the key and input it to hydrus?

>>1484 Thank you for this information! I think this is ultimately something I will leave to users to create once my downloader engine allows people to create their own gallery parsers. I don't want to mess around with non-open APIs if I can help it. For now, I suggest you make a job to check that tumblr every x days and manage the download and import cycle yourself. I do this for several webcomics and artists I follow that I would never have time to write individual parsers for, batching up a month's worth or so and then importing and tagging them in an efficient group.

>>1489 Alright, thanks for the advice. I'll probably end up writing that parser when hydrus does come to support that.

When a subscription fails to download a file, it does not give any notifications about it. I only found out about it when I realized I didn't have a new image from one of my subscriptions. This also ties in to another idea I had where you could scan your subscriptions and get, for example, subs that haven't gotten any new images in a long while, or ones where you've archived the least amount of images… But for now, maybe some way to know that your subscriptions have failed files?

>>1505 Thank you for your thoughts. I think I want a button somewhere that says 'retry all failed subs files' or something, and for per-sub as well, and maybe a way of reviewing this info en masse. Maybe I could stick a generic 'my subs' service on the review services window, under local? Or I could put some extra info in the list of subs in the manage subs dialog? Maybe colour 'inactive' and failing subs' names a different colour? Or have a checkbox to filter them? I can easily add some text to the manage subs dialog's subs panels, like '3 failed files', rather than hiding it all behind the import status button, so I'll start with that.

>>1508 I think some sort of generic "subscription management" window could be helpful. Considering subscriptions is the feature I use the most in Hydrus, it feels to me that being able to see various subscription stats and maybe even filter/order them (last update, last update with a new file, last update with a new currently archived file, failed files etc) would be very convenient. I do wonder, however, how slow such filtering would be.

>>1516 +1 to this. I also use subscriptions a ton. Are you going to allow update intervals to be lower than 1 day? I have a high traffic sub, and being able to set it to every 3-5 hours would be nice.

>>1521 Sure, I will create a new control to allow hours as well as days.

Minor change. The selection tags context menu should have "include this tag in current search" and "exclude this tag from current search" options. I know double clicking the tag will add it to the current search but I haven't found a way to exclude a tag using only the mouse.

hydrus newbie here: love the program! I would love it even more with two usability improvements: - a way to favorite tags. (where do you keep your lists of favorite artists?) - a way to drill up/down while browsing pictures. example: I look at pictures with tag a fullscreen. a certain picture looks really cool. I press key b. The program automatically switches to showing pictures with Artist/series tag of cool picture while downloading more of said tag. I press key c and resume looking tag a pictures. thanks for writing this program, keep up the good work.

>>1539 This is a great idea, thank you. I have written some intelligent menu items for selection tags and the active predicates boxes for next week–depending on whether the tag or -tag (for any of the currently selected tags) exists in the current search, the menu will provide up to four options for remove/require/permit/exclude. It seems to work well! Let me know if you have any problems with it in v185 >>1541 These are both great ideas. There is a thought to add a tag cloud or something in the future to quickly display and search popular tags–it would be easy to add favourites to that. For jumping between browsing and searching, for now you might try middle-clicking the the 'creator:blah' tag, which should open up a search page for that tag no matter where you are. You can then alt tab back to your media viewer window when you want. You can also right-click on a tag for a menu to quickly copy it to the clipboard, which might help quickly downloading more from a gallery.

I've had hydrus for over the weekend, and here some features and things I thought of. I wouldn't be surprised if a handful are already implemented. page tags in order hide page tags unless looking at only one image when in image view, just start typing to add a tag (can also ctrl+v tag) option for tagging order of images when importing, (or regex shortcut for this) booru-on-rails support would be nice support for groups of images on boorus (like pools or collections) option to remove dash between page chapter volume etc marks button to open externally when memoryerror occurs ability to add custom thumbnails for flash files option for regex to only apply to file name, and not full path can't exit view except for force quitting when fullscreen viewing a flash that is the same resolution/ratio of monitor

It'd be really cool if there was a way to mass add subscriptions. I bookmark all the artists I'd like to collect the art of and just recently wrote a script that turns the bookmarks into a list of tags categorised by service. It's A LOT of stuff to enter by hand and I'd much rather have a way of importing them automatically, whether it be through some custom import format or a direct python api.

>>1573 Thank you for this list. You can sort by volume/chapter/page namespace (not very well, but it generally works) under the sort dropdown on the top left of most pages. Also, you might be able to tag by import order like you want if you click 'add tags based on filename' instead of 'import now' after you have some pending imports listed. The little bit labelled '#', middle bottom, tags based on the # column in that dialog's list. You can set an offset and a step and a namespace, which should let you do quick page tags if that information isn't easily parsable from the filename. I force flash files to be slightly smaller than their media viewer frame, so you should have a slim gap on every edge to quit or bring the hover windows to the front if you are in fullscreen. Let me know if that doesn't work for you. The other things I have added to my to-do list. Some I already hope to get to as part of larger features. >>1574 Better mass-handling of subs is a planned feature that'll be on the soon-to-come voting list. Doing mass-adding is a great idea to add to it, thank you.

I'd like a way to know when I'm importing files when they are already in the db/deleted so I can quickly identify the image, like an icon next to the others. This would enable me to check better why I haven't deleted the original yet.

>>1598 Thank you, that's a great idea. Icons would be good, but I might be able to add this as part of another system I am planning, like the custom thumbnail border thing (i.e. you could set: "if file is less than an hour old, give it orange border"). I will think about it a bit. Until I can get something done on this, you could try sorting by oldest/newest first, which will put all the new files at the bottom or top.

We already have the '-' sign to exclude a tag from search results, how about adding the '+' sign to include a tag's results even if it doesn't match what's already there? Basically an "or" search.

>>1607 Now I think of it, adding OR searching logic was something on the big to-do list. I will put it on the poll for next week. Using + is a good idea, but I am not sure how you would differentiate which parts of the query you wanted to OR with and which you wanted to AND. I think my original idea for that was to say that shift+enter would start a predicate OR chain and normal enter would end one, so you could create complicated queries like: system:size>1MB character:shinji character:asuka OR character:rei OR character:mana Which would give you large images with shinji and one or more of the girls. Funnily enough, the db search code for this is fairly easy–it is getting the gui workflow and feedback right that seems to be the impediment.

I'm not sure if these have already been requested before, but here goes. How about an option to hide the tab bar if it's < 2 tabs? Would save on some screen real estate And one more, for regex favourites - When adding tags based on filenames you can only save favorite expressions, but not a namespace for them. Making the namespace default to the name you assigned the favorite would save a lot of typing if you like me keep re-using the same combination

I'd love if we could add a permanent tag to different sources. For example; Idol Complex, I'd love that every file downloaded from there would have a tag "3dpg" Alternatively, we could do save tags from what booru files are from when downloading (#130) and then it would be up to the user to parent those tags to other tags, like 3dpg.

>>1689 by save tags from what booru files are from when downloading (#130) I mean https://github.com/hydrusnetwork/hydrus/issues/130

Is there an IRC or something available for hydrus?

>>1678 I am not sure I can do your first suggestion easily–I use a wx.Notebook for the main page display, and I don't think it supports hiding the tab area. I would probably have to write my own custom notebook control to do that. But I may have to end up doing that anyway when I eventually go for tab dragging/reordering, so perhaps I can add tab hiding once I have that done. Your second suggestion is planned. I ran into the exact same annoyance myself yesterday. I will have separate regex favourites for the namespaced and the unnamespaced controls, and make it so if you select a favourite for the namespaced area, it pre-fils the text fields rather than copying to clipboard. Perhaps I will do it with dropdown choices. Anyway, thank you, I have it planned. >>1689 >>1690 You should be able to do this with file-options->default tag import options and then setting an explicit tag for a particular site. e.g. you could add 'site:hentai foundry' to the hentai foundry default import tag, and then anything you subsequently set to download from hf, or any new subscriptions you created for it, would have that explicit tag pre-set. I think the default tag import options extends to individual boorus, so you should be able to set '3dpg' for Idol Complex. >>1691 I haven't set one up, so I don't think so. I'm afraid I don't really do chatrooms/IM.

>>1696 >>1689 (me) >>1690 (me) Thank you very much.

Not necessarily a suggestion, but what do you think of https://nik.bot.nu ? It might interest you.

Is there a way to add new namespaces to the collection dropdown? I'd like to be able to collect by character or artist, for instance. Also, I've found there's a few inconsistencies in namespace tagging. Some boorus use "artist", others use "creator". Sometimes pokemon are tagged by "character", and other times by "species", and so on. Should I start adding in tag siblings to resolve these? Or could something like namespace siblings be possible? Although that's probably too dangerous and of generally limited use, now that I think about it. artist -> creator or vice versa seems like a pretty safe bet, but trying to do character -> species for pokemon only is bound to cause trouble doing it any other way than tag siblings. What about a 'soft' association of namespaces that's set by clients and doesn't actually touch the tag repository? Like, I could set my client so that if I search for 'creator:<tag>', then it searches for 'creator:<tag>' OR 'artist:<tag>', depending on how you decide to get OR predicates working. Right now you can just search <tag> to get all the namespaces, but that doesn't work if you right click to open a new search page with a tag.

>>1709 It does, thank you! I had never heard of it before. I love the aesthetics and animations of the ui, and I particularly like the five colours it puts beside the image in the detailed view. Hydrus might have looked a bit like this once, but I just found that browsers didn't give me the kind of large-data power I wanted. >>1710 You can change the collect namespaces, but be warned that it is very ugly. Go file->options->sort/collect. The collect by namespaces are derived from the sort by schemes, which default to 'series-creator-title-volume-chapter-page' and 'creator-series-title-volume-chapter-page'. To add a new sort entry, enter the namespaces in the text box at the bottom, ordered by sort precedence and separated by hyphen, and then press enter. Double-click an existing scheme to delete it. You might want to just add 'artist' to begin with. Then any new pages should show that new scheme in its sort by dropdown and any new namespaces in the collect by dropdown. I apologise that this ui is so ugly and unexplained–I threw it together after I first got collect working, and never got around to upgrading it from its current 'debug' level. Yes, please feel free to add artist->creator siblings to my PTR. I like the species namespace for pokemon and digimon and so on as well. Don't bother to give a long reason–I'll approve anything obvious. I do plan to support local namespace siblings, probably through an expansion of the current tag censorship system. I might also add that meta-sibling relationship to be sharable through repositories. I appreciate that some people would always want to see 'artist' instead of 'creator' or 'copyright' instead of 'series', or would want to unify semantically similar namespaces from different tag repos that use different standards, or even to translate these terms to different languages.

>>1712 Ah, thank you. I saw the sort/collect page, but I just assumed the text box was for sorting only. I'll go ahead and start adding in siblings.

I probably just didn't find it but is there a "always maximize pictures to fill entire screen" toggle? Because I really want it.

>>1720 I think images automatically fit to height. You can enter fullscreen using F11 and zoom in and out using + and - .

>>1722 yes large pictures get downsized, but I also want to automatically enlarge small pictures. the zoom button does this manually but I want to set this as the default option.

>>1720 >>1722 Try file->options->media->zoom smaller media to fit media canvas. I'd like to make this more complicated in future, as well, like for saying 'if a png is > 100% and less than 120% zoom at fitted size, just show it at 100% so its pixels stay sharp'. The default shortcut for making the media viewer fullscreen borderless is 'f'.

>>1735 thanks.

Yo! we have ffmpeg a part of the package Can we get a method to convert a video into another format (mp4->webm) on/before import for storage

>>1844 That's a good idea, thank you. I like the idea of conversion on import and also from the right-click menu. I would probably add a warning for any conversion to webm though, as the codec introduces some random data that means every generated webm has slightly different content (and thus file hash), which makes it an extreme pain in the ass to match up differently generated webms across clients. For instance, if I go: ffmpeg -i a.gif a.mp4 ffmpeg -i a.gif b.mp4 The two mp4 files will be exactly the same byte-for-byte, but: ffmpeg -i a.gif a.webm ffmpeg -i a.gif b.webm Creates two slightly different files. Importing a.mp4 and b.mp4 would give '1 successful, 1 already in db' and one thumbanail, but the webms would give '2 successful' and two thumbnails. This is fine right now, where webm generation remains rare and the same original webm file is usually shared across imageboards so everyone imports and tags the same file, but if I automate mass webm production via conversion, we'll just be massively diluting our hashes. Out of interest, why are you thinking of collapsing everything to webm specifically? Or did you just pick mp4->webm as a random example?

>>1848 webm is made with web content in mind, it's much smaller than gif/mp4 files when done right (unless it's a tiny gif or something) Imageboard is a plus, as webm has been adopted into it's culture, but more importantly it's easier for me to share/upload due to the compression(??) in the format I would suspect this 'random' data comes from it's unique ID, something I didn't know about five minutes ago

>>1848 VP9 encoding for webm sends different temporal samples to different processors during encoding. The processor used varies depending on system load, etc. Since different processors may do their work at different times, each has a slightly different view of the previous frames depending on how many other processors have finished their motion search analysis, and that view changes between encodings depending on which processors get done in what order. vpxenc has a –debug switch that forces slow and inefficient serialized analysis for deterministic output, but I have no idea if that switch found its way into ffmpeg's encoder.

Ugh, why dont use already existed webm converters? Image organizer and video converters are two different type of programms. Combine them is a bad idea. And why on the earth someone wants a lossy-lossy conversion (gif to webm)?

>>1868 >Ugh, why dont use already existed webm converters? I agree that scopes are good, and feature creep is bad, and Hydrus dev seems to be almost too accommodating with accepting feature requests… But ffmpeg is an existing converter already included in Hydrus. The functionality is already there, he just needs to add it to the gui. That being said, yeah, ew lossy-lossy conversions. The requester's intended use case is to share files more easily- perhaps this feature would be better implemented in hydrus' export dialog. Present it like foobar2000 does: Which exe are you using? What arguments are you passing to the exe? Save this for future use?

>>1869 Pictures of what foobar2000's export/conversion interface looks like. One is a dialog of different profiles, another is the dialog to edit a profile. Hydrus could have something similar under the existing export interface. >>1848 >we'll just be massively diluting our hashes I really think conversion on import is a _bad bad bad bad bad bad bad_ idea because of this. Sure, give people the option to convert on export, but if a user wants to ruin the integrity of what goes into their database, they should have to work hard to shoot themselves in the foot like that. My rationale: -Converting on export is a per-image, one-off event that you do when the content you want to share is incompatible with the destination. -Converting on import is, if available, going to be (ab)used for everything. Like I said, if the user really wants to do that, they can do it themselves, don't help them. >>1856 >webm is made with web content in mind, it's much smaller than gif/mp4 files when done right Mp4 can be just as small. webm/mp4 are just containers, and containers don't say anything about what is actually inside. While webm is typically pretty locked down, an mp4 can range from lossless video to webm equivalent. Twitter actually uses mp4 instead of webm.

While I'm here, I have two questions: Why does hydrus_dev not pay attention to github issues as much as this board? Actually it looks like you've just abandoned github's issue tracker. I get that you're catering to anonymous and github requires an account to post…. but bug reports and feature requests are a hell of a lot easier to navigate on github than on here. Is there any special dependence on the version of ffmpeg packaged with hydrus, or would it be fine to delete hydrus' included ffmpeg if we already have the latest version installed? (installation = on PATH). Package managers can include ffmpeg as a dependency.

Is there anything I can do on my side to speed up file import? IE, having more cpu cores/overclocked cpu/more ram/defragging the drive/etc, or is the main bottleneck just the code?

>>1874 Keeping your drive defragged is generally a good idea anyway. Besides that, the only other thing I can think of that might improve speeds is to get a faster hard drive.

Groups for services(like/ratings)/subscriptions. This is more of an organization suggestion. I'd like to keep my subs/ratings contained in a group. For example, pixiv subs will belong in the [Pixiv] group. Since it's a list, I figure it'd be easy to make it a tree node.(Pic-related) It also helps me achieve more narrow ratings, such as if I wanted to do 1~5 ratings based on images from a specific character.(I have global ratings and ratings based on other factors)

system:history when the system:history tag is entered, it will display the last n files you clicked on.

>>1871 I don't really 'get' github, I don't know why. Maybe it is their emphasis on collaboration, and perhaps diligence, which has never been my thing. I focused on clearing out the github bug reports a little while ago, and that was working fairly well, but then I lapsed back into my normal routine of checking this board and my email and then going straight to work on my existing to-do. When I am back to normal schedule, I am going to try again, maybe checking one github issue every time I sit down to code. I hate leaving people hanging, and I keep telling myself to sort it out, but I still can't seem to connect the two in this case. I can only apologise for being a sperg, really. If you delete the ffmpeg executable in the install_dir/bin directory, the client will try to just use 'ffmpeg', so if that is in your path, it should all work. >>1883 One of my big planned jobs is an overhaul of the subs dialog. I like the idea of grouping by tree, thank you. >>1896 That's an interesting idea, thank you. I will add it to my to-do list.

I'd like to see there be a "remember this" option in the dialog that appears if you paste multiple tags at once that have tag siblings, so you only have to decide once per tag replacement and allow for an easier workflow when mass pasting tags, please.

Also, while I'm at it, I'd also like the ability to wipe the deleted file hashes from the db to keep that fresh for the "don't reimport deleted files" option. I may or may not have at one point deleted everything and reimported things which left a lot of not quite correct deleted file hashes since I put them back anyways.

I have done an image de-duplicator in the past. you can use this snippet of code if you like: http://paste.pound-python.org/show/Hk4l44hPOVeFZf3XV4v4/ it generates a 64-bit visual similarity hash for an image. calculating it isn't the fastest, but once every image has its hash finding duplicates should be easy enough since you can just use exact equivalence. as input it expects a PIL Image or a filename. it could probably be sped up a bunch if you don't mind a dependency on numpy.

>>1980 Thank you for this suggestion. I have set it to throw up a yes/no dialog if it detects more than one pending potential sibling choice. If you hit yes, it'll preference siblings in every subsequent choice. Although this works when I test it, I am not sure if it produces a good workflow in the real world, so let me know if it causes you any problems. >>1991 I have added a button for this to services->review services->local file service. Again, let me know if it doesn't work right. >>2007 Thank you for this–that's an interesting way of doing it. I currently use a 64-bit DCT-based perceptual hash that describes the general colourless shape of the image. If you are interested, I do it in GeneratePerceptualHash in include/ClientImageHandling.py. My current challenge is searching hamming distance of two such hashes quickly, but perhaps I am being too complicated. Do you know how well your hashes stand up to similar images, like when one has from 9gag plastered on it? I suppose I could run a hamming search like I do for my current phashes. Once I am done with IPFS and a new tag suggestion control, I will be working on this. If I can't figure out an easy solution, I would appreciate your further suggestions.

Don't know if this is possible, but is there a way to order the selection tags so that namespaces always appear on top of the list first? character:a character:b creator:a creator:b series:a a b c Another idea is to group tags. [-][character] —→character:a —→character:b [+][creators] [+][series] [-][no namespace] —→a —→b And yet another suggestion is partial tag searching "character:a* " searches every file that has a character namespace that begins with a. Of course I understand there's already a huge backlog of ideas, but these are just small things that would benefit me greatly

>>2042 I should probably explain my reasoning too. -namespaces on top For the most part I don't look at a files non-namespace tags. The tags I use when finding files are namespaces such as character, creator and series, so it makes sense to have those displayed first. There's also the issue where some files have a shitload of non-namespace tags, and it becomes a pain when I have to dig through them just to find the character. Multiply that by every image -tag groups sorry I'm autistic about organization, especially when it comes to organizing millions of files. thats pretty much it -partial searching This is just an easier way to mass search without having to manually input every tag.

>>2042 >>2046 Thank you for these suggestions. Ordering namespaces is something I have also wanted, I just never got round to doing it. It should be fairly easy, as I just need to add an option for it and tweak the sort code. I'll bump it up my list. Grouping tags is doable but more complicated, as the tag lists are all custom controls I wrote, so I would have to overhaul the internal data representation and external layout stuff to support trees. I will put it on my to-do list and save it for the next time I do a big rewrite of that stuff. Partial tag searching should work already, much like you suggest. Putting a '*' in the search box should insert that exact search string in the autocomplete list below, and selecting that predicate should return all files that have tags that match the wildcard. For instance, putting in 'star*' should give you all the files matching 'star trek' and 'star wars'.

>>2042 >>2046 I second this, because I have stuff like this Tag groups is also majorly wanted. Now, for my own suggestion, I'd want a way to easily identify what file belongs to what url in the detailed file import status window. Something like a query tree instead of displaying everything in one go.

Instagram support? Is it possible?

Has an API or plugin support ever been considered yet? There seems to be a handful of devs here in the community that would be happy to knock out some small features, atleast until they're officially supported in the main program.

>>2164 When you say: >I'd want a way to easily identify what file belongs to what url Can you explain a bit more what workflow you are looking for? Would something like right-click->open selected in a new search window do what you want? Or are you thinking of something like 'as you move the mouse over the lines in the list, the thumbanil is highlighted' or similar? >>2196 Almost any website is possible, but I have generally stopped working on new website plugins so I can instead rewrite my downloader engine to support user-created plugins. >>2197 Yeah, that's my general plan. The next iterations of the downloader engine should support more detailed html/json parsing rules, login FORM and cookie info, and also a migration of all that custom info to external files that anyone will be able to create and import/export.

Regarding tag archives: Would it be possible to create a more streamlined/efficient method of updating a particular tag database, assuming that we as a community can get one or more of them to the point that we have regular updates releasing? I tested out my tentative e6 DB on a blank DB containing only ~7000 images, and it matched 1800 of those but still took 30-60 minutes to apply the tags from the DB. Removing those tags (so I could apply a new version) was a little faster, but still, it is a not insignificant operation, and I'm leery of how bad it will be on my live Hydrus install (which has probably 30-40,000 matches to this one tagrip, never mind the others I'll be working on). I ask because this time around, I'm proceeding with the goal of making the spidering, collation, generation, and posting of the rips I'm concerned with as fully automated as possible, so that - for example - if e621 takes ~8 hours to rip in its entirety, and that process requires zero input from me, I can set up a monthly cron job that releases updates automatically, so those of us concerned with e6-sourced tags have the most up to date metadata possible. Other DBs such as Furaffinity may not be feasible to do monthly - but then again, e6 is a booru, so tags on old files can be changed, and therefore since I can't query by mod date the whole site must be rescanned every update, whereas FA is generally past-immutable and thus incremental update tag archives could be released on a shorter schedule. So, I guess my question is, is it possible for you to code in a streamlined "update tags from archive with new version" operation in, or is that not how the system works? If that isn't possible, what can I as a tag archive producer do to reduce the length of the update operation for myself and other users? I could theoretically calculate a "diff" - where I release two archives every time there's an update, one with added mappings and one with removed mappings - but since I don't know the rate of mapping changes on a large booru, that might not turn out to save that much time when applying them. Is there anything else you can think of? Should this sort of thing be a live remote tag service, instead? I'd actually love that, from a user perspective, except that I'm not sure how to make the current Hydrus server interface work on a headless remote Linux server, which is where I do all this shit, and I'm also not sure how that could be made resilient - if I vanish, I don't want the data I've compiled to become unavailable to the community. Tag archives on Mega.nz will always be there, and can be shared around as files, but if I'm running some custom server, that relies on me providing that resource *forever*. I'm not down for that, really.

Also, I've been wondering - what happens with duplicate tags added from multiple tag archives? For example, Furaffinity and e621 are likely to have many of the same tags. You've mentioned before that duplicate tags mess with your UI code - and I've noticed this in places like the Edit Tags window, where a certain tag has a count higher than the number of files being looked at, because a given tag has been applied more than once (manually, by archives, by siblings/parents). Maybe a "deduplicate tags" maintenance op in the client?

- Rename the window - As someone who's more comfortable using different databases for different parts of my life, this would be a huge help. - Pull whitelisted tags from PTR to Local Tags - There are only certain tags I actually care about when it comes to auto-tagging, so this seems like a good way to only show the tags I want to see. Regexes and Namespaces would be excellent here. - click-drag selection - just a qol change. If it's unpopular, maybe have it default to disabled - a tag option that gives each file tags based on the names of the folders it's in - easy way to track and tag images after an import

>>2226 Also, did I miss a release note where the client_archives folder got deprecated in favor of the "apply archive" buttons, or am I missing the appropriate area of the UI to configure that now?

Is it at all possible to exclude files of a certain rating? Like "-system:rating for R = 0.5" or something. Or indeed, a ≠ operator.

>>2226 That's an interesting thought, and I am not sure the best answer. The more generalised and powerful I make tag archives, the more they look like hydrus services. I am strongly considering moving all tag archive functions to the new extracted service database I am planning. When I pull client.db apart, all service-specific data will be extracted to its own folder. It'll be a portable container for most sorts of hydrus-compatible content, so it'll probably be able to do everything tag archives currently do, and with siblings and stuff as well. So, in future, I'd like the user to be able to go something like file->import service->pick file/directory/whatever->would you like to import it as a new service, or merge it into an existing service?. Maybe with a single wizard that covers a lot of potential bases, rather than having different functions hidden all over the client. For your plan for now, it might be reasonable to make three-monthly update archives subsequent to your original 'e621 up to 2015-09' or whatever, and then people can add those smaller (~50MB? something like that?) files a few times a year rather than having to deal with larger ones all the time. Having two archives of 1970-01->2015-09 + 2015-09->2015-12 works the same as 1970-01->2015-12 from hydrus's perspective, it'll just take two db queries. Having your users manage multiple files could be awkward, but it is easy. The only difficulty there is figuring out what is the new content. If there is a way to grab that from the site (like if you know the guy who runs it, who has direct db access), then that's easy, but I presume you are expecting to have to do a full sync every time you update your tag archive, in which case you can go: A = Tag archive 2015-09 B = Tag archive 2015-12 C = New tag archive For every mapping in B: If it is not in A: Add it to C. I am not sure if tag archives currently support membership queries, but if you figure out any new calls you would like, just let me know and I can add them no problem. I can even add a 'create_diff_tag_archive( A, B )' or something. Or you can query them directly, if you are comfortable with the sql. I can easily add 'merge these two tag archives' code as well, but I don't want to get tied up writing client gui stuff for it when it might just be generalised to the new service db fairly soon anyway. >>2227 Exactly similar mappings are merged on the same service, so if a tag is 'current' for a file on a tag repo, trying to 'pend' that same tag to the file is a no-op. The duplicate tags issue comes in gui-side, with siblings and when the client has multiple tag services (e.g. 'local tags' has a lot of tags and you have multiple tag repositories synced that have file and tag overlap) but the viewed domain is 'all known tags'. It is just a bit more difficult to quickly and accurately union and count siblings and merged services on-the-fly, so in some places, those counts are inaccurate. Otherwise, applying multiple tag archives that have very similar content to a single repository isn't a big deal. Anything duplicated will be (quickly) ignored. >>2245 At the moment, I think you can do two things with tag archives: 1. Permanently sync to them, by putting them in client_archives and then booting the client and going services->manage services->tag service->archive sync. 2. Doing a one-time import via services->review services->tag repo->perform a service-wide operation. I don't like the workflow for either, really, nor the dichotomy of those two dialogs. The manage/review difference represents what is going on behind the scenes, but not what the user necessarily wants to do. I might end up merging them one day, or removing all commands from review services, or something. >>2244 - Rename the window Great idea, thank you. I would use this myself! - Pull whitelisted tags from PTR to Local Tags I hope to greatly expand tag display filtering and service-to-service copies and moves once the client.db split-up is done. I hope to pull all service data to separate directories, and I expect it to be easier and simpler to do service operations at this point. When I start work on that, please give me your feedback on the gui I create for it and make sure I include the features you are interested in. - click-drag selection

[Expand Post]

This has been on the wishlist a long time (so long that I think I put it there!), and I have just never found time for it. I think it was on the last 'vote on what to work on next' poll, but I cannot remember how well it did. - a tag option that gives each file tags based on the names of the folders it's in You can do this now, but it is technical. When you import some files, click 'add tags based on filename', which will let you automatically parse tags from filenames using regexes. These are powerful and can get quite complicated. If you are a programmer, you may be familiar with them and can figure it out yourself, but if you are not, let me or the board in general know, and someone will help you. My attached image shows an example regex that parses simple folder names. It is: (?<=\\).+?(?=\\) I am not very good at regexes myself, so there may be better ways of doing it.

>>2256 Your logic re new content is what I was musing about - the problem is removed tags. If there is a mapping in A that is not in B, I have no way, in the HTA, to say "this mapping should be removed if it exists". I suppose I could do an "add these" update db and a "remove these" update db, which could be one-time applied, but I'm not sure that's worth it. I'm especially not sure that running a DB that has 50,000 mappings on 50,000 files into Hydrus is going to be appreciably faster than running one that has 500,000 mappings on 50,000 files (just using numbers pulled out of my ass). Question: Does the permanent sync still take a really really long initial processing time, or does it just sit in the background? For FA, incremental updates can and will work, though. I like your services idea - that has potential.

>>2259 Ah, I see. Yeah, tag archives have no deleted knowledge yet. The service db will, so again that's probably the direction we'll head. Initial sync takes ages, although it no longer locks up the gui. It now runs off a pausable and cancellable popup window that does 50 files at a time or something, so the program runs a little loggier, but is still usable. The total time it will take will be roughly proportional to the number of files that are cross-referenced. If only 5,000 files are matches, it will be about ten times faster than if 50,000 were. (The exception to this is if the tag archive is sha256, in which case 100% of the files are matched, so it'll take ages every time.) I was heartened to see tag processing generally working faster with the recent autocomplete cache change, so we'll see how that generally holds up as I continue to alter the cache. I expect some further speed-up and also some slow-down, but I am not sure how much they will balance out.

Would it be possible to add a "censor this tag" option to the right-click menu you get in the selection tags menu?

>>2328 Sure, I will add this to the list.

How about an option in settings for slideshows to continue only after the current gif or video has finished playing once? This would be useful for viewing a mixture of static and animated images with a 5 second timer that could be extended only for long gifs and videos.

I have a lot of images that look similar but with small alterations. A new way to sort, placing similar images next to each other would be really nice. I'm not sure how Hydrus's "find similar images" works, but maybe it would be useful in this case? If not, sorting the list by dimension and file size would probably be good enough. Use something like "width height file size" as a string and sort by that, perhaps? It won't be perfect of course, but it would be good enough.

>>2357 That's a great idea, thank you. I'll set it to advance the slideshow if autoplaying durational media has played through at least once, and we'll see if further options are needed. >>2362 I have big plans to handle near-duplicate files' metadata in the future of the program, including sharing that between clients like tags currently can be. I'd love to be able to automatically merge tags across extremely similar images (e.g. when one has a watermark, or is just a resize), and also to group your sorts of 'small alterations' images together. After I am done with this first version of IPFS and then a tag suggestion control, I am due to write a new 'find similar images' algorithm that should work much faster, which will be the first step in being able to search for and display these sorts of 'these two images look the same–is one a better quality version?' types of filtering workflows.

How about a native iqdb lookup and tag? It could go like manage tags->lookup on iqdb->dialog with possible matches->click one and it shows your copy on the left, the booru's on the right, and a confirm tags button.

>>2378 Thank you, this is a great idea. After I am done with IPFS, I will be adding a 'suggested tags' control to the manage tags dialog to pull suggested tags from external sources, and iqdb would be a nice plugin to it.

Could you add FoolFuuka / Fuuka support to the thread watcher / gallery downloader? I'm trying to download the /e/ Ecchi Image Collection (around 120 threads), but it'll be a hassle to DownThemAll everything from the archive before importing it all.

>>2384 I have stopped manually adding complicated new site downloaders for now. My new plan is instead to ultimately expand my downloader engine to better allow users to create and share those site profiles themselves. The next time a 'what big thing to work on next' poll happens, please vote for the downloader engine expansion.

Could you add more useful info to the status bar when only one image is selected? Dimension and file type perhaps? Would make it easier and quicker to delete duplicates.

>>2402 Thank you for this suggestion. I have improved some of the status bar grammar, and when one file is selected, I have appended the same info text you see at the top of a single thumbnail's right-click menu. For instance, you might now see: 480 files - 1 image selected, 7.6KB image/png (200x200) Let me know how it works for you. In future, I would like to make this customisable, like how some mp3 players allow you to configure the in-program label of a song like: [artist] - [album] [album_num] - [title] But for hydrus like: if multiple selected: [selection_num] totalling [selection_size] if single selected: [focussed_file_mime] - [focussed_file_size] - [focussed_file_resolution] It'll be a larger job to allow this, but it is on my to-do list. If you are interested in this and I don't get around to it by the time I have the next 'what big thing to work on next' poll, please remind me then, and I'll put it on that.

>>2419 That's fine, thank you. Not that interested in having it fully customisable because I can't really think of anything else that's missing now that you added filetype and dimensions, but maybe someone else would find it useful.

So I am not sure who else would be interested in this, but I wanted to toss it up and maybe get some other ideas. So, since hydrus uses its own naming scheme, file names are messy as hell, and quite frankly, ugly. One step further, I have to admit, filename threads really are my bread and butter. I think the ability to export files based on specific parameters is certainly a worthwhile feature. One step further though - I think there is some good potential here for some powerful features that only something like hydrus could provide. Tags are awesome. When I say export, I am referring to drag and drop interfaces FYI, since that is the easiest way to share files on 4/8chan. So, I think an interface that allows some detailed information when exporting would be awesome. For instance - the ability to set custom titles based on namespace. For example - exporting porn? List the title and artist. Exporting a dank meme? Export based on a file name name parameter. Dumping a series? Add page number parameters. One optional step would be able to optionally name them all by "author:'paramater' - title:'paramater'" I don't know how many people would find it useful, but could certainly be a huge step for those who like to dump. Just tossing some stuff into the think tank. R8 and suggest stuff, because I'm sure there is some stuff I'm not thinking of.

>>2419 >In future, I would like to make this customisable How about this instead? https://github.com/hydrusnetwork/hydrus/issues/135 Also, I made this request earlier. I know you don't check the github issues often, so I'm just going to inject it into this thread I suppose :) https://github.com/hydrusnetwork/hydrus/issues/168

Can you add a shortcut hotkey for the "remove from view" function that's below archive in the right click menu? It would be a boon to one of my scripts.

>>2449 I would very much like to do something like this. If you haven't seen the export dialog, check out the regular thumbnails' right-click share->export->files, which lets you generate 'creator - series - chapter - page.jpg' type filenames based on existing tags. I would like to add a similar pattern-match to quick drag-and-drop exports as well. >>2469 I like all of these thoughts. I've wanted to do a couple of them for a while as well, but they are large jobs that will take me several weeks to do. Please bring these up again when I next have a 'what big things to work on next' poll (if they are not already on there), and I will add them to the list. >>2474 Done, thank you for the suggestion! It should appear dialog off the file->options->shortcuts page. Let me know if it doesn't work for you.

Larger thumbnails? I remember there was a technical reason they were capped at 200x200, is that still the case?

A suggestion for the ipfs plugin: How about exporting local booru galleries and pages as a way to share multiple files?

>>2589 I've had a think about this, and having recently played around with hydrus on a 4k screen, I can better see why people might want more. I'm open to adding larger thumbnails, but they'll appear a little bit slower the first time they are loaded and anything from a file repo will have to be upscaled from the network-delivered 200x200. It's now on the to-do list. >>2590 I would love to add all sorts of IPFS stuff like this in future. Ideally, I'll make a mini-API so you'll be able to deliver all sorts of shareable content on any sort of p2p network.

>>2592 In particular black and white comics get horrible moire artifacts in the thumbnails, so any improvement would be welcome.

I reckon there should be a window where you can manage tag siblings and tag parents at once. I think it'd be useful to be able to manage both of them at once without having to switch windows. Cheers

>>2688 I would like to do a lot with siblings and parents, mostly filtering the giant list and displaying and editing the relationships in a more visual way, but have yet to find the time. I like the idea of combining the two workflows into one ok button. Maybe if I turn it into a notebook with sibling/parent tabs, like review services has local/remote? Or were you thinking of displaying both sibling and parent information at the same time?

Can we get an option to hide all tags/text in fullscreen slideshows?

>>2715 Sure, I'll add it to the list.

>>2698 Yeah I was thinking the notebook thing

Something I've been thinking about for a while which could have a few uses are sub-tags, or a tag hierarchy. If you for example have tag 'X', and you add a subtag to that, the tag X will have a small button next to it in the tag list which can expand into a list of subtags. This is kind of like tag parents, but useful for other things. The main difference is in neatness and clutter. If you have tag Y which is the parent to tag X using the current parent system, they will both show up in the list of tags. If you're spergy like me and add a lot of tags for your own collection, it can get messy fast. However, if you had a tag Y which is a subtag to tag X instead, tag Y is only visible on the list if you press a little button or arrow next to tag X. Another thing I'm a little unsure of whether would be a good idea or not is extending this to seaches, where you would only search for tag Y by writing the whole tree that it is in. Therefore searching X(Y) or whatever syntax would be good would return images tagged with Y as a subtag of X, and simply searching Y would return images tagged with Y as a top-level tag. I hope this text was not too autistic to understand.

>>2756 I think I understand what you are getting at, and I think its interesting. Tag clutter is something that's been on my mind as well, and I think collapsible trees may be part of the answer. I'll also be adding 'gui censorship', so you'll be able to filter page: tags or whatever from spamming the selection tags box without removing them from sorting rules and so on. Before we can do any tag grouping, though, I need to write tree support into the taglist control. The data behind it is just a flat list at the moment, so I'll just have to rewrite it to be more clever and to do collapse/expand events. I'd like, in lexicographic views, to be able to quickly collapse namespace, and I'm sure final fantasy could be reasonably hidden behind final fantasy vi for many purposes.

How about an option to keep a copy of tags you apply on the PTR in the local tags as well just in case?

>>2756 I've been using a dotted notation in my tags to kind of simulate this functionality. I have tags like this: hair.black hair.black.blonde roots hair.long.knee length car.sports.ferrari.f355.spider Basically each dot creates a sort of parent tag for all the ones before it, since I can search for car.sports.* and find all the tags under it.

Can we get a way to empty the trash without doing the search window? I routinely delete tens of thousands of files at once, and currently when I open the trash search window and select everything to right click and delete, Hydrus takes off reading all the selected files' thumbnails and kills the disk for about 5 minutes.

Being able to rename namespaces would be nice, rather than having to create a ton of tag siblings.

Similar question as >>2796 Is there a way to remove specific PTR tags/namespaces from my local files? I don't really want "species" tags on my cat girl images…

This has probably been suggested before since it seems like an obvious feature, but in case it hasn't… Hydrus could really use a "browse" mode. I would suggest a new kind of page that lists all tags in your database on the left (with a dropdown menu to limit it to a specific namespace), and when you click/select a tag all images with it will be shown. Sometimes you don't have a specific tag in mind, and just want to browse your collection, you know?

>>2788 That's an interesting idea–do you mean as a backup, in case your uploaded tags get deleted serverside by a janitor? >>2792 Sure, I'll add a button to review services. Be warned that it will still likely be a very laggy operation, as transferring any single file between file domans (like local->trash or trash->None) needs to update a bunch of caches (like the autocomplete tags caches), which can add up if you are doing thousands at a time. If you haven't already, check out file->options->files and trash for some options about how files are automatically removed from the trash. >>2796 I want to add this in a future version of the tag censorship system. I would like it to work just like siblings, like artist->creator. >>2797 Check out the tag censorship system, at services->manage tag censorship. Select 'all known tags' and then enter 'species:' into the box and hit enter. You might need to refresh your views, but you shouldn't see any species tags in normal views again. You can put in 'species:catgirl' or whatever if you want to get specific. >>2822 I like this idea. Some people have asked for a tag cloud, which I like as well. It'll be a larger job, so I will put it on the next 'big things to work on next' poll when that next comes around. For now, you can emulate this behaviour by going 'system:limit=1,000' and setting your tags to 'incidence (desc)'. You'll get a list of your most popular tags on the left, which you can middle-click to quickly open the page for.

I'd absolutely love to see "greater" and "less" options for ratio search. This shouldn't be hard to implement, considering approximate search is there. I have dual display configuration with display oriented differently and separating portrait and landscape images it way too much PITA for no real reason. Also, image viewer' is definitely lacking "lock zoom" option.

>>2844 Though, given that ratio is two numbers, not a single one maybe "wider" and "taller" would be more descriptive.

>>2833 Yes, in case your tags are deleted by a janitor.

How about when you right-click a tag on a local search page there's an option called "remove tag from selected items". It would be so much quicker than right-clicking the items and opening the "manage file's tags" window (which by the way is pretty slow to open when you've selected many files). Since I used a lot of tag databases when importing my files there's quite a lot of garbage tags that I would like to remove, and being able to do it directly from the search page would be quick and easy.

A small thing that I would really like to see: score-like ratings, like numerical ratings, but without a fixed star number, every image would have 0 by default and there would be key bind to add or subtract 1. This would make it way easier to have an "used" count for pictures, currently I have to set a 10+ stars rating and bind a key for every single star in my filter. Even better, have an option later to automatically add 1 automatically every time an image is displayed in the media viewer for at least x seconds without user interaction.

>>2844 >>2845 Taller/wider ratio options is a great idea, thanks! I also have it in my to-do list to add a lock-zoom to the media viewer. >>2854 Ok, I will think about how to do this. I've wanted to do ongoing service-to-service sync for a while, so I think this could be folded into that. >>2855 I hope to reduce the manage tags open time for a large number of files in the current 'suggested tags control' work I've just started. The delay isn't necessary–I just need to rework some things. I like the idea of pushing some quick tag management out of that dialog as well. I'll think about how I can do it in the right-click menu. >>2856 I really like the idea of an integer rating. I've added it to my to-do list. I also like adding some 'viewed for 2 hours'-type metrics to files, and making that searchable. I've created a sub-project to think about it, although it might be a bigger job.

>>2859 While we're on the subject, do you have any thoughts about these ideas? https://github.com/hydrusnetwork/hydrus/issues/154

>>2867 I used to have a 'ratings filter' that did almost exactly what is described there, with two media windows to compare. It had a neat workflow and decent algorithms to remember past comparisons, estimate certainty, and provide maximally useful pairs to look at–and it didn't work very well. I am not sure why. Not many people used it, and those who did didn't like it (myself included). I think I might have made the comparison calculation too granular, so any real world use case would have taken too many comparisons (i.e. n log n) to accurately figure a score for everything, or maybe my comparison algorithm should have presented less random comparisons (by showing files with common tags or something), or perhaps (I suspect) there is something fundamentally irrational/emotional about human ratings, so trying to build a rational workflow based on information theory is not going to meet success. I think I could bring something like that back, but I'd need to come from a different angle. It has been a very long term thought to share ratings on a service like with the PTR, but merging that data in a 'hydrus network' way is significantly more complex. I often find my opinion differs from others, so merging ratings might not be the right direction to go. Maybe for more discreet ratings like 'this is technically well put together'. I think stat collection could be neat, although I think it is easy to get cluttered and to spam the db with too much useless info. I think I'll try 'total view time' stats and we'll see how that works. It might be neat to tie that sort of thing into a future auto-ratings system. It feels to me more organic to harvest rating data from the shape of normal usage rather than literally asking for 'how good is this' out of the blue. It might be neat to see a 'collectively, users have spent 5.3 years looking at this image'.

>>2868 Relative tagging sounds like a lot of manual labor (worst case n*n/2), but there is indeed an issue of one's views changing and all. Maybe adding (searchable) age for ratings will help? Then one can look up stale ratings and refresh if necessary. >>2488 BTW any plans for this?

I'm actually not sure if this is already possible, but here goes anyway: How about a way to black/whitelist namespaces for uploading to the repository? I would like to contribute to the database, but some of my namespaced tags are entirely subjective and would just be clutter to other users. Having all these under one namespace that gets ignored for the commit would still allow me to upload everything else

>>2877 I'm split on adding timestamps to each content row. The server needs it to generate updates, but for the client it would add 20-40% database size and be another dimension of metadata to load for every file. I'm still thinking about it. I wonder if trying to subjectively measure/rate/tag any single collection, beyond simple decisions like inbox filtering, is too much work for anyone alone. Maybe it is something else we'll end up offloading to a neural network in five or ten years. >>2488 would be a big job, so I will have to put it off until my next 'what big thing to next work on' poll. Please mention it again then, and I'll add it to the list. >>2904 Could you store the subjective namespaces on your 'local tags' service, but put everything objective on the PTR? Stuff on 'local tags' stays private on your hard drive.

>>2906 >Stuff on 'local tags' stays private D'oh, that makes a lot of sense. And should do just fine. Think I better re-read the help section before I end up suggesting another feature that's already in.

Is it possible to disable thumbnails during download sessions? I'm currently at 40000 and performance is noticeably beginning to degrade. If I have multiple download sessions up simultaneously, then navigation becomes impossible.

>>2927 Also, this isnt exactly a feature request as I dont think its consistent with the dev's vision of what hydrus ought to be, but does anyone know a platform where I can curate based on artist and possess rudimentary tagging features? I wish to organize via art style and thematic tone, something that requires a lot of finesse and nuance. And tags are too blunt an instrument for this. The best way I can think of to do this is to organize based on artist since most artists dont change their art styles or thematic matter very often. Even better if the platform allows user input such as descriptions and thumbnail viewing.

>>2907 No worries. Let me know if there's anything else you aren't sure about, or if you have any ideas on how to make the help more readable. >>2927 Yeah, at the moment, you want to go: Pause downloading. Select one file. Hit Ctrl+A to select all (wait a few seconds for it all to go through and animate). Right-click->remove media. This just cleans your current view. Pages aren't great at showing >10k files, so I generally suggest you keep you downloads/imports/searches/whatever a bit smaller than that if you can, or if you are importing, to regularly cull them back to a manageable size. I might add a 'just do this download in the background, don't show me the results' mode, although subscriptions somewhat already fulfill this, and I want to shy away from letting people accidentally fire off a 200,000 file download that gets them a bandwidth ban or whatever. It is very easy to get files, but not to filter and manage them.

Could we get somewhat better tag sibling and parent controls? Right now the only way to disable tag siblings and parents is to petition them, which can lead to a petition war. You should be able to mass select certain parents and siblings and choose from a right click menu to either petition or disable them (or enable, if they were disabled). Petition disables them automatically, and they're hidden from view unless a "view disabled" checkbox is checked, at which point they appear as they do now with (-) if they're disabled.

I'd really like if there was a way to edit a subscription's name.

>>2942 Also, this is really becoming a problem. The window doesn't have a scrollbar, so I can't confirm the rating.

Smooth scrolling would be amazing!!

>>2948 As an option, maybe. Smooth scrolling would interrupt a few of my workflows that rely on it being normal scrolling.

>>2949 I always get lost while scrolling, the jumping is too big! Do you have a suggestion on how to "fix" it? While I usually don't use smooth scrolling in anything this time I actually need it.

>>1111 any news on this quads?

One thing I'd like to see is a sort of 'soft blacklist' of specific tags. Meaning that images carrying those tags would not show up in any searches unless the tag is specifically searched for.

So I can send directories full of files to people via IPFS, but it's less useful when the files are pages etc. and people end up with a folder full of out of order files named by their hash. For other users with Hydrus, can there be a way to send tags from local services like with the txt files? For people just downloading the files, maybe an option to generate a text file of a manifest in the IPFS directory which lists the files and tags, or at least one of the namespaces like pages? If we want to really get fancy, generate a batch file or bash script (or both) to automatically rename the files from their hash to some combination of namespaces like file exporting can do now.

How many would be interested in Farbfeld support? http://tools.suckless.org/farbfeld/ tl;dr it's a lossless raw format that relies entirely on external compression. The way in which the data is organized in the ff format aids compression, and you compress the images with whatever archive tool you prefer. bzip2 is suggested. You end up with: file.ff.bz2 and you must decompress it to view it. Hydrus' viewer would have to implement support for the ff format, AND support for decompressing bzip2/other. I've been playing around with it, and my limited testing suggests it's only useful as a png replacement. For simpler images, ff's compressability via bzip2 is nearly twice that of png! For complex images, ff still beats png compression by 5-10%. I think it'd be useful to have ff support in hydrus; it could save a significant chunk of disk space for us png hoarders. However, for it to be practical, hydrus would need a few other features first. #1 - "Convert this/these png to ff & compress with bzip2" conversion within hydrus, but the db retains the original png hash, which is associated with the new ff image. #2 - On the fly conversions back to png (or any format of your choosing), for export, drag and drop, and opening with external viewers. This is probably more hassle than it's worth, and this is certainly last on the priority list, but it's food for thought.

>>2946 Anyone know how he manages to get that artist rating thingy up?

>>2868 >It has been a very long term thought to share ratings on a service like with the PTR, but merging that data in a 'hydrus network' way is significantly more complex. I often find my opinion differs from others, so merging ratings might not be the right direction to go. Maybe for more discreet ratings like 'this is technically well put together'. To be clear, I would like a rating system for my own purposes, not to merge with other hydrus users. I envisioned it being an entirely offline personal rating. >It might be neat to see a 'collectively, users have spent 5.3 years looking at this image'. This would be very cool, though. Not sure how you'd implement that net code.

>>2939 I like this idea a lot. We just hit this with the recent 1girl->??? stuff. Let me think about it a bit, and then I'll figure out sibling censorship or a list of checkboxes or however I can best add it. >>2942 I recently changed how the 'listbook' control handles the labels on the left to make this easier. I can definitely do this now, and I've made a job for it. >>2946 I have a superjob to convert all dialogs over to auto-scrollbars, but it will require significant rewrites. My current work to make manage tags a frame should kick this off somewhat, as I abstract out the internal panel from the external wrapping. Please hang in there for now. >>2948 >>2949 >>2951 I don't think my graphics library supports smooth scrolling natively, so I'd have to write it myself, which is probably more work than I can easily fit in. I can however add an option to change the minimum scroll unit, which is currently one thumbnail height. As I understand it, most mice send three scroll events per scroll wheel event, so hydrus typically moves three thumbs up or down per scroll wheel hit. If the scroll seems to be more than three rows at a time, check your mouse settings. >>2954 Unfortunately, my gallery downloader engine isn't sufficiently sophisticated to do this yet. I hope to be in a better place on this once the suggested tags control (which will need a better html parsing engine) is done. >>2975 That's an interesting idea. It has been an ongoing thought to have 'nsfw' domains that can be easily hidden, and I hadn't thought of a 'soft blacklist' before. I'm currently leaning towards allowing multiple named 'local files' services, so you can completely separate the search domains, guaranteeing no crossover or tag counts popping up in autocomplete results–would that do what you want? >>2980 I like these ideas. I don't like exporting to hash-filenames, and I have a plan to add auto-filename generation phrases, like for the export files dialog, for drag-and-drop exports, and this could be extended to IPFS. I'd also like to add cbr/cbz support of some kind to ditch the whole problem with page: tags. >>2981 That's an interesting format. I think it is a little obscure for our purposes, and it doesn't look like it has a simple python library I can grab, but I could emulate the behaviour by compressing files before they get stored to disk, and then uncompressing them as requested. I do a bit of that now with bmps in the in-memory image caches. I can see a few benefits and drawbacks to that. For the moment, I think CPU is in greater demand than disk space, but if that changes, this is something to think about. >>2984 This is the normal F4 manage ratings dialog, but he has created a lot of ratings based around artists. Go to services->manage services to create your own ratings. >>2993 Yeah, I agree on both points.

>>2997 >That's an interesting format. I think it is a little obscure for our purposes It was created just a few months ago ;) Should you choose to tinker with it, hydrus would be the first program to support it. I doubt FF will ever become widely adopted, but hey, hydrus itself is rather obscure. If FF proves to be valuable, I could see it becoming popular with hydrus users (particularly as ipfs grows). FF's complete obscurity is why I suggested two other features be implemented first (easy conversion to and from FF, associate pre-conversion hash with FF file). I really do not expect you to waste any time implementing support for FF unless you and the community think it would be worth having. >I could emulate the behaviour by compressing files before they get stored to disk, and then uncompressing them as requested. This would actually be counter productive. FF is easily compressed because it stores /raw/ image data in repetitive, predictable sequences. Conversely, any other image format is already compressed (for compression is a part of the format). Trying to compress what is already compressed will yield no benefits. In other words, the FF format here is not ornamental; trying to use external compression tools without using FF would not work.

>>2997 >I like these ideas. I don't like exporting to hash-filenames, and I have a plan to add auto-filename generation phrases, like for the export files dialog, for drag-and-drop exports, and this could be extended to IPFS. I'd also like to add cbr/cbz support of some kind to ditch the whole problem with page: tags. I don't think you're going to avoid the page problem with cbr support, since the same problem would occur with any namespace the user cares to sort with. For the moment if you could make a way to export filenames along with a tag manifest, that would be good enough. I'm not aware of any other programs that generate IPFS directories so easily. Hydrus could be a huge kick in the ass for IPFS adoption once directory hashes start getting posted.

Could you make thumbnails for GIFs and 8-bit PNGs be resampled properly? While the irony of pic related is kind of amusing, those ugly thumbs are not fun to look at.

I have a question, or if not a suggestion, can you change the program that the client uses to open external ?

>>3025 Your system's default program for that filetype is used.

>>3027 Sorry i think i wasn't clear. I'd like to keep my system default image viewer but when i go and open externally i'd like it to open in explore with the file high lighted. I hope this was more clear

>>3019 Can you please post that original image here, so I can try it my end? I have thumbnails set to Antialias right now, which I think is the best quality my image library permits, but perhaps it isn't firing correctly. Also, did you recently import that file, or might it be much older? I have used worse resizing algorithms in the past, so it might be something I should cause to be regenerated on update or something. >>3025 >>3027 >>3028 I'll be adding custom launch commands for all mimetypes in future. If you set 'explorer' as your command, I think this'll do what you want.

>>3034 In the process of drafting a reply, I may have found the root of the problem. As I said, any image with indexed color, either GIF or 8-bit PNG, produced the same effect. The first attached image is an indexed PNG, and the second is the thumbnail generated by Hydrus. The third and fourth are the original image scaled to 200x200, using bilinear and nearest-neighbor resampling, respectively. As you can see, the actual thumbnail is close, though not identical, to the nearest-neighbor version. I believe this is because whatever is resampling the thumbnails is doing so only within the indexed color space, meaning any intermediary colors that would be generated are instead rounded to a color that exists in the original, making it effectively a nearest-neighbor resample. If this is indeed the problem, then a copy of the image would first have to be converted to full RGB space, then scaled, to make the thumbnail. And yes, this is definitely a current problem, as deleting and re-generating the thumbnails didn't help.

Here's a request. Can the external tag databases written on sqlite also store the ID from external boorus if there is one? I would like a way to match the booru image ID to the md5 or other hash. I'm using it in some custom code to make sure I round-robbin on all booru's I have configured that have a given hash along with validate I have a full copy. To store it in my database, I need the image id though as it is part of the key. I'm actually using ms sql so I can offload the database processing to purpose built image on my regular VM host but keep the images in a cheap magnetic filesystem over iSCSI. By the way, awesome work on this program.

>>3037 Thank you for this excellent followup. I think you are correct. I have written a check to convert explicitly to RGB/RGBA before any scaling, and set all png files (rather than just those with transparency) to create png thumbs, and now things look much better. This will kick in in tomorrow's release if you regen your thumbnails again. Let me know if you encounter any more problems! >>3039 The db structure of an HTA isn't that complicated, so check out HydrusTagArchive.py if you haven't already. If you want to extend it, you can do so manually with the help of the sqlite command line or SQLiteStudio: SQLiteStudio: http://sqlitestudio.pl/ SQLite command-line shell program: https://www.sqlite.org/download.html You could add something like: CREATE TABLE booru_ids ( hash_id INTEGER, booru_id INTEGER, booru_image_id INTEGER, PRIMARY KEY ( hash_id, booru_id ) ); And have a fixed enumerated value for your boorus, like DANBOORU = 1 and so on. Or you could just use the existing system and add namespaced tags to your images, like 'gelbooru_id:123456' and then fetch and filter them in whatever script you are using to put this together. It sounds like you are doing something complicated, so let me know if I misunderstood your question.

>>3055 Going to try posting this as 2 posts because 8chan won't let me post the whole thing. Sorry, I didn't explain that very well; it was a driveby post. I have a custom built image suite similar to HTA that I started writing in 2002 on VB.NET. I started writing a program that acts like tineye does today; finds images that are duplicates based on how similar they are (I use custom algorithms though). It morphed into a image categorization suite, and I hooked it into a bunch of different booru's API's just like HTA so it could download the tag and image data because I am lazy. It stored the images on disk very similar to HTA, except it was two layers of the MD5 (first 2 in a folder, then next 2 in a subfolder) so that there were 65536 folders for regular images, then a 200 by 200 thumbnail that was auto-generated in a mirrored folder structure. There was a UI that pulled up images and such based on search terms, and kicked out 'booru style XML. The last thing I did with it was to create a archive torrent of rule34, which is 66 GB (and can still be found easily) The database was MS SQL, specifically so I could leverage the full text index processing that was possible, and almost no other RDBMS's supported. Shortly thereafter, my array exploded and I lost the front end source and some of the database information because I was dumb and forgot to back it up. So I dropped the project for around 10 years. I just started it back up a month or so ago, and am improving it.

But while searching for a extract of tag databases from the 'boorus so I wouldn't have to scrape them all again (and piss off my ISP), I ran across HTA. I checked out the sqlite tag databases that were uploaded/ripped from the sources, and I could get a feel for how HTA works at the database layer pretty quick. The only thing I saw missing when compared to the program I use is that I have a reference table that tracks the MD5 for an image and the source site. Besides that, I do have a config table with the API (or website layout for scraping) for a ton of 'boorus. Danbooru is actually ID 5 :) The reason I was asking for the reference table was twofold. It would allow me to not have to scrape other sites if someone already did, because I could just import directly from the sqlite database. Second, it would further my personal project of making sure that a website could be super easy to rebuild, all the way down to the image ID's. Just export it pretty much. This quick query will actually kick out the hashes and tags from the sqlite database into a simple format that my program can use, I just can't capture the image ID of the booru. I used it as a test to make sure I could easily integrate with HTA. <removed because 8chan hate sql aparently> Honeslty, once I'm done on this end, I'll probably just kick my tag list out to a HTA sqlite database (and probably 'booru xml) that can be imported or used wherever. My data goes back to 1997, and I have stuff for sites that don't even exist any more. My tag database (tags only) is 14GB, not including indexes or any other structure. I need to get a copy off my box so that it can be free out in the world. It will probably take me 6 months or so to clean up all the stuff I have to make it useful, and I plan on doing that extract no matter what.

So I accidentally deleted a couple of tags from a selection. I opened system:everything, selected the all the files and found the 'Show deleted tags' button. However I couldn't undelete the tags, if I tried to re-add them, it would add to every file selected. So I'm here asking if it's possible to add a 'recover tag' option

There needs to be some options for large folders that have a bunch of other folders inside of them. Currently it just looks through the entire folder, but there should be some options to have it ignore a folder or something.

Can we get the thumbnail generator for video mimes to read a frame from ~20% into the video instead of the beginning? I'm seeing a lot of black screens for video thumbnails.

>That's an interesting idea. It has been an ongoing thought to have 'nsfw' domains that can be easily hidden, and I hadn't thought of a 'soft blacklist' before. I'm currently leaning towards allowing multiple named 'local files' services, so you can completely separate the search domains, guaranteeing no crossover or tag counts popping up in autocomplete results–would that do what you want? That would be absoloutely gorgeous! I already suggested a sfw-mode in another thread, and it's exactly what would make this program absolutely perfect for my uses. This would be especially nice combined with an algorythm to check booru sites for their 'rating:' tag and automatically categorize them into their respective file services. The problem with these tags is that they aren't displayed as normal tags, but are in the sidebar under the normal tags on most boorus. It would require an advanced scraping mechanism. That's my main request, but I wrote down some more during my time using Hydrus: Small things: -Abort and hide to tray button on the data management progress screen after closing the client -Reminder or login-button at the place of the pixiv download button when not logged in Big things: -Multiple local tag services -SFW Mode/multiple local file services -Enhanced Scraping: -Pixiv Gallery Downloader -Page scraper scrolls down until the end before fetching links option (=Twitter Downloader) -Download 'rating:' tag from boorus Unreachable(?): -Mobile Support

I'm stuck with internet from circa 2002 because my provider is stupid and won't put in fiber unless we got far too many fucking people per square mile. downloading a shitton of images takes a long ass time because of my less than stellar connection. and for every piece of gold there's twenty corns of shit that I have to prune. would it be possible to do a different download mode that gets only thumbnails based on the search? ideally it should prevent opening the file itself, but allow you to choose files to queue for download. I'm hoping that by using only thumbnails before queuing file downloads that it will make cherry picking files significantly smoother, even if more may have to be done in the viewer.

>>3136 I just noticed one more thing - you can't rename subscriptions. Not a big thing, but kinda inconvenient.

I clicked the "Remove all tags" button in the manage tags window with 13588 files selected. Hydrus has been frozen or working for 6 hours now. It would be great to know which. It seems like a very long time for a seemingly uncomplicated function. I'm afraid to terminate the process in case it will mess up my db. Please add progress bars/counters to every major operation so we have an idea how fast it's working or if it's working at all.

>>3175 After 12 hours Hydrus was still frozen and using 11GB of RAM so I shut down the process. Started it up again and it seemed like nothing had changed. Seems like a bug if you ask me!

>>706 >You can do this with a custom filter, although the interface is not ideal. I want to make shortcuts custom across the program so that eventually you'll be able to set specific tags or ratings from anywhere. It'll just take time. really looking forward to being able to use these shortcuts outside of filtering. something ive also noticed is you cannot filter on collections, which is quite awkward. suggestions id have are: sessions saving the sort/collection that's set. being able to customize the thumbnail overlay. have my own dreams for the server setup too, but i have to take a moment and thank you for hydrus, it's in a league of it's own, much <3. crazy amount of effort.

Sorry for the late replies here–there's a lot going on for me IRL, and I'm having trouble keeping up. >>3059 >>3060 Your program sounds really interesting. Since you are an experienced programmer and specific needs, I think your best bet is to add the table and methods that'll fit what you want yourself. If you want to pair md5 to booru image id, then the table in >>3055 should do that. To manipulate the schema easily, check out either SQLite Studio or the official SQLite command line shell: http://sqlitestudio.pl/. https://www.sqlite.org/download.html. >>3063 Sure, I'll add a specific 'undelete' button to manage tags. >>3118 I plan to have options on this when animated thumbnails are done. >>3136 >>3144 Thank you for these suggestions! >>3140 Unfortunately, I really designed the downloaders as scrapers, so I would have to write a whole new engine to support 'browse' downloading. Please bring this up again the next time I do a 'what big thing to work on next' poll. >>3175 >>3178 Thank you for this report. I am sorry the client froze on you–I added that button as a shortcut for the double-clicking a few dozen times, and did not expect it used on many tens of thousands of tags at once. If it freezes again for you, it is generally safe to kill the process, as the database is atomic. It is much more likely to have a db problem due to hardware fault than software interruption. If you would like to mass-edit tags, please go to the 'advanced operation' dialog, which can be launched from the appropriate button on manage tags or review services. It sends queries directly to the database where they can be processed in a raw and efficient way rather than at the gui level, which will wrap most actions in human-friendly objects and throw up 'do you want to do A or B?' type gui stuff, and which will break if you throw more than a few hundred things at once at it. >>3181 Thanks–both of those are planned!

Can you add keyboard shortcuts for the options on the manage tags dialog, like paste and apply? AHK can't seem to reliably ControlClick paste tags as the Button classID changes randomly when you open the manage tags dialog, and match by text also doesn't work half the time. I used to use absolute X and Y values to do this, but that's gotten difficult with the recent window changes, and I have two installs I work with that might not match, so I'd rather have something more reliable than X and Y values or screenshot matching.

This might already be possible and I just can't figure it out, but how about a way to 'batch' find similar images? That is, for each selected image, run 'find similar' on it, and if it turns up more than one result, add them to a set of results to display in a new page. This would make it easy to resolve duplicates at different sizes and compression levels, especially for images that aren't extensively tagged. I'm guessing this would be a fairly expensive operation, but even limiting it to batches of 100 at a time or something would be far better than checking every image individually.

>>3224 That has been 'in the works' for some 2 or 3 years now. Also, with subscriptions, they only check the first image to see if there was anything added, correct? What if that tag gets a new image by someone editing a year old post? There's no way to get those images without nuking the entire subscription and redownloading it from the start. Could we get something like that? A deeper check or something? Maybe an option, at least. There's a lot that could be improved with the subscriptions but that could be something, at least.

>>3229 A way to retry all those failed entries in subscriptions automatically would be nice. And a way to do that for all the subscriptions. It's not that easy to do when you have hundreds of them and hydrus crashes around the 50th losing all your progress.

I may well be the only person to use Hydrus for sharing images, but here goes anyway Is there any chance of shared files getting a unique filename, or even one that is generated from a user-set syntax, like creator-title-page? Currently every single file is 'file' without an extension, which makes saving things a real pain Even just using the md5 plus mimetype extension would be a massive improvement

>>3194 Actually, I am about 90% scraping all the boorus already because I left my program running while I was AFK for 2 weeks; currently 15 millon hashes I pulled about 1200 boorus I believe (I'll have to check exact count later). That means importing already-existing databases would duplicate what I have already for the most part. ibsearch site has a broken API, so I am scraping it slowly using a random seed (I think I have about 20% now) However, I think there are a few non-booru style sites that I don't have but there is a hydrus database for. I already have the SQL to export from the sqlite db into my format; that was no problem. I'll probably toss that data in just so I have it on my side. Anyway, since I am basically done at the data layer, and the database format of the hydrus site rips is simple, I'll just export my data into sqlite databases. I have a few options though. My engine is similar to hydrus where the tags are attached to the hash. but I think it would be better to export them as pieces; meaning have a database for 'tbib' with the tags specific to 'tbib' in that database instead of all tags from all boorus that I scraped. That way, folks could pick specific sites to import tags from; some are more 'trusted' than others. Do you have a sugestion or preference? Assume that database knowledge is no issue; I can do anything at that layer. I'll probably also kick out XML files from them all too in the standard danbooru API return format so that they can be processed in whatever systems are out there. One of the things my progaram does is generate 5 hashes per file; MD5, SHA1, SHA256, SHA384 and SHA512. I'll probably dump those too as it is a good data source to join against nearly any other process; such as IPFS or such. I need to finish downloading the unique images that are pending though (about 9 million of 15 million total). that way I can generate the other hashes and load the rest of the metadata (my program adds some metadata about the file itself) I saw this: >>1268 and I thought that was already done (I see the post is oooold). but just in case, I'll probably also dump the signed integer from my program that is used for 'not dupe, but really close' detection. Catches things like watermarks and resizes. The input is the binary image and the output, stored at the database layer, is a 32 bit signed integer that is a representation of the data stored in each pixel of an image in any format. That makes it so that finding similar images is just a database lookup across an indexed field, so all 15 million images can be searched in a few miliseconds. I'm currently upgrading the SAN this is runnig on, but once it's done and I finish an up-to-date backup, I'll start dumping stuff so it can be reviewed.

>>3223 I would like to add shortcut hooks for almost all actions, but it will have to wait for an overhaul of the whole shortcut system, as it is in a hacked-together mess and requires hardcoding for every new action. I don't know much about AutoHotkey, but it might be having problems with identifying the paste button because there are separate buttons for each service page. I also don't know if the id I can give a button is what AHK looks for, but I've given them wx.ID_COPY and wx.ID_PASTE for tomorrow's release–let me know if that lets you identify them, and I'll start a job to do more of this. I don't know if this helps your situation, but hitting enter on the manage tags tag entry box while it is empty should pass an 'apply' event. It has wx.ID_OK for the dialog, and I've given the 'close' button on the frame version wx.ID_OK as well–let me know if you see any difference. >>3224 This is an important and difficult problem and something I have extensive plans for. I'll be working on a faster dupe searching algorithm after I'm done with suggested tags control, which will be the first step. I don't have a clear idea on how the gui workflow for this stuff should work, so please give me your feedback as it rolls out! >>3229 >>3230 I can't think of a good solution to the missed files problem that doesn't hammer the destination server with mostly superfluous gallery page requests. Maybe I could have an option to do a complete resync of urls on a longer time period, like once over three months? I'd llike to improve subs a lot, including better simultaneous multiple-subs editing (to say like 'retry all failed for all these subs'. I expect to add multiple queries for single subs (so you can have twenty different tags running on the same 7 day clock or whatever with the same tag options) once the next download engine overhaul is done, which should aid this. The dialog could also do with a rewrite as well, as it crashes the whole program after viewing a large number of subs in one session. >>3242 Yes, I'd like to have this for all exports, including mouse drag-and-drop. Some sort of default phrase, like: [artist] - [series] - [unnamespaced] that'll try to autopopulate, falling back to the current ugly sha256_hash.jpg. Although maybe I have misunderstood something in your post–where are you seeing files without an extension? Files in install_dir/db/client_files should have .jpg or .swf or whatever, and that should persist on an export. >>3248 I'm impressed with what you've done! Can't have been many people in history who have had access to that much data! My suggestion about this stuff is to keep it as simple as possible, and to plan for later fixes–every time I've tried to make something clever and 'perfect', it has ended up a headache down the road. Lots of the HTA stuff needed revisits to cull badly parsed tags, or to establish or filter namespaces, so making sure the indices and the python-side access supported both read and write was important. Also reducing columns has been super important–I don't know if sqlite is just inefficient at packing its indices, but I cut something like 65% off the hydrus db size recently by removing service_id columns and having dynamically named tables for each separate service. I would suggest having 1200 different tables or database files rather than any attempt to make any sort of master ultra db of everything. I'm planning to work on better dupe search after I'm done with the current suggested tags control. Unfortunately, I've always got more stuff to do than I have time for, so big projects can take a while! My current system uses a 64-bit perceptual hash based on the DCT of the image–is that similar to yours? I search for similar images by comparing hamming distance between two hashes. Hamming distance of 5 tends to be good for my phashes, but sometimes anything up to 12 produces good results. I've been looking into using a VP-Tree to reduce the search from the current O(n) time per hash searched to I think O( log n ), although I'm still not confident on the workflow I want, and so what exact db requests I want to do. Do you search for exact bit-for-bit match on your 32-bit phash? Does that work well for you?

>>3288 >where are you seeing files without an extension? Ah, I'm talking about sharing files on the local booru, as in on the http://x.x.x.x:x/gallery?share_key?=x address. If anyone tries to save shared files from this external link, they'll all have the same name - 'file' - and no extension. Although having control over filenames on drag & drop would be another very useful addition, to be sure.

Hotkeys for ratings/likes. I bulk rate a lot. It'd be nice if there was a way to assign hotkeys to specific ratings. IE, Ctrl + 1~5 = Rating 1~5.

>>3308 Those are already somewhat supported, you'll just have to use a custom filter for now. Eventually there'll be an overhaul to make those and most other actions available for key assignment globally, though

>>3288 Yes, it sounds very similar. The duplicate detection works like this: Step 1: convert the image to 200 by 200 but keep the ratio to width and height. This happens as part of the thumbnail generation process, which is also part of the import process. So this is free from a import cost standpoint. Step 2: for every pixel, get the hashcode https://msdn.microsoft.com/library/system.object.gethashcode.aspx (an integer) and sum them all together. This is a really interesting function, but it will return the same value for the same pixel color. The reason they are all summed and 'some cool process' isn't used is becuase this will catch images that have been rotated, resized (if > 200 by 200) mirrored, etc. The hash returned for each pixel is an int (32 bits) and the result of all the sums is stored as a 64 bit integer just in case. However, I don't believe a summed hash value has ever left the 32 bit space. I will have to check to be sure. Takes about 10 ms to generate becuase the image is still resident in memory from the thumbnail generation process. It is a form of block mean value hashing, though simplistic. I went for raw speed over perfection. Unfortunately, that means XOR/Hamming distance is useless for the result, but a side effect is that means the distance is basically built into the output value. So there's no additional processing (besides a database lookup) that needs to happen. Think of it kinda like generating a fingerprint with a base value already built in, so all you need to measure (query) is the distance from the base. If 4 images are the same (or really close) to the same distance from the base, they are probably very similar. Step 3: do the same thing again, but convert the image to greyscale first. This will catch things like bloom or hue changes. It isn't as useful as I originally expected, but it only takes about 10 miliseconds to generate the hash, so I continue to generate it. Step 2 finds pretty much everything that step 3 does. The only real bonus here is if there is both a color and a greyscale version of the image out there in the wild (and is thus in the database) Step 4: Store those 2 values in the database as part of the metadata for the image. Step 5: Search. It is incredibly fast at the database layer to do a search like this. I can return matches for a specific image in 8 miliseconds even with 15 million records and a database size > 50GB. There's a few options I have on my end, which is a recordcount in either direction from the given input image, or a value difference (ex: +/- value of 10). The value difference is faster to query. Then, I just spit out the thumbnails that were in that range so they can be verified as duplciates. There isn't a super good reason that I am using the pixel hash data. My requirements where: fast at the application layer (<15 miliseconds) the output was a small value (< 100 bits) Always the same for a given input where small variations would not drive a gigantic change in output these basically require a perceptual hash. You could do something similar using a DCT perceptiual hash as the input instead of the pixel hash I am doing. Just calculate the hamming distance to a base image or value, and store that result at the database layer. Then it becomes a simple databsae lookup. The only gotcha I can think of off the top of my hed is image size needs to be standardized. So pick a size. One of the thigns I wanted to do is catch crops. Since I calcualte a hash for the entire image, if an imace is cropped, the hash I get will no longer line up even though it is a true dupe (or I would count it as one). To solve for that, I plan on splitting the image into N areas (probably 20 by 20 px, which would give a hundred areas) and hash each. Then, a cropped image would be a match to N of those areas. If all of them match, or say 80 are exact and 20 are close, it's a cropped image. I have to actually test it to make sure my thought process works, but it will be done at some point.

>>3291 Thanks–I'll look into this. There must be a header I can send to set a filename.

Could there be a way to search for japanese tags, besides tag parents/siblings? For instance, to search for the artist "にしくん" I'd like to be able to just type in "nishikun". The obvious way is to make a tag parent, but some kind of automatic system would be neat.

>>3353 Why not set up japanese input for your OS and actually type the kana instead? Seems like a much easier solution

>>3354 it probably is, yeah

Any chance to have an additional entry in the IPFS share context menu that prefixes the hash with a given web gateway, so you would end up with something like http://ipfs.io/ipfs/HASH? It would make sharing with people who don't have the daemon running a lot easier

with searching by ratings, it would be nice to be able to 'not this' a selection. ex. "anything not rated -1 favourite". essentially want a sane way of marking images that while i don't want to get rid of, also want to be able to exclude them from a search.

Some suggestions: A way to find file types without having to tag files as such, like system:gif or system:webm to find all of that certain format. Perhaps an option in user settings to select what file formats show up in the search dropdown. Be able to have each thumbnail browser tab pop out and be moved around via dragging it away from Hydrus and reattaching by doing the reverse. Be able to have Hydrus recognize an animated image/video and give it a etiher a different title bar color and/or background color (that the user can set) to identify them easier in the thumbnail browser. File → Options → Sort/Collect to add to the sort options you need to type text and hit enter. This is unintuitive to the user as it doesn't match the rest of the UI (nor is there any indication as to what a user should do here). Adding a 'add' button that the user can click on to add their sort options to the list would bring it in line with the rest of Hydrus' UI. There are some images and videos that have stories, translations, or other text that would be nice to include with the image. A way to add a description of sorts to these files would be great. Would be nice if videos could also be included in the upcoming new comic format that has been hinted as there are video series that would be nice to link together Be able to change the color of the splash screen's text. >>1848 >>1844 I would love to see something eventually come about for this. GIFs are massive space wasters when we do have better file formats available. Since webm aparently would totally mess with the hashes, I'd support an automated mp4 convert as long as it would actually reduce the file size and could carry over the tags or another alternative file format. If this couldn't be done, some kind of client file folder compression would be a nice solution too, better maybe as it would be a global reduction in the size of the user's images/videos imported into hydrus.

>>3389 >A way to find file types without having to tag files as such, like system:gif or system:webm to find all of that certain format. Perhaps an option in user settings to select what file formats show up in the search dropdown. In the tag search dropdown menu, double click on system:mime.

>>3389 >>1844 >>1848 That's a terrible idea. Hydrus should not touch the content of files at all; you should able to import and export a file without any change to it whatsoever, as is currently the case. Adding an automatic conversion or 'optimizing' system that loses the original data would be unacceptable. Not to mention, there's only a subset of GIFs that can reasonably be converted to other video formats, anyway. Anything with a transparent background is out, because even if you could use VP8 or 9 which do allow an alpha channel, there's very limited support for it; Hydrus doesn't even render it at the moment (although this seems like an easily fixable problem, but that's another discussion). And anything that needs to be pixel-perfect - that is, anything that's not film footage converted to GIF for portability - would have to be saved as lossless, which actually increases filesize, even using h.265. Now, adding an opt-in, per-file option to choose whether to convert each GIF on import is another story. It could definitely be useful for film footage clips that were made into unwieldy, huge GIFs just for the sake of posing on imageboards. However, that's a decision that has to be intelligently made by each user, for each file. I think the best way, if you had to include this sort of feature, would be to add an option to convert an animated GIF to an h.264 video, move all the tags to the new file, and optionally delete the original, and then put that option somewhere where no idiot could accidentally stumble upon it without knowing what they were doing. There are a lot of people who don't know shit about video formats, and would gleefully fuck up their entire collections if it you made it easy to do so. For any user really concerned about GIFs taking up space, I think you should just convert them on your own, rather than lobby for such a dangerous feature being included in Hydrus.

I've noticed that the options for Hentai Foundry are only artists and tags, if possible could the option to download by category be added as well?

I personally think it'd be awesome if we had a popularity tag, or something, that could record the number of times the item has been opened. It would help finding favorites and useless items, which is probably the most common issue with most unorganized people. What do you think?

>>3395 >and then put that option somewhere where no idiot could accidentally stumble upon it without knowing what they were doing. like right here It's best not have a feature to handle it within files already stored Only allow it on import and export methods therefore, data in the database is left integral you can probably also set up the tag .txt parsing to export and add files back in with tags of the same if chosen to do so

>>3353 >>3354 >>3357 Yeah, if you want to do language or unusual symbol conversion without setting tag siblings, I recommend you use an external program that'll do whatever you want in a better way than I can knock up. Something like this: http://www.lexilogos.com/keyboard/japanese.php I'm fairly confident there are some japanese tags from pixiv that have their english translations set as siblings on my PTR, so this does already work for some things. >>3358 Yeah, check out services->manage services->your ipfs service and the multihash prefix option. >>3361 Sure, I'll add this to the todo list. For now, although it is ugly, you might be able to munge this for some single large operations by adding a temporary local tag to those files and then negating the tag in a search. >>3389 Thank you for these suggestions. I hadn't thought of detaching pages–I like the idea, but I am not sure my window library, wx, can handle it. I'll make a note to try it out. I've been thinking of adding per-mime thumbnail border colour options or something, and I was just talking to someone about adding video/audio icons as well. I think I can figure this out. The sort/collect options page remains ugly and debug-tier. I have a note to clean it up, but haven't got around to it yet. I haven't thought of a good way to do larger notes added to files yet. I wouldn't mind adding this stuff, and even emulating comments, which is essentially the same metadata, but it might have to wait for the network to be in a different, more p2p place. Maybe I can add 'local only' notes before then, at least to experiment with what works. The new format should support arbitrary files, so video sequences should work. Might not work well for very large files though! Bunch of related webms should be good. I'm of two minds about mass conversion of video. I think I'm for it eventually, especially if I can pin down a guaranteed deterministic system, but I can also appreciate that hydrus isn't a conversion program and I'm no expert either, so I probably can't do as good a job as someone who knows what they are doing hashing it out with ffmpeg on the command line. I'm still thinking about it. >>3398 This will be possible after I rewrite the downloader engine, which I will start once I am done with suggested tags and a faster dupe search algorithm. >>3404 Yeah, I've been thinking about something like this. I really like the idea. I'm planning a no-limit integer rating that you left/right click to increment/decrement which would be a great backend for this sort of thing to jump on.

>>3431 >mass conversion of video Once again, I have to say, I think this is an awful idea. Just on principle, it crosses a line from Hydrus organizing and displaying images, to actually editing the content of them. If you're going to do that, why not convert every PNG to JPEG? That would save space, right? Or why not convert every video to some common-denominator h.264 profile, like YouTube does? And even if you were okay with destroying data, mass conversion means it would be easy to accidentally convert GIFs that can't be represented with MPEG video. Hydrus can't tell if a GIF is an illustration or footage, after all. There's simply no good way to do it. And actually, I'm against that sort of conversion in general. If the GIF was created from a video originally, that means it was limited to a 256 color palette, resampled, the audio was stripped, the framerate was probably fucked up, and maybe there was even a watermark thrown on. And then you convert it back, piling on MPEG artifacts, chroma subsampling, and probably some incorrect colors from repeated conversion between RGB and YUV. In the end, you save a few megabytes, but you're left with a tiny, mangled version of the original. It would better to track down the source and remake it in high quality, and if that's not possible, just suck it up and leave the GIF as it is. I understand the desire to use 'better' formats, but in cases like this, it's not actually an improvement.

>>3431 >multihash prefix option Yess, that is exactly what I needed. Thanks! I have no idea how I missed that option in the first place

>>3431 >>3434 >I think this is an awful idea Agreed. Hydrus is about archiving data. Conversion is literally destroying data- the exact opposite of hydrus' purpose. Also, feature creep; why the heck would you even add conversion to hydrus in the first place? There are plenty of programs dedicated to doing this. I'm not sure exactly how this conversation started, but it seems to have something to do with converting collections into a single file? Imo manipulating the files is the least effective way to handle this. First, because this limits you to certain file types, second because tagging subcomponents of the collection becomes a difficult problem to develop for, and a logistical clusterfuck for the user. A collection should only be a virtual construct. It's the only sane way. That being said. by all means add basic support for cbz if users want it.

>>3438 >I'm not sure exactly how this conversation started Went back and read properly. Nothing to do with that, whoops. Still, yeah, terrible idea. Why bother to hash files at all if you're just going to make certain that anything imported is run through the shredder? You'd be contributing to the degradation of overall data as users share their destroyed reaction gifs/etc back into the world. There is no benefit to this.

>>3434 >>3438 more agreement - IF we are talking about importing information. Having conversions as a potential export option, then i think that it's perfectly reasonable and puts the choice and responsibility on the user rather than hydrus.

>>3442 >Having conversions as a potential export option, then i think that it's perfectly reasonable and puts the choice and responsibility on the user rather than hydrus. ehhhhhh. If this were my project, I wouldn't. For a few reasons. 1) It sounds practical on the face of it, but it's really not when you get down to it. If you're trying to share something but it's not in a sharable format, then you're probably trying to post it to an image board or some online community. What are the chances this is the _ONLY_ time you're going to share this particular something? Slim, I'd say. You're going to want to have a share-ready version of this file on hand, in your database, so you don't have to run through a potentially lengthy conversion-export every time you use it. You ask for convert on export. Next, someone else will ask for creating an in-database child-file conversion. Then someone comes along and demands the ability to trim video content- after all, there's a converter in there, why is it incomplete? Why don't you want to make your program better, huh?! Once a feature gets its foot in the door, it becomes easy to rationalize expanding on that feature down the road. Normally this wouldn't be a problem, but *we already have no shortage of great conversion software*; there is no reason to reinvent the wheel here. 2) On the flipside, you have something in a perfectly standardized sharable format, but the service you want to share it to is from the stone age. Maybe they don't support webm/mp4, or maybe the largest filesize you are allowed to upload is 1MB. This is a specialized case- you're exporting a specially tailored file to this one site just this one time. Counter argument: urge the maintainers to enter the 21st century, or don't use that shitty site. —— Hydrus needs to clearly delineate that it is NOT a conversion program. That being said, hydrus already includes ffmpeg. If the userbase felt it were absolutely imperative (for some reason), it would be acceptable for hydrus to do what foobar2000 does: offer a generic popup that will execute an arbitrary command, substituting filenames in for variables. You can save your own presets. Pic related.

>>3443 You made some good points, I do feel that the slippery slope argument does lend itself to people saying 'but im not gunna do that', never-the-less I do agree. Conversions (as long as support exists for the format in hydrus) don't really interest me in general - and if that were to ever be and issue and i really wanted whatever it is, in hydrus, i would follow the 'tried and true' applications to do so. The idea of having a popup to generate an arbitrary command though, I think that actually has a lot of room to play - and may placate those that do want conversions, aswell as being generically useful. It would really depend on how it was worked into the UI how useful it would be in actual use.

Can we get an option to make the tag search box assume that any tags you select are going to be added instead of toggled? I can't think of a single situation where I searched for a tag in there with the intention of removing it, if I wanted to remove it the tag is in the list above anyway so why go through the trouble of searching it? What I *have* done, unintentionally, is remove tags by adding them twice and not spotting the tag already in the list, so they get toggled off. If this is already an option I'm not seeing it…

>>3458 Thanks, I want to add this in a future update to manage tags. I want to wrap the current checkboxes and any other boolean state variables like 'only add' into a single gear icon that spawns a checkbox menu.

Under advanced operations for the manage tags dialog it would be nice to have options to import and export tag archives only for the selected files instead of globally.

Any chance of implementing a folder monitor feature? Because I searched around and couldn't find it if it's already a thing. EIther that, or a keybind for adding single urls, because it's troublesome every time i want a single pic from a thread to do all the steps to add it to the program. Currently the F9 keybind doesn't show the option to download a raw url.

>>3492 Thank you for this report. This was unintended. I've fixed hta import to only work for the files manage tags was launched with and updated the 'are you sure' text to reflect this better. I think the hta export is filtering ok already. Let me know if this gives you any more trouble in v219. >>3493 Check file->manage import folders, which will suck up anything in an imageboard download folder nicely. Also, since you reminded me how awkward it is, I've made a job to convert the raw url download, which was originally just a debug thing I threw together, into a proper download page with a queue and multiple paste and so on. It'll actually be a very simple job, so it might be ready pretty quick. Once it is out, have a play with it and let me know if it breaks anywhere for you.

>>3494 >Thank you for this report. This was unintended. I've fixed hta import to only work for the files manage tags was launched with and updated the 'are you sure' text to reflect this better. I think the hta export is filtering ok already. Neat. I thought you added that as an easier way to (globally) apply HTAs and didn't know it was a bug. Thanks.

Some more suggestions: A random/shuffle option for slideshows that will randomly display an image from the selected files while showing the slideshow. An option for deleted files to be moved to a user specified folder. A tag blacklist mechanic that allows a user to set tags that they do not want the downloader to download. Some booru's have limits on how many tags can be used in a search and this would allow for more filtering of subscriptions / gallery downloads. It would require the download part of Hydrus to parse the tags of the files it tries to grab and then compare to the blacklist before downloading the file itself. A way for us to manually input source URLs for files and edit them once they have been added. Expanding on the source URLs concept, it would be neat if they could be included in the PTR data so that everyone can contribute to finding sources for the images. I would imagine this would work very similar to how tags do now with the PTR, you can add or edit the sources for images and they get uploaded and downloaded just like tags. This would allow for people to have multiple sources for images should they ever need to find them again. A QoL addition for the tag list where if you click on a tag in the list and then press any letter it will jump the list to the first tag that matches the letter. It would make finding tags much easier than having to manually scroll through the entire list to find if a tag is attached to a file. An option for duplicate/similar file search to be performed automatically for each imported file. Some booru's do have different versions of the same file and, especially with subscriptions, it's easy to get multiple copies of the same material. In order for it not to be a pain to the end user, maybe Hydrus could pop up a list of imported that had similar matches at the end of the import process giving users a chance to manually investigate the potential duplicates with the current similar file search feature. Alternatively, a more robust duplicate found screen could be created that acts / looks similar to the thumbnail gallery and shows the imported file along with its suspected similar files highlighted in different colors.

This might not be possible until the new downloader overhaul, but it would be amazing if we could have multiple subscriptions downloading at once. Perhaps a hard limit could be in place so only one (2?) subscription(s) per website/booru, to prevent any anti-scraping / mass-downloading bans. Does the downloader skip a download if it detects that it Hydrus has already downloaded that file via source URL? If not, could that be a feature? Some subscriptions could overlap files and it would save everyone's bandwidth if we Hydrus was able to skip previously downloaded files.

I know one of the other anons requested this feature over a year ago regarding booru annotations, but I was just wondering if it was possible for it to be put back on the table, devbro?

>>3503 I asked that question previously, and he said that Hydrus skips files that matches the md5 hash it has stored in its database. This goes out the window if the image you want to download is slightly different than the one you think you already have though (different res, upscaled, downscaled, zoomed, etc).

>>3505 Hmm, that is good to know and does help a bit to keep duplicates from being imported. Though this sounds like it doesn't save any bandwidth for anyone (the booru, or you). I assume Hydrus still downloads the full file and only when it attempts to import would it discover the matching hash and then deletes the duplicate file? My suggestion would be one possible way to avoid the downloading stage entirely, thus saving everyone time and bandwidth. However, if Hydrus is able to get the md5 before it downloads the full file to your computer (i.e. it's somehow able to get a md5 from the image on the booru's server) and is able to skip duplicate files that way then my suggestion would be moot as this is a much better way of doing it.

When you zoom in filter mode (the one where you decide whether to archive or delete a file with mouse click) there's no way to move around to see other parts of the picture besides the center. It's a bother for stuff like thread screencaps and comic strips. Would be swell to have a way to zoom to other parts of the picture or have a key modifier that lets you drag the view with the mouse.

>>3507 Also, I take that html folder download is not supported by Hydrus? I have a large folder of pictures in my server and it'd be easier to just make hydrus download them than download the folder manuall and hen importing it in hydrus.

>>3507 Please disregard this post, I'm a retard.

>>3506 From my experience trying to scrape multiple boorus these past two months, I've had on occasion needed to make multiple passes through the same sections of the site (accidents, miscounts, UI crashing, etc). And what I've noticed is that the "already in db" counter would increment an entire order of magnitude faster than it would take downloading a completely fresh file. While I dont doubt that during this process I would be consuming bandwidth that could have been saved had I not made another pass, but it is only a fraction of the bandwidth spent downloading the file.

>>3506 >>3506 From my experience trying to scrape multiple boorus these past two months, I've had on occasion needed to make multiple passes through the same sections of the site (accidents, miscounts, UI crashing, etc). And what I've noticed is that the "already in db" counter would increment an entire order of magnitude faster than it would take downloading a completely fresh file. While I dont doubt that during this process I would be consuming bandwidth that could have been saved had I not made another pass, but it is only a fraction of the bandwidth spent downloading the file.

Could there be an option to re-open the last page closed? I tend to do big import jobs of 300-500 files and have to use "find similar" to consolidate tags from multiple copies of the same image and I sometimes accidentally wind up closing the import page since right clicking on "close page" counts as a normal click.

>>3519 That's what "undo" next to "file in the menu bar is for.

Any chance that you could add tab protection? I use a lot of tabs when I import from multiple sources, and sometimes when I am distracted I accidentally close them. And if I close the client after that without realizing it, they're lost forever because "undo" is not saved between sessions. I was thinking of something like, for example, adding "protect tab" to the right click menu so that said tab cannot be closed.

I second a twitter scraper

I would welcome a Nicoseiga Gallery Downloader (seiga.nicovideo.jp/). It's the second most popular Japanese anime art website (after pixiv) and there's a lot of exclusive art that doesn't make it to the boorus.

I'd like to be able to use hydrus for organizing my bookmarks or for any other kind of textual data. Instead of generating tiles of images, a query of textual data would generate tables similar to windows explorer details view.

>>3541 I second this. Kind of like for PDFs. I personally save a lot of stuff like greentexts, copypastas and monstergirl fanfics in txt form so it'd be swell to have them inside hydrus.

Check/Download Spacing? This might be complex depending on how hydrus is coded, but what about a sort of global or site-based checking and download scheduler? For instance, ten tabs are scheduled to check threads or pages every 20 minutes and were set up about 20 minutes apart so they check close together, but you don't want to be autob& from some picky booru so you go in your site check/dl settings and set minimum check interval to 90 seconds, that is, any tab (or subscription) that checks that booru sets a cooldown timer that stops other checks from happening for 90 seconds. So when one tries it doesn't get rejected and try again next time its interval comes, it just gets paused (for checking, not downloading) and held in a check queue until the cooldown expires, so you might have a few stack up but they go through after a delay and reset their own timer from there, so in future iterations they'll all be spaced apart correctly to begin with and it won't need to queue again ideally unless something new is introduced. Same could be done globally instead of site-or-tool-based, and with downloads as well as checks. Also, if it's not implemented yet, a global download rate limit and max simultaneous downloads would be very desirable, like jDown or torrent clients have. Eventually, even a customizable scheduler like a few torrent clients have would be great too. The rate throttling wouldn't have to be exact, just a rough and dirty indicator. Rate limit/simultaneous dl limit by website/domain instead of/in addition to global would be great too, for not getting kicked off of a booru or Tumblr/DA while running multiple collect tabs on the site. Of course, even with the check intervals correcting themselves after the first queuing, setting too much of a spacer could still sabotage quick-rechecking tabs for a while, but hydrus relies on basic sanity checking by the user to begin with so it wouldn't necessarily be so bad.