/hydrus/ - Hydrus Network

>>14944

Same anon, I can't grab stuff from Gelbooru pools. It's as if Hydrus thinks the pool URL contains no image links or next page links whatsoever. The status displays as "working" for a second, then "DONE".

An example pool URL to test with: https://gelbooru.com/index.php?page=pool&s=show&id=45305

>>14943

Thank you for these reports. I am sorry for the trouble. PTR processing is running too slow with the new virtual siblings and parents, and I will be working on it more in the coming weeks to get things faster again, and using fewer system resources. We just went over a billion tags, and things are starting to creak.

The autocomplete sounds far too slow, though. Thank you for the details here about media_predicates, I will investigate this. It would help me if you try to do this operation while help->debug->profile modes->db profile mode is turned on. That will make a new 'profile' log in your install_dir/db directory. If you can pastebin that here or otherwise email/discord DM it to me, I can check it out. It shouldn't have any private info in it, but feel free to check. I'm interested in the info below the 'media_predicates' call, if you want to clip the file.

And please let me know how future versions go for you. I don't want this to be normal, and I am confident I can improve things.

>>14944

>>14945

Hey, I am sorry, some Linux flavours have problems with the libmpv I bundle with my Ubuntu. You may have luck using your system libmpv, as this guy just did: >>14936

If that works ok, you can change the setting for animation and video under options->media from the native viewer to mpv, and I expect these will all fix immediately.

When I load that gif in the native viewer, I am afraid I get the same problem. Thank you for the example file, I will put it in my 'weird files' collection to see if I can fix the bugs on my end.

For pools, I also get the same problem. My guess is gelbooru changed their format. Thank you, I will check with the guys who make downloaders to see if we can figure out a fix.

Ever since the upgrade to macOS Big Sur 11.0.*, Hydrus (I upgraded to 419 and was on 415 or something before that) launches but hangs while taking 100% of a CPU core. I thought it might have been a DB update or something so I left it alone, but it eventually crashes. No GUI elements appear. I can't post the forcequit report here because of length limits, but here's a paste of the heaviest stack, taken from the report info collected after a force-quit: https://ghostbin.co/paste/afk2

>>14958

Thank you, this is helpful. Current main plan to fix this is to move from building macOS release on my old 10.12 laptop to online github workflows or whatever it is called, when I commit a new release. Some users are exploring the different ways to do it, with luck I'll be using the working scripts before the end of the year and we'll be building for 10.15 (or equally more-compatible 10.14 or something) on release day and this will all clear up.

If that doesn't work out for whatever reason, or it is delayed, or the 100% CPU continues even with newer built libraries, I will dive deeper into this precise problem. I had first thought that Big Sur wasn't happy with loading my Qt's older .so files, since that is usually the sort of problem when this thing happens, but yeah it actually boots the splash screen and gets stuck here, which means core Qt and QApplication and so on are OK. Also, apparently running in Big Sur from source is fine, so it likely isn't a problem in my code.

Best quick guess looking at your stack is this is it either blitting an updated bmp to the splash, or it is drawing a custom shape, maybe one of my icon bmps being initialised, and it is there that it is running into a .so file version incompatibility. Not sure why it would look 100% rather than throw an exception, but it might be stuck in an infinite paintevent or something.

Fingers crossed, this new build process will fix things up. Any more feedback you can provide as we roll out solutions here would be great.

On v420, just started getting errors like this:

OperationalError: no such table: mem.temp_int_tag_id_0… (Copy note to see full error)

Traceback (most recent call last):

File "hydrus\client\importing\ClientImportFileSeeds.py", line 1350, in WorkOnURL

did_substantial_work |= self.WriteContentUpdates( tag_import_options )

File "hydrus\client\importing\ClientImportFileSeeds.py", line 1464, in WriteContentUpdates

for ( service_key, content_updates ) in tag_import_options.GetServiceKeysToContentUpdates( self.status, media_result, set( self._tags ), external_filterable_tags = self._external_filterable_tags, external_additional_service_keys_to_tags = self._external_additional_service_keys_to_tags ).items():

File "hydrus\client\importing\ClientImportOptions.py", line 1386, in GetServiceKeysToContentUpdates

service_tags = service_tag_import_options.GetTags( service_key, status, media_result, service_filterable_tags, service_additional_tags )

File "hydrus\client\importing\ClientImportOptions.py", line 1713, in GetTags

existing_applicable_tags = HG.client_controller.Read( 'filter_existing_tags', service_key, applicable_tags )

File "hydrus\core\HydrusController.py", line 615, in Read

return self._Read( action, *args, **kwargs )

File "hydrus\core\HydrusController.py", line 194, in _Read

result = self.db.Read( action, *args, **kwargs )

File "hydrus\core\HydrusDB.py", line 1074, in Read

return job.GetResult()

File "hydrus\core\HydrusData.py", line 1843, in GetResult

raise e

hydrus.core.HydrusExceptions.DBException: OperationalError: no such table: mem.temp_int_tag_id_0

Database Traceback (most recent call last):

File "hydrus\core\HydrusDB.py", line 675, in _ProcessJob

result = self._Read( action, *args, **kwargs )

File "hydrus\client\ClientDB.py", line 14838, in _Read

elif action == 'filter_existing_tags': result = self._FilterExistingTags( *args, **kwargs )

File "hydrus\client\ClientDB.py", line 6839, in _FilterExistingTags

with HydrusDB.TemporaryIntegerTable( self._c, tag_ids, 'tag_id' ) as temp_tag_id_table_name:

File "hydrus\core\HydrusDB.py", line 1186, in enter

self._cursor.executemany( 'INSERT INTO {} ( {} ) VALUES ( ? );'.format( self._table_name, self._column_name ), ( ( i, ) for i in self._integer_iterable ) )

sqlite3.OperationalError: no such table: mem.temp_int_tag_id_0

Happens when I attempt to paste or drop a url for an image into the client. Oddly, despite the image not popping up into the url import tab, the images still go through and come up into the inbox.

>>14955

Thanks for the tips. I tried installing libmpv1 and it did successfully install and change the behaviour of the Hydrus preview, unfortunately the behaviour is still buggy. It does not display sheared frames any more, but it seems to skip the frames instead. I can see it go from frame 14 to frame 17 (lingering a bit on frame 14) and after frame 18 it skips all of the remaining frames (there are 26 in total).

Thanks for the incoming fix for Gelbooru pools.

>>14967

Damn, thank you for this report. I will look into why this is happening, it is related to my new temp table cache.

There is another issue this week related to problems importing files while the 'only add existing tags' filter is on.

This stuff is a pain, it should be ok to rollback to 419 this week, I am sorry for the trouble.

>>14972

"Exact match" doesn't necessarily mean that, it just means they are very similar. It's just math. A censor bar might not be enough difference to make the numbers go up enough for it to not be considered an exact match.

"Reset potential duplicates" is meant to be used to reset matches when changing from a higher search distance to a lower one. So it doesn't make sense to search again files that are already marked as not having potential duplicates.

Since (I think) the last version, v419, the namespace:* search doesn't bring up suggestions for every tag in that namespace anymore. It's just empty, with no suggestions now until I type a letter instead of *. I have the option set to on, but it still doesn't work.

I'm running the Linux version.

I'm on version 420 and currently the twitter/nitter downloader currently fails on videos saying "Count not find a file or post URL to download!".

Are there plans to fix this? IMO one of the major advantages of hydrus is to download an artist's twitter. It's easy/simple to browse and find what you want on e621 or even fur affinity where things have tags etc. but it is a nightmare on twitter having to just scroll through a lazy-loading timeline full of retweets and other shit.

I'm guessing they're just streaming the video using DASH or something rather than an easy to grab URL. This is probably out of scope for a downloader, but youtube-dl can grab twitter videos and gets packaged with everything else. That might be a solution?

Exception

Problem while trying to fetch External IP:

Traceback (most recent call last):

File "hydrus\client\gui\ClientGUIDialogs.py", line 373, in EventCopyExternalShareURL

url = self._service.GetExternalShareURL( self._share_key )

File "hydrus\client\ClientServices.py", line 473, in GetExternalShareURL

host = HydrusNATPunch.GetExternalIP()

File "hydrus\core\HydrusNATPunch.py", line 50, in GetExternalIP

raise Exception( 'Problem while trying to fetch External IP:' + os.linesep * 2 + str( stderr ) )

Exception: Problem while trying to fetch External IP:

No IGD UPnP Device found on the network !

No IGD UPnP Device found on the network !

It gave me this error when I tried to set up a local booru wih external ip. How do I fix it?

>>14980

iirc the youtube-dl solution is planned, but in the mean time there is another parser that will fetch video through a third-party website in hydrus already. You just have to change your parser links as it isn't linked to it by default.

>>12890

This doesn't exist anymore.

>>14974

Thank you, this is great. I am really sorry for the shit here, it looks like it was lagging twelve minutes. All of that time was spend doing one query, and it was not where I expected, so this profile helps a lot.

It looks like there were 65,000 tags in the page that needed to be populated, with the lookup done twice, and 5ms per row is crazy slow, so something is going wrong here. I will examine this further, and I can think of a better way of doing it as well. Please let me know how 422 works for you.

>>14979

Thank you for this report. I am sorry, this got recently broken. I am going to fix it this week and write some unit tests for the new cache that is failing here so these logical typos do not slip through again.

>>14987

Hydrus uses a program in install_dir/bin to try to talk to your router to get UPnP information, which includes external IP. If your router does not respond to the UPnP request, or you are on a VPN of some sort so you can't talk to your gateway at all, this happens. I will brush up the english of this error.

>>14989

It is now on options->gui pages, right at the bottom, 'hide the bottom-left preview window'.

This file caused the memory usage to spike up to 11 gigs when I came across it in the archive/delete filter.

https://gelbooru.com/index.php?page=post&s=view&id=5735585&tags=paintrfiend

>>14996

I was about 200 in of the 1100 total, if that's important.

>>14988

So I found the 'manage parsers' screen and I assume what I'm interested in is the "nitter tweet parser (video from koto.reisen)". I went to edit it but I'm not really sure what to do from here? Could you elaborate a bit?

Also is there a way to sort by 'source time'? I imagine there must be because I can sort by everything from resolution ratio, to import time, to number of tags which all seem less useful than when a file was posted.

>>14998

That is the right parser to use, but you want to change to it in the 'manage url class links' menu.>>14998

>>14996

>>14997

These gigantic pngs are a pain. I am not really sure what to do about them. Unfortunately, it is one of my image libraries needing most of that space, likely OpenCV, possibly Pillow.

A long time ago, some clever guys constructed large malicious pngs with extremely inefficient pixels/palettes/whatever, and they were able to crash people's machines as the decompression routine of loading the png suddenly ate up all the memory of the machine and crashed it. These were called 'decompression bombs'. We now get this effect naturally when patreon artists sperg out.

The good news is memory usage goes down quickly once the image is rendered, although 11GB is larger than I have heard of before. Even then, just having that image at 100% in memory is a 750MB bitmap.

Although I can detect the size of a png, being able to detect how efficiently it will unpack is not something (or my libraries) seem to know how to do. Perhaps in future they will be handled better, or I'll find a new library or a new way of rendering just part of an image at a time so I don't need 100% to get the scaled version, but for now I do not have great answers.

I will save that png though as a great example of something that will probably kill my 8GB dev machine. I am sure there is extra inefficiency in my loading routines, so I will be able to examine what is going on easily with it.

>>15000

Follow-up: If the problem is your database does not have the required indices for some reason (my best guess here atm, either due to hard drive failure or a weird update logic problem), I have fixed this for 422. The database now automatically checks if those indices (and some others) are missing on every boot and recreates them if so.

>>15003

My problem seems to be solved.

'%siblings%' in my DB lookes nothing alike compared to yours (sorry if I fail with the formatting):

sqlite> .open client.db
sqlite>
sqlite> select name from sqlite_master where name like '%siblings%';
tag_siblings
sqlite_autoindex_tag_siblings_1
tag_siblings_service_id_good_tag_id_index
sqlite>

So I started suspecting that I miss some indexes/caches/etc, and once again performed database→regenerate→* from menu, except "clear service info cache". Since then autocompletion works perfectly. '%siblings%' list however didn't changed. I also was unable to perform explain query:

sqlite>sqlite> explain query plan SELECT 1 FROM sqlite_autoindex_tag_siblings_1 where bad_tag_id = 1 or ideal_tag_id = 1;
Error: no such table: sqlite_autoindex_tag_siblings_1
sqlite>
sqlite> explain query plan SELECT 1 FROM tag_siblings_service_id_good_tag_id_index where bad_tag_id = 1 or ideal_tag_id = 1;
Error: no such table: tag_siblings_service_id_good_tag_id_index
sqlite>
sqlite> explain query plan SELECT 1 FROM tag_siblings where bad_tag_id = 1 or ideal_tag_id = 1;
Error: no such column: ideal_tag_id
sqlite>

For the record, here is my tables list:

sqlite> .tables
alternate_file_group_members       local_file_deletion_reasons      
alternate_file_groups              local_ratings                    
analyze_timestamps                 options                          
client_files_locations             potential_duplicate_pairs        
confirmed_alternate_pairs          recent_tags                      
current_files                      remote_thumbnails                
deleted_files                      repository_updates_11            
duplicate_false_positives          service_directories              
duplicate_file_members             service_directory_file_map       
duplicate_files                    service_filenames                
file_inbox                         service_info                     
file_modified_timestamps           services                         
file_notes                         statuses                         
file_petitions                     tag_parent_application           
file_transfers                     tag_parent_petitions             
file_viewing_stats                 tag_parents                      
files_info                         tag_sibling_application          
ideal_client_files_locations       tag_sibling_petitions            
ideal_thumbnail_override_location  tag_siblings                     
json_dict                          url_map                          
json_dumps                         vacuum_timestamps                
json_dumps_named                   version                          
last_shutdown_work_time            yaml_dumps                       
sqlite> 

Once again thanks for your help.

>>15005

Sorry, it had to be client.caches.db, not client.db, when you did the first '.open' line. No worries.

Since you ran the regenerate and that fixed it, that I think suggests this was an indexing issue, which is automatically fixed for all other users going forward tomorrow.

If you are interested, the slow bit for you was the 'is this tag involved in the siblings system?' method, which is part of when the client needs to figure out which siblings your autocomplete results (generated from thumbs) need to have so it can swap in typed text for a real result based on sibling mappings. This thing basically says: 'for every tag, is it either a bad tag or is it an ideal tag in the fast(lmao) siblings table?'. To speed it up, I have both the 'bad' column and 'ideal' column indexed. For some reason that I am not sure about, you did not have the 'ideal' column indexed, which meant for every one of those 65,000 tags, rather than being able to quickly test the second half of the OR, a few iterations into a search tree, it instead had to iterate the entire table for every negative result. Assuming you sync with the PTR or a similar big-sibling-number service, that probably meant something like 63,000 * 100,000 row fetches, or several billion loops on a handful of kilobytes in memory (six minutes), rather than a couple hundred thousand (about 50ms or so).

You may not be completely fixed, however. Since you have done the regenerate, your sibling tables were emptied, so if the problem was not in fact a missing index, you may see bad performance creep back in as your sync progresses under tags->sibling/parent sync->review. Let me know if this happens. If this has fixed you long-term, thank you for your feedback and help.

>>14999

So I changed the parser to the koto.reisen one in the manage class links screen for 'nitter timeline' and 'nitter tweet' but it doesn't seem to have done anything. Is there something else I need to change or is this just broken for now?

>>15009

just change for 'nitter tweet' URL class is needed, other nitter should be default.

>>15011

I tried this and it got some of the videos, I think the failing posts are because they have links that have since died. Either way thank you for the help.

>>14955

>For pools, I also get the same problem. My guess is gelbooru changed their format. Thank you, I will check with the guys who make downloaders to see if we can figure out a fix.

Wanted to follow up on this, has the fix been made? I assume I am supposed to be looking here for it https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Downloaders/Gelbooru and since the last non-readme change was six months ago, I gather that it's not fixed.

>>14960

This problem still occurs with version 423, by the way. Hydrus launches, spawns no GUI, and shoots a CPU core to 100% indefinitely.

I'm having trouble, I go to services > review services > remote, and it shows the PTR is paused. I click to resume, and it starts processing requests. I leave it overnight, then come back and it shows a notification in Hydrus stating:

Failed to process updates for the public tag repository repository! The error follows:

DBException

TagSizeException: Received a zero-length tag!

Traceback (most recent call last):

File "hydrus\client\ClientServices.py", line 1724, in _SyncProcessUpdates

num_rows_done = HG.client_controller.WriteSynchronous( 'process_repository_definitions', self._service_key, definition_hash, iterator_dict, job_key, work_time )

File "hydrus\core\HydrusController.py", line 852, in WriteSynchronous

return self._Write( action, True, *args, **kwargs )

File "hydrus\core\HydrusController.py", line 237, in _Write

result = self.db.Write( action, synchronous, *args, **kwargs )

File "hydrus\core\HydrusDB.py", line 1100, in Write

if synchronous: return job.GetResult()

File "hydrus\core\HydrusData.py", line 1843, in GetResult

raise e

hydrus.core.HydrusExceptions.DBException: TagSizeException: Received a zero-length tag!

Database Traceback (most recent call last):

File "hydrus\core\HydrusDB.py", line 679, in _ProcessJob

result = self._Write( action, *args, **kwargs )

File "hydrus\client\ClientDB.py", line 19772, in _Write

elif action == 'process_repository_definitions': result = self._ProcessRepositoryDefinitions( *args, **kwargs )

File "hydrus\client\ClientDB.py", line 14780, in _ProcessRepositoryDefinitions

tag_id = self._GetTagId( tag )

File "hydrus\client\ClientDB.py", line 11320, in _GetTagId

HydrusTags.CheckTagNotEmpty( tag )

File "hydrus\core\HydrusTags.py", line 181, in CheckTagNotEmpty

raise HydrusExceptions.TagSizeException( 'Received a zero-length tag!' )

hydrus.core.HydrusExceptions.TagSizeException: Received a zero-length tag!

Database Traceback (most recent call last):

File "hydrus\core\HydrusDB.py", line 679, in _ProcessJob

result = self._Write( action, *args, **kwargs )

File "hydrus\client\ClientDB.py", line 19772, in _Write

elif action == 'process_repository_definitions': result = self._ProcessRepositoryDefinitions( *args, **kwargs )

File "hydrus\client\ClientDB.py", line 14780, in _ProcessRepositoryDefinitions

tag_id = self._GetTagId( tag )

File "hydrus\client\ClientDB.py", line 11320, in _GetTagId

HydrusTags.CheckTagNotEmpty( tag )

File "hydrus\core\HydrusTags.py", line 181, in CheckTagNotEmpty

raise HydrusExceptions.TagSizeException( 'Received a zero-length tag!' )

hydrus.core.HydrusExceptions.TagSizeException: Received a zero-length tag!

Then under services, it shows the PTR is paused again.

Not sure what's going on. I will try updating to the latest version right now though. I'm currently on 420, network version 19. Windows 10.

>>15076

Btw, it looks like it did run through the repo processing because I reached the bandwidth limit.

I am working on updating because my repo is outdated.

I just updated my Hydrus though to the latest, I will let it run tomorrow after the bandwidth cap resets, and see if I still get any errors.

Pixiv artist lookup seems to be broken as it uses the old url style that no longer works

>>15104

Try deleting pixiv url classes, parsers, and GUG's then add the defaults but only add the ones with "api" in the names and then make sure the url class links are linked correctly.

Hey, I am sorry, the holiday ended up killing my schedule, and I couldn't give this thread proper attention.

8kun are now saying the expect to go off the clearnet in the near future and go TOR only. I have been wanting to move to a different board for a while, I was just procrastinating, so this seems like the time to do it. Endchan will be our primary board for the time being. I am going to clear this thread of newer posts, make an archive of it to post on Endchan, and then make a sticky announcing I will delete the board next week.

>>15054

Ah, I am sorry, I am not sure what the answer back was. I just talked to the guys again now. They aren't hosting copies of what I bundle by default, only downloaders that can do different sites or a whole bunch more.

It appears pools now have slightly different format for some users. Pools still works for those guys, but they are logged in, I am not. I will fix this myself this week, since I get the problem, and roll it into the update. Sorry for the trouble!

>>15057

Unfortunately the problem seems to be with PyInstaller, not my code. A couple of guys have been working on a different build method here:

https://github.com/ReAnzu/hydrus/actions/runs/452850300

I hope to catch up with their work when it is finished and working and fold it into my build too.

>>15076

>>15080

Hey, I am sorry for the trouble here. Thank you for the report. This should be fixed in the latest versions. Please let me know if you still have any problems.

>>15117

>>15118

I will be folding this into 426 btw.

>>15125

Thank you for this report. I am sorry for the continuing trouble. If searches are slow when you are searching a page of results, but fast when you search an empty page (or one with 'searching immediately' off, which then searches the database, rather than the tags for the files in front of you), then the routine that is generating search results from thumbnails is slow.

Some more feedback would be very helpful. Please run 'db profile mode' and do some slow autocomplete searches with files in front of you, and then pastebin or email me that profile log. It is the 'media_predicates' routine that is running slow for you.

The >>15000 query on client.caches.db would be helpful. Let's see what SQLite wants to be doing here.

This thread is now archived at https://archive.is/cq5Tc . I will not post here any more, as the board will be deleted next Wednesday. Please move to the Endchan thread at https://endchan.org/hydrus/res/9.html , thank you!

>>14683

>If you do have it, is MPV the player under options->media? If your hydrus does not have libmpv access, or is failing to import it, it will fall back to my native viewer, which has no audio support.

Fixed. Thanks a lot.

I'm new to Hydrus. Using Linux MX.

Before I was using Tagspaces which is a sluggish piece of trash. Looking for a more suitable alternative I bumped into Hydrus; so far I'm very pleased by its performance and rich settings.

>>15118

Is there a way to get the blacklist back to working? There's usually a cookie for you blacklist on gelbooru but for some reason the downloader doesn't use it and stopped working for me a month or so ago. I can't remember if it was something I did or if the site changed or what.

The plan to move to Endchan only has changed!

Codexx on 8chan.moe has kindly offered to host me on /t/ over there. As I have never been a comfortable BO, this is a much better solution for me as the primary place to talk Anon about hydrus.

I will be maintaining a Hydrus Network General, the first instance is here: https://8chan.moe/t/res/2219.html

The links in the release tomorrow will reflect this. Endchan will remain as a bunker.

>>15141

If you happen to see this in the next 18 hours or so, I think you want this: https://github.com/CuddleBear92/Hydrus-Presets-and-Scripts/tree/master/Downloaders/Gelbooru

test

>>11542

join