ReadFullDatabase: Invalid packet Format: Storage.db3

[Alex Atkin UK]

I just got ERROR ReadFullDatabase: Invalid packet Format: Storage.db3 and all downloads halted. Any idea what I should do about it?

I do have extensive headers as I tend to search old pictures newsgroups so want to go back full retention. Its not the end of the world if I have to download all headers again but it would be somewhat annoying if I lost records of what I have downloaded already.

If its relevant I'm on 6.90RC1 Build 5424.

[Quade]

I'm not completely sure. There's an issue in RC1 where it won't read old header formats but, I don't believe it would give that specific error.

RC2 will be up soon and you can try it. Headers changed from a 2 file setup to a one file and RC1 only knows how to read the new one (changed some time ago).

[Alex Atkin UK]

Oh I found the log files, is there any way the view the DEBUG entries in the application?

After closing and reopening the application:

Code: Select all

[08/24 20:31:05]  DEBUG Periodic Search:  Executing!
[08/24 20:31:05]  DEBUG Periodic: 0 Watch List Items found
[08/24 20:31:05]  DEBUG Periodic Search: Waiting for next Period 1800 Secs
[08/24 20:31:15]  DEBUG Periodic Search: Waiting for next Period 1790 Secs
[08/24 20:31:23]  DEBUG NZB File Load Checking
[08/24 20:31:25]  DEBUG Periodic Search: Waiting for next Period 1780 Secs
[08/24 20:31:35]  DEBUG Periodic Search: Waiting for next Period 1770 Secs
[08/24 20:31:43]  DEBUG NZB File Load Checking
[08/24 20:31:45]  DEBUG Periodic Search: Waiting for next Period 1760 Secs
[08/24 20:31:49]  ERROR Downloader:  Socket Exception NEWS SERVER ERROR: Tweaknews Unable to Connect to news server  Tweaknews
[08/24 20:31:55]  DEBUG Periodic Search: Waiting for next Period 1750 Secs
[08/24 20:32:03]  DEBUG NZB File Load Checking
[08/24 20:32:05]  DEBUG Periodic Search: Waiting for next Period 1740 Secs
[08/24 20:32:15]  DEBUG Periodic Search: Waiting for next Period 1730 Secs
[08/24 20:32:23]  DEBUG NZB File Load Checking
[08/24 20:32:25]  DEBUG Periodic Search: Waiting for next Period 1720 Secs
[08/24 20:32:35]  DEBUG Periodic Search: Waiting for next Period 1710 Secs
[08/24 20:32:43]  DEBUG NZB File Load Checking
[08/24 20:32:45]  DEBUG Periodic Search: Waiting for next Period 1700 Secs
[08/24 20:32:55]  DEBUG Periodic Search: Waiting for next Period 1690 Secs
[08/24 20:33:03]  DEBUG NZB File Load Checking
[08/24 20:33:05]  DEBUG Periodic Search: Waiting for next Period 1680 Secs
[08/24 20:33:15]  DEBUG Periodic Search: Waiting for next Period 1670 Secs
[08/24 20:33:23]  DEBUG NZB File Load Checking
[08/24 20:33:25]  DEBUG Periodic Search: Waiting for next Period 1660 Secs
[08/24 20:33:35]  DEBUG Periodic Search: Waiting for next Period 1650 Secs
[08/24 20:33:43]  DEBUG NZB File Load Checking
[08/24 20:33:45]  DEBUG Periodic Search: Waiting for next Period 1640 Secs
[08/24 20:33:55]  DEBUG Periodic Search: Waiting for next Period 1630 Secs
[08/24 20:34:03]  DEBUG NZB File Load Checking
[08/24 20:34:05]  DEBUG Periodic Search: Waiting for next Period 1620 Secs
[08/24 20:34:15]  DEBUG Periodic Search: Waiting for next Period 1610 Secs
[08/24 20:34:23]  DEBUG NZB File Load Checking
[08/24 20:34:25]  DEBUG Periodic Search: Waiting for next Period 1600 Secs
[08/24 20:34:35]  DEBUG Periodic Search: Waiting for next Period 1590 Secs
[08/24 20:34:43]  DEBUG NZB File Load Checking
[08/24 20:34:45]  DEBUG Periodic Search: Waiting for next Period 1580 Secs
[08/24 20:34:55]  DEBUG Periodic Search: Waiting for next Period 1570 Secs
[08/24 20:35:03]  DEBUG NZB File Load Checking
[08/24 20:35:05]  DEBUG Periodic Search: Waiting for next Period 1560 Secs
[08/24 20:35:15]  DEBUG Periodic Search: Waiting for next Period 1550 Secs
[08/24 20:35:23]  DEBUG NZB File Load Checking
[08/24 20:35:25]  DEBUG Periodic Search: Waiting for next Period 1540 Secs
[08/24 20:35:35]  DEBUG Periodic Search: Waiting for next Period 1530 Secs
[08/24 20:35:43]  DEBUG NZB File Load Checking
[08/24 20:35:45]  DEBUG Periodic Search: Waiting for next Period 1520 Secs
[08/24 20:35:55]  DEBUG Periodic Search: Waiting for next Period 1510 Secs
[08/24 20:36:03]  DEBUG NZB File Load Checking
[08/24 20:36:05]  DEBUG Periodic Search: Waiting for next Period 1500 Secs
[08/24 20:36:07]  DEBUG Pausing Downloaders

On attempting to update all headers, had to close and reopen the application as it was stuck paused:

Code: Select all

[08/24 21:12:15]  DEBUG CNNTPDbImportThread -Ninja-nl-000003810968-000004010968.txt
[08/24 21:12:15]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:16]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:16]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:16]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:16]  DEBUG Periodic Search: Waiting for next Period 1740 Secs
[08/24 21:12:17]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:17]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:18]  DEBUG Feeder: Stalling Disk Reader....
[08/24 21:12:18]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:18]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:18]  DEBUG CNNTPDbProcessingThread - Importing Compacts Finished
[08/24 21:12:18]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:18]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:19]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:20]  DEBUG Feeder: Stalling Disk Reader....
[08/24 21:12:20]  DEBUG CNNTPDbImportThread -Ninja-nl-000004010968-000004210968.txt
[08/24 21:12:20]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:20]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:21]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:21]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:21]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:21]  DEBUG CNNTPDbProcessingThread - Importing Compacts Finished
[08/24 21:12:21]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:21]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:21]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:21]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:22]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:23]  DEBUG Feeder: Stalling Disk Reader....
[08/24 21:12:23]  DEBUG CNNTPDbImportThread -Ninja-nl-000004210968-000004410968.txt
[08/24 21:12:23]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:23]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:23]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:23]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:23]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:23]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:23]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:23]  DEBUG CNNTPDbProcessingThread - Importing Compacts
[08/24 21:12:23]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3
[08/24 21:12:24]  ERROR ReadFullDatabase: Invalid packet Format: Storage.db3

[Quade]

How many connections are you using to tweak? How many are you allowed to use? If you use too many connections, you'll get that connection error. It could also imply you account has lapsed. Seems unrelated to the original report.

[Alex Atkin UK]

How many connections are you using to tweak? How many are you allowed to use? If you use too many connections, you'll get that connection error. It could also imply you account has lapsed. Seems unrelated to the original report.

Yes the connection error is unrelated as it was a bandwidth based account used for fills only so likely just used up my quota.

It was the pausing downloaders I couldn't understand, as the app showed unpaused and there is no reason for anything to pause that I can tell.

Despite the above in the logs, headers ARE updating but they're going very very very slowly. This didn't happen the last time I downloaded all headers, so something is definitely not right.

Oh and just a reminder, this is under wine-5.8 (Staging) on Fedora.

[Quade]

I'm just going to assume it's old versus new header DB format thing that I already know about. You can try again in rc2.

[Alex Atkin UK]

The joy of using Beta versions. It DOES seem to be downloading now after refreshing all the headers despite the error messages. More than that, its hitting 400Mbit which as you recall I was struggling with before.

I have also noticed 7z support seems to be much more stable lately, a few versions back I was still having some not extract. So were definitely getting there, and considering how long ago I bought this product its amazing you still keep improving it without charging an upgrade fee.

The one thing that puzzles me is how much CPU seems to be used just idling with a group open. In fact a lot of the issues tends to stem from seeing CPU activity but nothing seemingly happening in the download list or logs (in the application). I will have to keep an eye on the log files themselves I think as I do wonder if that CPU thing is a database issue.

[Quade]

The one thing that puzzles me is how much CPU seems to be used just idling with a group open

I sort of suspect you're seeing this error because you forced it to re-download a bunch of headers you already have. Maybe you added another server and let it download headers (which isn't required). So you have a bunch of header blocks back-logged waiting to import. Or maybe you used "Use download age" on the group to triggered new header downloads of headers you already have.

If you look down where it says "Cache X/Y (ZZ) " ZZ represents the number of header blocks awaiting import. Until that hits zero, Newsbin will be banging away at the disk and using CPU.

[Alex Atkin UK]

I meant in general, I see spikes as high as 50% (though usually more like 25%) of a CPU core, just with Newsbin open and nothing in the queue at all, no group open. It causes the CPU to boost regularly if Newsbin is running and a very lazy/forgetful so Newsbin is usually running.

Its constantly jumping to the top of the list of CPU usage on my box, even though it has a constant Firefox window open showing server stats and ktorrent running.

Should I just blame WINE?

[Quade]

I test it under wine but I don't watch what it's doing that closely. It seems to run as well under Wine as under windows. I see download speeds that are the same as my Windows box. Someone reported they were seeing stuttering while moving the mouse. Makes me wonder if Newsbin is too dependent on having multiple cores at the ready? So it stutters on single or 2 core boxes. I'll see what it looks like it's doing when it's idle. You do apparently have periodic search firing off in the background.

[Alex Atkin UK]

Yeah I don't understand what that periodic search is about, I have nothing configured to do that.

[Quade]

The periodic search thread is running waiting for a periodic search to be defined. I guess it's waking up, sees it's got nothing to do then goes back to sleep for awhile.

[Kjdata]

Dear Quade,

I'm not sure what all that lead up too regarding a solution for the "[12/12 10:06:25] ERROR ReadFullDatabase: Invalid packet Format: Storage.db3" error.
So I'm asking you, as I'm using the RC6 and are getting that error - what do I have to do to solve it ?

[Quade]

Is it an older group? What it suggests is an old data format or corrupted file.

How big is the group?

My solution would be to purge the group, set a short "Download Age" then download headers normally again to rebuild the group database.