how do I get Newsbin to download ALL headers?

[confusednewb]

I know - under GROUPS, I right click, drop down, and highlight DOWNLOAD ALL HEADERS.

But while Newsbin starts downloading, it then hiccups, the total header count drops, and eventually I'll end up with 30-60 days, instead of the 600 day retention my news provider has.

Wouldn't be an issue, but there are some posts that are only available that far back. (I can find them using your internet search option, but since they ARE on my server, I'd think I should be able to do a LOCAL search and just download them) But no - since I can't get headers that far back loaded in a large group (boneless)

[apsen]

I've noticed that too. When I tried to get all headers for testing from a.b.boneless - I got only about 25 days worth.

Haven't tested that much and didn't check with provider to make sure there's no problem on their side so was not inclined to report it yet.
BTW Provider's retention supposed to be over 1000 days for the binary groups.

[itimpi]

You need to check what retention your provider has for headers separately. Many servers have longer retention for files than is reflected in their headers.

Such a discrepancy might explain why you cannot get files via headers, but can get them via Internet search.

[confusednewb]

I'm using Easynews - I know they have bucco retention - in smaller groups, Newsbin has no trouble going back 500 days. (which is where I have it set - what a coincidence)

So I'm guessing its some kind of glitch with the retrieval or storage in Newsbin, causing it not to work with large numbers of posts. I can use the Easynews web interface and go back 209 days - it also lists 13 million posts. But its a lot easier to download and put together a large file with Newsbin's AutoPar.

Any ideas, mods/techs? fwiw - I've had header retrievals crash with an error message saying something about "byte limit reached." I checked with Easynews, they said the limit wasn't on their end, and they were able to download all 120 million posts on their end with no problem.

C'mon - I got Newsbin cuz it was supposed to be the best... Image link not allowed for unregistered users

[Quade]

This is common on many servers. They have a hard cap of 125 million headers on every group which on a large group might only be 30-60 days.

It's easy to tell if this is the case. In the options, network turn on "Show Server Commands". Then right click a group and "Download Latest" on it. Then check the logs.

[13:23:15] LOW InterSocket - SSL Connection Server: news.giganews.com Stats: AES256-SHA 256 TLSv1/SSLv3
[13:23:15] HIGH NNTPSocket - XFEATURE COMPRESS GZIP
[13:23:15] HIGH NNTPSocket - GROUP <bla bla>
[13:23:15] HIGH NNTPSocket - 211 1056210953 226921634 1283132586 <bla bl>
[13:23:15] HIGH NNTPSocket - XOVER 1282699487-1282999487

[13:23:15] HIGH NNTPSocket - 211 1056210953 226921634 1283132586 <bla bl>

211 - A-ok
1056210953 - total records
226921634 - Min range
1283132586 - Max range

1,056,210,953 - total records. So this group has more than 1 billion posts in it. It's not the largest group either. If you do a larger group and only see 125 million headers, the problem is the server.

Pretty sure Easynews is one of these servers.

Any ideas, mods/techs? fwiw - I've had header retrievals crash with an error message saying something about "byte limit reached." I checked with Easynews, they said the limit wasn't on their end,

I suspect they're lying to you. It's common for servers to limit total transfers on a connection and then force a disconnect. It doesn't matter to Newsbin, it just re-connects and continues the download.

[confusednewb]

Thanks, Quade. I didn't know how to read the 211 information, so thank-you very much. You're right, Easynews is coming back with a 120,000,001 headers.

Guess I need to do more searching here, and find a server that has more headers available. Although - sir - my first problem is that I can't get even 120,000,000 to download. NBP always stops somewhere around 80 million. Which would take me back 400 days, which would be far enough...

[Quade]

If you're getting 80 million, keep in mind you're already getting the 80 million OLDEST headers. Header downloads are always oldest to newest. If you can't see your 400 day old posts in the 80 million you already have, you won't be seeing them. "Show All Posts" is how you see them all. Make sure the "Storage Age" is set at least as long as the number of days you want to keep.

To make Newsbin get the rest, just "Download Latest". It'll take off where it left off. Giga has the longest header retention and I think Astra is a close second. Giga is the only server that has as many headers as they have retention. You could always sign for a month and download all the headers.... Then download the files from Easy.

It's normal for the numbers to be smaller on subsequent header downloads because you already have some stored.

[confusednewb]

Will Do. Thanks - again, you've educated me. Appreciated!

I have the Storage Age set, but will do the other things you mention.

Thanks again