Page 1 of 1

6.8 final HDD load

PostPosted: Fri May 04, 2018 6:37 pm
by jonas
Hi,

I just upgraded to 6.8 final from a previous beta (not quite sure which one) and I noticed that since updating, the load on my HDD where the Newsbin profile is stored and the chunks get written to before the actual files are created (the download folder is on a different HDD) goes to 100% while downloading and my download speed drops by almost half. I didn't have this problem before so I assume it might be related to the new RAR processing? Anything I can do on my end?

Re: 6.8 final HDD load

PostPosted: Fri May 04, 2018 10:35 pm
by Quade
You mean the amount of disk space used? How much free space do you have on this drive?

Re: 6.8 final HDD load

PostPosted: Sat May 05, 2018 5:38 am
by jonas
No, I don't mean the amount of disk space (the HDD has around 200GB free if that still matters). I'm talking about simultaneous read/write operations. I did some further tests last night and I noticed, that the problem does not occur with every file I tried. Sometimes Newsbin behaves as I would expect: checking in Task Manager the activity on HDD 1 (where the chunks are stored) goes to below 20%, activity on HDD 2 (where the actual files are created) goes to around 40% and I get a speed of around 400Mbit downloading.
But with a lot of files, I get the problem I described above: Activity on HDD 1 spikes and the speed drops to around 250Mbit for the entire download. It is definitely not random, since re-downloading the files I tested always leads to the the same behaviour.

EDIT:
I did some more tests and it seems it is actually the minority of files it happens with, so it is quite possible, that the problem is not new but that I simply did not notice it until yesterday.

Re: 6.8 final HDD load

PostPosted: Sat May 05, 2018 11:12 am
by Quade
Newsbin buffers chunks in memory but if the chunk count of a file > memory chunks the chunks get flushed out to disk. The chunk count is down where it says "Cache" at the bottom. The default is 400 chunks in 6.80. If you're downloading files that are > 400 chunks per file, some of that partial file will hit disk.

Your symptoms suggest high chunk count files.


[PERFORMANCE]
ChunkCacheSize=X

If you add this to the NBI file and set the chunk cache to larger than the per file chunk size, you might make this go away. It's about 1 meg per chunk block worst case. Some people set it to 1000 or 2000. There's a point of diminishing returns too. If you're using an SSD for this drive, you probably don't need to change this.

Re: 6.8 final HDD load

PostPosted: Sun May 06, 2018 9:47 am
by jonas
Thanks for the suggestion, I've set it to 2000 and performance seems to be improved while still not perfect. Setting it to a higher value makes no more difference.

Re: 6.8 final HDD load

PostPosted: Thu May 10, 2018 9:04 am
by jonas
Would it be possible for me to download older beta releases? I would really like to test, when the performance drop was introduced.

Re: 6.8 final HDD load

PostPosted: Thu May 10, 2018 10:30 am
by Quade
You might try playing with the URL for the download. I think Dex leaves the older ones on the site.

I'm kinda surprised you're seeing any performance difference in the last couple beta's. I might uninstall any non-Microsoft security software then reboot and try some downloads. More often then not sub-standard performance when everything else is the same is the result of security software stepping in.

Re: 6.8 final HDD load

PostPosted: Thu May 10, 2018 12:09 pm
by Quade
One of the users on the IRC channel was hitting 800 with peaks to nearly 1 Gbps with the 6.80 release. It's just a data point but it means it's possible to get high speeds with it.

Re: 6.8 final HDD load

PostPosted: Thu May 10, 2018 12:33 pm
by jonas
Quade wrote:You might try playing with the URL for the download. I think Dex leaves the older ones on the site.


I've tried that, didn't work unfortunately.

Quade wrote:I'm kinda surprised you're seeing any performance difference in the last couple beta's. I might uninstall any non-Microsoft security software then reboot and try some downloads. More often then not sub-standard performance when everything else is the same is the result of security software stepping in.


There is no security software apart from windows defender installed on my machine.

Quade wrote:One of the users on the IRC channel was hitting 800 with peaks to nearly 1 Gbps with the 6.80 release. It's just a data point but it means it's possible to get high speeds with it.


I'm sure if you have a faster HDD or SSD it won't be as much of a bottleneck as it is for me, but now that I've watched performance the last couple of days I am sure that the problem is definitely new for me. Might be related to something other than a change in newsbin but without the previous betas I can't test it...

Re: 6.8 final HDD load

PostPosted: Thu May 10, 2018 5:21 pm
by dexter
jonas wrote:Would it be possible for me to download older beta releases? I would really like to test, when the performance drop was introduced.


Sorry, I cleaned out all the beta releases once the final release was out. You can run the last production release which was 6.73 and see if there is a difference if you like. It is on the Archive Page

Re: 6.8 final HDD load

PostPosted: Thu May 10, 2018 8:39 pm
by jonas
dexter wrote:
jonas wrote:Would it be possible for me to download older beta releases? I would really like to test, when the performance drop was introduced.


Sorry, I cleaned out all the beta releases once the final release was out. You can run the last production release which was 6.73 and see if there is a difference if you like. It is on the Links not allowed for unregistered users


ok, so I installed version 6.73 as suggested and infact I don't experience any unusual activity on my HDD and get a consistent 450 mbps. Installed 6.8 final again and HDD activity spikes and speeds go down to about 250 mbps. Something in the latest version does not play nice with my setup Image link not allowed for unregistered users

Re: 6.8 final HDD load

PostPosted: Thu May 10, 2018 10:18 pm
by Quade
You could use "Procmon" to examine disk IO.

Re: 6.8 final HDD load

PostPosted: Fri May 11, 2018 10:16 am
by jonas
Quade wrote:You could use "Procmon" to examine disk IO.


The only thing that jumps out at me is, that in both versions I'm seeing a lot of "QueryOpen" operations on chunk files that return "NAME NOT FOUND" and while the old version followed by performing a single "WriteFile" operation on the Logfile.txt, the new version performs two of those and inbetween a "FlushBuffersFile" operation. Seems like that could be a factor in the increased activity I'm seeing?

Re: 6.8 final HDD load

PostPosted: Fri May 11, 2018 10:27 am
by Quade
he new version performs two of those and inbetween a "FlushBuffersFile" operation. Seems like that could be a factor in the increased activity I'm seeing?


Yeah, this one is a possibility.

Are you logging server commands? You might want to disable logging server commands in the network options and try turning down the logging in the advanced page.

Re: 6.8 final HDD load

PostPosted: Fri May 11, 2018 11:08 am
by jonas
Quade wrote:
he new version performs two of those and inbetween a "FlushBuffersFile" operation. Seems like that could be a factor in the increased activity I'm seeing?


Yeah, this one is a possibility.

Are you logging server commands? You might want to disable logging server commands in the network options and try turning down the logging in the advanced page.


Setting logging to normal under advanced options makes no difference. With disable logging server commands you mean the "show server commands" option? Other than that I don't see an option to control logging in the network tab and that option was disabled to begin with.

Re: 6.8 final HDD load

PostPosted: Fri May 11, 2018 12:28 pm
by Quade
Yeah "Show server commands". On a high speed connection logging that might consume more disk and CPU than you want.

Re: 6.8 final HDD load

PostPosted: Tue May 15, 2018 11:49 am
by jonas
So I moved the DataPath from one of my HDDs to my system drive, which is an SSD, and that eliminates the bottleneck I was experiencing. Somewhere between version 6.73 and 6.8 Newbin got a lot less efficient and it seems to download at high speeds you now need to use an SSD at least for the profile folder.

Re: 6.8 final HDD load

PostPosted: Fri May 18, 2018 10:31 am
by LaRokka
hello

pretty similar issue. just updated from 6.72 (release) to 6.80 (release) and my hard drive is getting hammered. in 6.72 all was smooth.
using a quite normal samsung 7200rpm sata 3 hdd as my download drive.
my download speed is officially 150mbit, usualy sits between 150 and 160mbit.

since 6.80 the hdd is constantly hammering when downloading. the drive LED is permanetly on, just stops for nanosecs every few secs.
the larger the files it is downloading are, the worse it gets.

1-3gb i am still getting 150-160 mbits.
3-6gb the harddrive is hit so hard, download goes down to 120ish mbit.
8gb down to 100ish mbit.
didnt test bigger files yet as i just installed 6.80.

i have "pre-allocate files for spinnign discs" enabled. tried to disable it, does not change behaviour.
it somehow seems as 6.80 is not caching (just from belly feeling, cant say for sure). but i know it wasnt this bad at any time before this release in the past years.

help greatly appreciated ! :)

no native speaker, plz excuse me :)

Re: 6.8 final HDD load

PostPosted: Fri May 18, 2018 10:39 am
by Quade
I suspect it has something to do with the logging doing "FlushFileBuffers". I've disabled that in code but you won't see it till 6.81ish.

It's looking like we need a 6.81 because some people are reporting "Add to download list and it stalls" problems.

Re: 6.8 final HDD load

PostPosted: Fri May 18, 2018 2:05 pm
by LaRokka
i just downloaded 6.73 from the archive. can i install it over 6.80 like a normal update, with my settings kept ?
i would then continue to use 6.73 until 6.81 is out. didnt have issues with the older version anyhow, just updated to be up-to-date :)

Re: 6.8 final HDD load

PostPosted: Fri May 18, 2018 2:11 pm
by dexter
Yes, you can install 6.73 over 6.80 with no issues. I've done it myself during the testing phase. If you are a header downloader, I'd make sure the import folder is clean before downgrading since they are no longer compressed in 6.80 so 6.73 won't process them.

Re: 6.8 final HDD load

PostPosted: Fri May 18, 2018 2:52 pm
by LaRokka
ok guys, thank you for your quick replys.

see you in 6.81 :)

Re: 6.8 final HDD load

PostPosted: Thu May 31, 2018 10:02 pm
by jonas
Looks like the just released 6.81 beta fixes the issue.

Re: 6.8 final HDD load

PostPosted: Thu May 31, 2018 11:12 pm
by Quade
Thanks for the feedback. I removed some "flush file buffer" calls.

Re: 6.8 final HDD load

PostPosted: Sat Sep 22, 2018 7:40 pm
by LaRokka
updated from 6.73 to 6.81, currently downloading a 15gb file.
download jumping between 120 and 150 Mbps, should be 150-160. high HDD activity happening when its jumping.

doanloaded several 1-3gb files last night, wasnt as appearant then. seems it gets worse the bigger the files are.

it is better than 6.80 but not as good as 6.73

Re: 6.8 final HDD load

PostPosted: Fri Sep 28, 2018 3:06 pm
by LaRokka
please ignore my last post. currently downloading a 39gb file @steady 150-160Mbps.
so all is well for my serrverĀ“s hdd in 6.81

maybe something funny was going on with the files last weekend, or my isp/usenetprovider was acting ... /shrug