Version 6 Autopar updates

Technical support and discussion of Newsbin Version 6 series.

Version 6 Autopar updates

Postby Quade » Sun Nov 21, 2010 7:26 pm

Was just working on the par2dll.dll. It already does multi-core but, I've updated the MD5 routines (which it uses heavily) to use Intel "Performance Primitives" which means it'll take advantage of all the special instructions your CPU might have. In my testing, MD5 calculations are 25% faster than before.

I'd had an idea for awhile now for a way to speed up repair but, the code in the par2dll is pretty damn complicated so, I could never figure out how to hook in my changes. Well, I finally figured it out. What you should see when you use it is that there's far less disk IO when doing a repair. I think for some of those monster 10+ gig repairs some of you are doing, it should make a large difference.

When I release 6, I'll punch up the source for par2dll.dll. It's derived from an open source library so, we publish all the code changes I make to it. Basically, I just work around the edges, adding interface code to it so, Newsbin can use it when it needs it. For perfect files, Newsbin doesn't even use the DLL. All file checking and rar tracking is done internally using code I wrote. It's only when a repair is needed that I make use of the DLL.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44867
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Version 6 Autopar updates

Postby neilt0 » Sun Nov 21, 2010 7:38 pm

I'm not sure what your code base (if that's the right term) is, but I've been experimenting with phpar2 and par2cmdline TBB (64-bit version)

phpar is currently the fastest to verify, and I think they're about equal in repair times.

If you haven't already looked at their code optimisation, it might be worth a look:

http://paulhoule.com/phpar2/index.php
http://www.chuchusoft.com/par2_tbb/
neilt0
Active Participant
Active Participant
 
Posts: 61
Joined: Sun Dec 14, 2003 4:20 pm

Registered Newsbin User since: 12/07/03

Re: Version 6 Autopar updates

Postby Quade » Sun Nov 21, 2010 8:02 pm

Looking at phpar2 is what gave me the idea to update the MD5 generator. Basically that's what's he's done but, using a different method. Par2dll.dll uses TBB too. Since Newsbin is commercial, I bought my own copy of both TBB and Intel performance primitives (IPP). In 6, verification is done before the file is closed, after it's initially written, in order to keep virus scanners from getting in the way.

Like I said though, most of the time the par2dll.dll isn't even used. I seldom need to repair anything. I have a test suite of broken files I generated myself because, using Giganews, I get good files, 99.9% of the time.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44867
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Version 6 Autopar updates

Postby neilt0 » Sun Nov 21, 2010 8:16 pm

Quade wrote:Like I said though, most of the time the par2dll.dll isn't even used. I seldom need to repair anything. I have a test suite of broken files I generated myself because, using Giganews, I get good files, 99.9% of the time.


I used to be with GN, but now I'm a cheapskate with AstraWeb, so I get to test out par2 repair pretty frequently... :wink:
neilt0
Active Participant
Active Participant
 
Posts: 61
Joined: Sun Dec 14, 2003 4:20 pm

Registered Newsbin User since: 12/07/03

Re: Version 6 Autopar updates

Postby richy99 » Sun Nov 21, 2010 9:18 pm

im with astraweb and dont get many incompletes
User avatar
richy99
Elite NewsBin User
Elite NewsBin User
 
Posts: 6353
Joined: Fri Nov 21, 2003 8:04 pm
Location: Wales

Registered Newsbin User since: 12/31/03

Re: Version 6 Autopar updates

Postby neilt0 » Sun Nov 21, 2010 9:24 pm

richy99 wrote:im with astraweb and dont get many incompletes


Try grabbing 500GB a week (Blu-ray) and see if you don't get any...
neilt0
Active Participant
Active Participant
 
Posts: 61
Joined: Sun Dec 14, 2003 4:20 pm

Registered Newsbin User since: 12/07/03


Return to V6 Technical Support

Who is online

Users browsing this forum: Google [Bot] and 3 guests