Duplicate detection no longer working

This is the place to help test and discuss Version 6 Beta releases.

Duplicate detection no longer working

Postby dash » Fri Jun 15, 2018 3:07 pm

Hi guys,

i have a prob with dup detection. It just stopped working a few versions back for me.
Now I'm on 6.81B2 and the problem still persist.
Can't tell for sure what was the last version where it worked properly. I think it was before the all the RC-releases of 6.80.

"Use Duplicate Detector" option is enabled, I've also tried deleting all .DB3 files in Data folder and starting with completely new config, but no avail.
I can download the same file over and over again, without newsbin failing the download due to dup detection like it was before.

Any Idea what else i can try?
dash
Occasional Contributor
Occasional Contributor
 
Posts: 20
Joined: Sun Sep 23, 2012 8:40 am

Registered Newsbin User since: 04/21/12

Re: Duplicate detection no longer working

Postby Quade » Fri Jun 15, 2018 10:29 pm

If the file is already in the folder Newsbin just looks at it, decides all is good and pretends it actually downloaded.

Are you deleting the downloaded file between attempts?

If I add a single file to the download, list, watch it download, then delete it and add it again, it gets sent to the failed list second time through.

Bla [04/70] - "bla.part02.rar" yEnc,X:\Down-Load\,52.2 MB,[1 files, 0 pars] D:0 DL:0 Retry:0 N:0,Duplicate Detection: file has already been downloaded ,2d:03h:10m
bla [05/70] - "bla.part03.rar" yEnc,X:\Down-Load\,52.2 MB,[1 files, 0 pars] D:0 DL:0 Retry:0 N:0,Duplicate Detection: file has already been downloaded ,2d:03h:10m
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44867
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Duplicate detection no longer working

Postby dash » Sat Jun 16, 2018 3:54 am

>If I add a single file to the download, list, watch it download, then delete it and add it again, it gets sent to the failed list second time through.
I've tried it with a single file and indeed does work like you describe and this is what i expect.

But it doesn't work with with "file groups" (bunch of grouped .rar and .par2 files, that you typically get when you open some .nzb file).
In older versions it did work though. Newsbin failed the download for any already downloaded file group, doesn't matter when you downloaded it.
The whole group was then moved to "Failed Files" tab with Duplicate Detection reason/error.

In newer versions i can open the same .nzb file over and over again and newsbin will download and extract the file group creating copies of extracted files:
xubuntu-18.04-desktop-amd64.iso
xubuntu-18.04-desktop-amd64-(0001).iso
xubuntu-18.04-desktop-amd64-(0002).iso
dash
Occasional Contributor
Occasional Contributor
 
Posts: 20
Joined: Sun Sep 23, 2012 8:40 am

Registered Newsbin User since: 04/21/12

Re: Duplicate detection no longer working

Postby dash » Fri Jun 22, 2018 5:09 am

So any ideas?
dash
Occasional Contributor
Occasional Contributor
 
Posts: 20
Joined: Sun Sep 23, 2012 8:40 am

Registered Newsbin User since: 04/21/12

Re: Duplicate detection no longer working

Postby Quade » Fri Jun 22, 2018 1:20 pm

I'm seeing it prevent re-downloads of stuff I add from headers. Are you using NZB's? NZB's bypass the dup checker I believe.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44867
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Duplicate detection no longer working

Postby dash » Fri Jun 22, 2018 4:36 pm

> Are you using NZB's? NZB's bypass the dup checker I believe.
Yes i do. I rarely index headers myself. I mostly download nzbs from index sites and Dup checker did work with NZB files as well.
dash
Occasional Contributor
Occasional Contributor
 
Posts: 20
Joined: Sun Sep 23, 2012 8:40 am

Registered Newsbin User since: 04/21/12

Re: Duplicate detection no longer working

Postby Moerges » Sun Jul 15, 2018 7:24 pm

Hello Quade, me again. I've noticed this new dupe handling behaviour as well (well, "new" compared to my 5.59 experience with Newsbin) and was on my way to open a thread about it when I found this one.

I almost exclusively download via .nzb files, and found the dupe checker in 5.59 rather useful to avoid unnecessary redownloads. Now that .nzbs simply bypass the dupe checker, it unfortunately no longer does the job it used to do for my usage scenario.

If it is not too much of a bother, would it be possible to reimplement the dupe checking for downloads added via .nzb files—perhaps as an option so the user can choose which behaviour he prefers?

---

Also, while testing the dupe checking behaviour of 6.80 build 5058, I came across an oddity that I suspect might be linked to the duplicate detection:

My Newsbin is set to download to G:\Newsbin temp\$(NZBFILE)\ and to unrar to D:\Newsbin\$(NZBFILE)\.

When I add a download via an .nzb file for the first time, it works as one would expect: the .rars are downloaded to G:\Newsbin temp\$(NZBFILE)\ and are then extracted to D:\Newsbin\$(NZBFILE)\.

However, when I add the same download via the same .nzb file a second time, instead of downloading the .rars to G:\Newsbin temp\$(NZBFILE)\, Newsbin downloads them to D:\Newsbin\$(NZBFILE)\ (the unrar folder) and afterwards extracts them there, too.

This happens no matter if I have deleted the first file or not, both from the "Files List" and from my disk. It also does not matter whether Use Duplicate Detector in the Advanced tab of the settings is checked or not. Still, I could imagine this is linked to the dupe checker in some way, or possibly some other form of download history.

I hope you can reproduce the issue on your end, or tell me how I can fix it on mine in case I am doing something wrong. (I am not running any filters or scripts, by the way. I'm simply adding the .nzbs—either via a watched folder or via double click on an .nzb file—without altering any settings or download paths in between.)

Again, thank you in advance for any insights you can offer.

Moerges
Moerges
n00b
n00b
 
Posts: 5
Joined: Sun Jul 15, 2018 3:25 am

Registered Newsbin User since: 01/29/10

Re: Duplicate detection no longer working

Postby alanstarr » Tue Jul 24, 2018 9:39 am

Moerges wrote: I almost exclusively download via .nzb files, and found the dupe checker in 5.59 rather useful to avoid unnecessary redownloads. Now that .nzbs simply bypass the dupe checker, it unfortunately no longer does the job it used to do for my usage scenario. If it is not too much of a bother, would it be possible to reimplement the dupe checking for downloads added via .nzb files—perhaps as an option so the user can choose which behaviour he prefers?
Moerges


I second this notion, I miss having the dupe checker on NZB files.
alanstarr
Occasional Contributor
Occasional Contributor
 
Posts: 47
Joined: Sun Jan 23, 2005 2:49 pm
Location: Lawrence, MA

Registered Newsbin User since: 01/20/05

Re: Duplicate detection no longer working

Postby dash » Mon Aug 27, 2018 5:05 pm

looks like no one cares...

Downgrade it is then i guess.
Does anyone remember the last version where it was working? 5.57? 5.59?

Honestly, for me, the last updates broke more useful features than they actually added:
- Duplicate detection no longer working
- Newsbin not failing password-protected files anymore, if password is not in password manager.
- Newsbin not failing archives with bogus data, like it used to
- Mp3 not downloading into subfolders (MP3 Folder mode not working)
- Indexing/searching got much slower
etc etc etc
Cant say much about the bug-fixes in updates, since I only started noticing bugs and annoyances after I updated and they are still there (6.81B4).

I even tried installing NB from the scratch, thinking that config of my old version somehow fucks with the new version of NB.
While doing so I also had to reindex a few newsgroups anew. Thats also when I notices 140GB of text files under "\Newsbin Data\Import\" after downloading the newsgroups headers... It took a whole day for Newsbin to import those text files into DBs under /SPOOL_V6/ (10GB after the import was done).
Why cant NB just import the headers directly into DB when downloading those? There must be a better way to do it, that doesn't involve creating hundreds of thousands of texfiles first...

Anyway, all my bitching aside, NB is a great program that I've been using for like 8-10 years. Im just annoyed with the last few years, where subjectively more useful stuff was broken than was added/fixed (at least that's the way i perceived it)...
Last edited by dash on Mon Aug 27, 2018 5:43 pm, edited 1 time in total.
dash
Occasional Contributor
Occasional Contributor
 
Posts: 20
Joined: Sun Sep 23, 2012 8:40 am

Registered Newsbin User since: 04/21/12

Re: Duplicate detection no longer working

Postby itimpi » Mon Aug 27, 2018 5:13 pm

Why cant NB just import the headers directly into DB when downloading those? There must be a better way to do it, that doesn't involve creating hundreds of thousands of texfiles first...

This is done because on modern broadband links the headers can be downloaded MUCH faster than they can be processed for insertion into the DB.

For large groups processing headers is not normally worthwhile any more. Much easier to subscribe to system that provides searching and NZB files.
The Newsbin Online documentation
The Usenettools for tutorials, useful information and links
User avatar
itimpi
Elite NewsBin User
Elite NewsBin User
 
Posts: 12604
Joined: Sat Mar 16, 2002 7:11 am
Location: UK

Registered Newsbin User since: 03/28/03

Re: Duplicate detection no longer working

Postby dash » Mon Aug 27, 2018 5:39 pm

itimpi wrote:This is done because on modern broadband links the headers can be downloaded MUCH faster than they can be processed for insertion into the DB.

yep, I guessed that much. But since NB needed 5 sec to import a single txt file afterwards, the whole process took a day. The importing process seemed unnaturally slow.
But well, since you do the initial indexing of a group only once, i can live with it.
(I do have a subscription to a few NZB indexers, but i also prefer to index a pair of groups myself (books))

I'm honestly more annoyed with the stuff that worked and got broken in new versions. The slow importing of headers was just a side note.
dash
Occasional Contributor
Occasional Contributor
 
Posts: 20
Joined: Sun Sep 23, 2012 8:40 am

Registered Newsbin User since: 04/21/12

Re: Duplicate detection no longer working

Postby Quade » Mon Aug 27, 2018 5:44 pm

- Newsbin not failing password-protected files anymore, if password is not in password manager.


I just tested this in 6.81RC1 which should be on the forum today. I picked a password protected file set to download and it failed immediately.

bla - [02/60] - "lbQrFCdV4OOKqRs.part01.rar" yEnc,X:\Down-Load\,5.91 GB,[52 files, 8 pars] D:0 DL:0 Retry:0 N:0,UnRAR Failure: File set is Encrypted and no password was found

- Mp3 not downloading into subfolders (MP3 Folder mode not working)


I tested this. It's working. Even when it did a repair and moved the files to the unrar folder, it still used MP3 folder mode. I tried different modes too though not all of them.

While doing so I also had to reindex a few newsgroups anew. Thats also when I notices 140GB of text files under "\Newsbin Data\Import\" after downloading the newsgroups headers... It took a whole day for Newsbin to import those text files into DBs under /SPOOL_V6/ (10GB after the import was done).


Because your machine and mine, can't do it. There's two choices:

A - Slow down the header download to the speed it can feed the database.
B - Run the header download at full speed and catch up on the backend.

Obviously B is how Newsbin works.

140 gigs turns into 10 gigs because of how Newsbin compacts the data.

Some groups are useless for header downloads these days. People aren't posting files to them, just posts that can only be used when using NZB files. You might want to look at the groups you're using and see if any of these have gone rogue. Your first 3 issues can happen if you Ctrl-Y the downloads (force the downloads) instead of downloading them normally. If you're one of those people who mark posts, turns them into NZB's then feeds the NZBs into Newsbin again (don't ask me why people do this but some of them to) the NZB's will automatically do Ctrl-Y. So I'd recommend not doing that.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44867
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Duplicate detection no longer working

Postby dash » Mon Aug 27, 2018 6:04 pm

Hi Quade,

thanks for the reply. Did 6.81RC1 also fix the duplicate behavior with NZBs?

Quade wrote:- Newsbin not failing password-protected files anymore, if password is not in password manager.
I just tested this in 6.81RC1 which should be on the forum today. I picked a password protected file set to download and it failed immediately.
bla - [02/60] - "****.part01.rar" yEnc,X:\Down-Load\,5.91 GB,[52 files, 8 pars] D:0 DL:0 Retry:0 N:0,UnRAR Failure: File set is Encrypted and no password was found

Tested it with the file from your post with 6.81B4 and the download didn't fail (NB kept on downloading until i deleted the file from filelist).
So thanks for fixing it in RC1.

Quade wrote:- Mp3 not downloading into subfolders (MP3 Folder mode not working)
I tested this. It's working. Even when it did a repair and moved the files to the unrar folder, it still used MP3 folder mode. I tried different modes too though not all of them.

in 6.81B4? It didn't work for me several times just today... All mp3 were inside of default download folder and not as expected in "Album-Artist" subfolders (all mp3s had required metadata for that)

Edit:
Sorry, my bad.
Just checked the NZB files again and not all of them did contain MP3s. Some NZBs contained .rar archives.
That explains why MP3s were extracted directly into default download folder.
NZBs with MP3s did all download correctly under "Album-Artist" subdirs.
Sorry about the false report and for wasting your time with it...


Because your machine and mine, can't do it. There's two choices:

A - Slow down the header download to the speed it can feed the database.
B - Run the header download at full speed and catch up on the backend.

Obviously B is how Newsbin works.

140 gigs turns into 10 gigs because of how Newsbin compacts the data.
...

Yeah i understand it. It just that the import process seemed to take too long. But its ok.
dash
Occasional Contributor
Occasional Contributor
 
Posts: 20
Joined: Sun Sep 23, 2012 8:40 am

Registered Newsbin User since: 04/21/12

Re: Duplicate detection no longer working

Postby Quade » Mon Aug 27, 2018 10:15 pm

Tested it with the file from your post with 6.81B4 and the download didn't fail (NB kept on downloading until i deleted the file from filelist).
So thanks for fixing it in RC1.


There's no fix for this in RC1. RC1 and B4 are nearly identical. How are you adding the download to the download list.

I was just about to mention MP3 mode doesn't apply to rar sets....

Yeah i understand it. It just that the import process seemed to take too long. But its ok.


People act like 140 gigs of data is just a hop, skip and a jump.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44867
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97

Re: Duplicate detection no longer working

Postby dash » Tue Aug 28, 2018 12:42 pm

Quade wrote:There's no fix for this in RC1. RC1 and B4 are nearly identical. How are you adding the download to the download list.

I added it as NZB file.
Search for the file you posted on any indexer, ex. http://www.nzbindex.com and download NZB.
The download wont fail for me, even though the option "Fail rar files that need password" is checked under "Spam Filter Settings".

This feature just stopped worked some updates ago, just like duplicate detection for NZB files.
(Same with detection of bogus files within archives or download of .EXE files if those were added through NZBs)
Everything of the above did work for NZBs in older versions of newsbin (I believe last in 6.56 and maybe a few versions after that, not sure).
dash
Occasional Contributor
Occasional Contributor
 
Posts: 20
Joined: Sun Sep 23, 2012 8:40 am

Registered Newsbin User since: 04/21/12

Re: Duplicate detection no longer working

Postby dexter » Tue Aug 28, 2018 1:05 pm

Maybe you missed it but Quade mentioned earlier that all NZBs load with a ctrl-y, which means it bypasses filters and duplicate detection. This change was made because people were having issues with NZBs coming in from Sonarr not downloading. Quade and I have discussed re-enabling duplicate detection for manually loaded NZBs but the change was too complicated to safely implement in this beta cycle.

It's always a tradeoff with features like this. Either we field technical support from people asking why their files don't download, or we get complaints that duplicate files are getting downloaded. If a duplicate does download, it is clearly marked as such.
User avatar
dexter
Site Admin
Site Admin
 
Posts: 9511
Joined: Fri May 18, 2001 3:50 pm
Location: Northern Virginia, US

Registered Newsbin User since: 10/24/97

Re: Duplicate detection no longer working

Postby dash » Tue Aug 28, 2018 5:39 pm

dexter wrote:Maybe you missed it but Quade mentioned earlier that all NZBs load with a ctrl-y ...


Hallo dexter,

I've indeed missed that bit.
Well that's explains pretty much all the issues i'm having with newer versions of newsbin...

Ive subscription to a few indexers and they all provide NZBs when you search there for something. That's how i was getting stuff for years.
Have I been missing out on something? What did change?
dash
Occasional Contributor
Occasional Contributor
 
Posts: 20
Joined: Sun Sep 23, 2012 8:40 am

Registered Newsbin User since: 04/21/12

Re: Duplicate detection no longer working

Postby dash » Thu Aug 30, 2018 10:23 am

dexter wrote:Quade and I have discussed re-enabling duplicate detection for manually loaded NZBs but the change was too complicated to safely implement in this beta cycle.

Cant you just make it option based (checkbox under Options or something)?
If its checked, then manually added NZBs will be loaded without "forced mode", just like in old versions?
dash
Occasional Contributor
Occasional Contributor
 
Posts: 20
Joined: Sun Sep 23, 2012 8:40 am

Registered Newsbin User since: 04/21/12

Re: Duplicate detection no longer working

Postby Quade » Thu Aug 30, 2018 1:41 pm

That might be doable too.
User avatar
Quade
Eternal n00b
Eternal n00b
 
Posts: 44867
Joined: Sat May 19, 2001 12:41 am
Location: Virginia, US

Registered Newsbin User since: 10/24/97


Return to Newsbin Version 6 Beta Support

Who is online

Users browsing this forum: Orion! and 2 guests