Jump to content

Sign in to follow this  
jrisles

Do you maintain digital file integrity?

Recommended Posts

Do you maintain digital file integrity?

I don't even know what that means! :confused:

Share this post


Link to post
Share on other sites

I have one set of wav files, of which I have a copy on an external hard disk.

Any conversions I do (which is very rare), I do to anther internal disk and copy out from there.

 

Has anyone found some comparison software yet?

I'd be keen to compare my two sets of wav files.

 

That said, I don't seem to have come across any dud files so far.........all ripped via dbpoweramp on the most accurate setting.

Share this post


Link to post
Share on other sites
Guest myrantz

Has anyone found some comparison software yet?

Comparison isn't quite the same as integrity... (you can compare two corrupt files and they will pass because they are bit for bit exact or have the same checksum etc)..

 

integrity checking ensures the file itself has no errors. (like testing ZIP files..), e.g. with flac

flac.exe -t test.flac
 
flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007  Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.
test.flac: ok

Caveat that the integrity check actually comes with absolutely no warranty.. but apparently it's well tested enough to be problem free... So far so good (*touch wood*)

 

Share this post


Link to post
Share on other sites

What about scratches and dust? Or the corruption going on in vinyl prices.

 

nah, not a real issue and I use a record cleaner, so all good.  Pricing is getting to be a rip off thou.

Edited by metal beat

Share this post


Link to post
Share on other sites


I know enough about computers to expect nothing ... :P RAM problem isn't deterministic... Test it and see anyway... A lot of systems are running on bad RAM, and they are 100% working, until they're not...

 

Is there a reason for ALAC? Why not use FLAC? JRiver plays FLAC, and it supports verification on encode, and testing after encode... Best of both worlds?

There's no perfect solution out there.. remember this that stirred up a paranoia a while back? Silent errors are the worst kind of corruption, can't really avoid 'em.

 

I started ripping using ALAC years ago and stuck with that format. After a lot of reading and listening to different formats when i was deciding to look at uncompressed formats i decided to stay with the apple proprietary AIF format to convert all my ALAC files from. So the ALAC decision is more a legacy decision than anything else.

 

I have one set of wav files, of which I have a copy on an external hard disk.

Any conversions I do (which is very rare), I do to anther internal disk and copy out from there.

 

Has anyone found some comparison software yet?

I'd be keen to compare my two sets of wav files.

 

That said, I don't seem to have come across any dud files so far.........all ripped via dbpoweramp on the most accurate setting.

 

I am confident the ripping process has not actually caused the number of files that has become corrupted over time. I can't say for certain that not every file was ripped with 100% accuracy because i don't actually replay in its entirety every album that i have ripped and i guess not many people do. But for the most part i don't put the corruption down to ripping. I suspect it has probably more to do with the number of transfers i have made as i move my library from one disk to another as my library has grown over the years. I also suspect unexpected outages have also contributed to the problem. 

Share this post


Link to post
Share on other sites

Ive got two corrupted files of 7,000. I found out when i used Jriver to analyse the entire library for its dynamic range for the r128 volume normalisation feature (which i don't use anymore). No idea what caused it...

Share this post


Link to post
Share on other sites

Ive got two corrupted files of 7,000. I found out when i used Jriver to analyse the entire library for its dynamic range for the r128 volume normalisation feature (which i don't use anymore). No idea what caused it...

 

Care to elaborate?

Share this post


Link to post
Share on other sites


Guest myrantz

Sorry, on which point?

Container used: wave/flac/ALAC/ape, etc.

Source of files: Ripped from CD (if yes, which encoder), digital downloads

 

Where are files stored: NAS, USB HDD, local HDD, etc; What filesystem (NTFS, FAT32, etc)

What OS: Mac, Linux or Windows

 

Activities: Did you transfer files back/forth between mediums in the past? What tagging software did you use?

Doubt can find the cause of the corruption now, But if you can provide the above hopefully there'd be some clues...

Share this post


Link to post
Share on other sites

Sorry, on which point?

 

Also keen to understand what you mean by "dynamic range for the r128 volume normalisation feature" - how can this be an effective tool to determine or pick out corrupted files?

Share this post


Link to post
Share on other sites
Guest myrantz

Also keen to understand what you mean by "dynamic range for the r128 volume normalisation feature" - how can this be an effective tool to determine or pick out corrupted files?

R128 is a standard, normalisation is sorta like adjusting the volume (can be very simple like a volume control or more complicated)..

It's effective because normalisation will involves decoding the whole file to find the peak and average volumes (and so on).. You can probably do the same by running the dynamic range plugin and update the metadata tags with the DR number.. It's the decoding the file part that helps detect corruption..

Common things between you and him so far appears to be JRiver :P.

Share this post


Link to post
Share on other sites

Hi jrisles,

 

There is a program called Perfecttunes which is software that you can get as an extra to dbpoweramp; or on its own.

https://www.dbpoweramp.com/perfecttunes.htm

 

Perfecttunes has three components:

 

DeDup: Checks for duplicates in your music library and removes them (with user controls).  This program is quite clever in that it allows you to listen to the tracks in questions and even switch the stream during playback.

 

Album art: Automatically adds missing album covers.

 

Accurate Rip: This will verify previously ripped tracks

 

From the webpage:

“PerfectTUNES AccurateRip verifies previously ripped discs, perfect for validating iTunes or other rippers not capable of checking AccurateRip.â€

 

If you want the ability to check rips that have not been ripped with Accurate Rip, or check on “data integrity†in these terms, then perhaps this may be something that you’re looking for.

 

Milo.

Share this post


Link to post
Share on other sites


Container used: wave/flac/ALAC/ape, etc.

Source of files: Ripped from CD (if yes, which encoder), digital downloads

 

Where are files stored: NAS, USB HDD, local HDD, etc; What filesystem (NTFS, FAT32, etc)

What OS: Mac, Linux or Windows

 

Activities: Did you transfer files back/forth between mediums in the past? What tagging software did you use?

Doubt can find the cause of the corruption now, But if you can provide the above hopefully there'd be some clues...

Ok so they were ripped onto ssd via dbpoweramp as a mildly compressed flac (later all compression was removed). I think i had the check feature turned off as i had 300 cds to rip, so the problem might have been right at the beginning. They were then transferred to another ssd when i outgrew the original. I can't remember if i did the original rips on w7 64 or w8 64, i think 7.

The other poster is right, i only became aware of the issue when i scanned all 7,000 files, at the end it said there werwere two left, so i investigated and thats when i realised... Hth.

Share this post


Link to post
Share on other sites

Ok so they were ripped onto ssd via dbpoweramp as a mildly compressed flac (later all compression was removed). I think i had the check feature turned off as i had 300 cds to rip, so the problem might have been right at the beginning. They were then transferred to another ssd when i outgrew the original. I can't remember if i did the original rips on w7 64 or w8 64, i think 7.

The other poster is right, i only became aware of the issue when i scanned all 7,000 files, at the end it said there werwere two left, so i investigated and thats when i realised... Hth.

 

Did you attempt to replay the two files (lucky you) you found were corrupt? Could you play them? Mine are able to be played but during the song the file makes a static white noise and the file is then skipped with the player moving to the next song. You don't know the file has become corrupt until it is played. The issue has nothing to do with the ripping process for 99.9% of my files. So even with all the best error correction when ripping in the world this is not the actual problem. The problem is with this phenomena called bit decay or bit rot over the years i have had my files.

Share this post


Link to post
Share on other sites
Guest myrantz

Ok so they were ripped onto ssd via dbpoweramp as a mildly compressed flac (later all compression was removed). I think i had the check feature turned off as i had 300 cds to rip, so the problem might have been right at the beginning. They were then transferred to another ssd when i outgrew the original. I can't remember if i did the original rips on w7 64 or w8 64, i think 7.

The other poster is right, i only became aware of the issue when i scanned all 7,000 files, at the end it said there werwere two left, so i investigated and thats when i realised... Hth.

Hmm. Interesting, and it's just normal copy in Windows?

Now I'm worried/paranoid.. Gonna re-do my integrity check and report back.

Share this post


Link to post
Share on other sites

I'll try to play them tonight, i can't remember what happened last time i tried.

Myrantz - I'm not sure what you're asking, i use legit windows, jriver (latest) and the rips were nothing out of the ordinary. Tell me if that doesn't answdr your question.

Share this post


Link to post
Share on other sites


Guest myrantz

I'll try to play them tonight, i can't remember what happened last time i tried.

Myrantz - I'm not sure what you're asking, i use legit windows, jriver (latest) and the rips were nothing out of the ordinary. Tell me if that doesn't answdr your question.

normal copy as in copying via Windows Explorer, which I think you are...You've answered all my questions.. Thanks..

Which is why I'm paranoid, coz your workflow is similar to mine (except I use EAC). Will be good if you can download flac.exe and then just manually run that test and see if it says ok or not.. And then play it back and see..

Hoping the error is a false positive from jriver.

Share this post


Link to post
Share on other sites

Okay I had a look. Each of the two tracks are 23kb in size, so this must mean they never ripped properly in the first place - as mentioned DBpoweramps checksum feature was turned off so it must have just slipped under the radar.

 

I guess that's not much help to you, sorry dude.

Share this post


Link to post
Share on other sites

A good primer on the issue http://en.wikipedia.org/wiki/Data_corruption#SILENT

 

ZFS is good for data integrity http://en.wikipedia.org/wiki/ZFS#Data_integrity

If you are running linux as your NAS you can use BTRFS, also good for maintaining data integrity http://en.wikipedia.org/wiki/Btrfs#Checksum_tree_and_scrubbing If there is a problem with the file system there will be an entry in the logs

 

The easiest way I have found to find a corrupt file is using my Linn DS. It will simply stop playing a flac file and move to the next in the playlist when there is an error, this is usually pretty obvious and you will know to check and replace the file if damaged.

 

I use linux and BTRFS and span the volume across multiple spindles. I also use flac and test my library with the flac -t option perhaps once a year and used to find the occasional file, however it was not consistent and was related to other computer hardware not the disk itself. I moved to the setup mentioned and have seen none of it since

 

Cheers, Nick

Edited by nandrzej

Share this post


Link to post
Share on other sites
Guest myrantz

A good primer on the issue http://en.wikipedia.org/wiki/Data_corruption#SILENT

 

ZFS is good for data integrity http://en.wikipedia.org/wiki/ZFS#Data_integrity

If you are running linux as your NAS you can use BTRFS, also good for maintaining data integrity http://en.wikipedia.org/wiki/Btrfs#Checksum_tree_and_scrubbing If there is a problem with the file system there will be an entry in the logs

You need ECC memory for this to work reliably.. Else the silent errors will propagate when you scrub, and apparently you'd lost the whole volumes...  Not sure how true this is (odds wise), but scared me enough to switch to a machine with ECC RAM... :P

Share this post


Link to post
Share on other sites

You need ECC memory for this to work reliably.. Else the silent errors will propagate when you scrub, and apparently you'd lost the whole volumes...  Not sure how true this is (odds wise), but scared me enough to switch to a machine with ECC RAM... :P

 

Good call. You would have to be unlucky to have two things go wrong at the same time, a failure in disk and a coresponding failure in RAM whilst executing the volume fix. What did Murphy say?

 

It makes sense though especially as the topic is integrity and the argument I heard was that it is becoming incresingly important as machines have more and more RAM.

Share this post


Link to post
Share on other sites

I recently come across the same situation of corrupted files, where the files I played have produced patches of noise.

 

In all cases so far it has been problems with the initial rip from CD using EAC, that is basically faulty CD's, the count of faulty CD"s so far is low (~2) and then only a few tracks (~5 out of >10,000).

 

The flac.exe (https://xiph.org/flac/download.html) integrity check will not find this type of initial rip corruption as it checks if the encoded data has changed since encoding, this does not help if the original rip has noise in it, as the noise is encoded as well.

 

Windows users can use WinMerge (http://winmerge.org/) to compare files / directories, handy to check files on different HDD's if there is some suggestion of files being corrupted during copying.

 

Some copiers have a verify option eg. FastCopy (http://ipmsg.org/tools/fastcopy.html.en) and there is hash check software available eg. QuickHash (https://sourceforge.net/projects/quickhash/) for use when copying and data integrity checking.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...