Jump to content

Do you maintain digital file integrity?


Recommended Posts

When using EAC, I added an extra option "-V" to the command line for encoding (-V stands for verify)

 

You can probably do the same with other GUI apps - e.g. with foobar, just create a new custom converter and supply the -V as one of the arguments. I'm sure you can do the same with JRiver (as long as you're using the flac binary, and the wrapper is able to trap the returned exit codes)

 

 

after you're done, once in a while you can whip up a script to check the integrity with -t flag... 

 

 

See here for full options. 

 

And i can do this on either ALAC and AIF files? I am not ripping from or to FLAC

Link to post
Share on other sites
  • Replies 72
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

There's "one" in every thread

I am keen to reach out to this audience to find out what (if anything) you might do to maintain your digital file integrity? I have just completed an arduous process of converting all 32,000 digital f

You are right, a real time "on the fly" integrity checker would be great. Not sure whether one exists..until something better comes along testing the FLAC and LAME files with audio tester ( freeware )

Guest myrantz

And i can do this on either ALAC and AIF files? I am not ripping from or to FLAC

Not sure about that as I don't use that format.. Google search gives me this: Linky.

So at least with dbpoweramp it can verify the encoding, but that's ripping, not converting.. You have to find out how your converting software (be it jriver, etc) does the encoding, and if it uses a binary executable, see if it supports testing/verifying.. 

Do you know if JRiver is using afconvert? qaac or something else? I'm trying to look for commandline arguments for that but can't find it quickly..

Link to post
Share on other sites

Are you using Mac or PC. Also how did you transfer the files from HD to NAS, just drop a massive folder or use a program to clone the disk

Link to post
Share on other sites

Not sure about that as I don't use that format.. Google search gives me this: Linky.

So at least with dbpoweramp it can verify the encoding, but that's ripping, not converting.. You have to find out how your converting software (be it jriver, etc) does the encoding, and if it uses a binary executable, see if it supports testing/verifying.. 

Do you know if JRiver is using afconvert? qaac or something else? I'm trying to look for commandline arguments for that but can't find it quickly..

 

Not too sure if there is a command line interface with J River. I have never used it if there is only what is available in the gui app.

Link to post
Share on other sites


Are you using Mac or PC. Also how did you transfer the files from HD to NAS, just drop a massive folder or use a program to clone the disk

 

I am using PC to rip and convert and  also use a Mac to connect to the NAS to be able to read the files to play.  The file format on the NAS can be read by both PC and MAC but only write with PC.

 

I did a staged drop from HDD to NAS when i moved the files across. By staged this was done by alphabetical order - so i didn't attempt to transfer all of the files in the same transaction - and this was along time ago and would of easily have been less than 50% of the number of files i have on my NAS now. The number of files then were much more manageable. Since then all of the files have been directly ripped directly to NAS using the ALAC format. I have just converted my library to AIF and in the process of correcting all of the corrupt files found. An arduous time consuming process

Link to post
Share on other sites
Guest myrantz

Not too sure if there is a command line interface with J River. I have never used it if there is only what is available in the gui app.

Looking at JRiver manual: http://wiki.jriver.com/index.php/Encoding_Settings

you just need to pick "external encoder" and set up the encoder manually, just a matter of finding a ALAC encoder that supports verification (if one exists)... If there's no support for this you can only do manual checks afterwards I guess.

  • Like 1
Link to post
Share on other sites

Looking at JRiver manual: http://wiki.jriver.com/index.php/Encoding_Settings

you just need to pick "external encoder" and set up the encoder manually, just a matter of finding a ALAC encoder that supports verification (if one exists)... If there's no support for this you can only do manual checks afterwards I guess.

 

And that is my point. The iTunes player must have some built-in verification. As i receive a message (ERR: File format not recognised) when it attempts to decode the corrupted file from ALAC to AIF (and i have done nothing special to the iTunes app other than just use the ALAC to AIF file converter utility that comes with the app). Whereas the J River player when using it's conversion utility converts the corrupt ALAC file to a corrupt AIF file with no message what so ever. A bug in their converter if you ask me or a poorly designed piece of utility software?

Link to post
Share on other sites
Guest myrantz

And that is my point. The iTunes player must have some built-in verification. As i receive a message (ERR: File format not recognised) when it attempts to decode the corrupted file from ALAC to AIF (and i have done nothing special to the iTunes app other than just use the ALAC to AIF file converter utility that comes with the app). Whereas the J River player when using it's conversion utility converts the corrupt ALAC file to a corrupt AIF file with no message what so ever. A bug in their converter if you ask me or a poorly designed piece of utility software?

A bit of everything I guess... Although to be fair to them it's hard to determine where the problem lies - e.g. have you do a RAM test as suggested to make sure they're fine?

 

Bug could be anywhere, JRiver, OS, etc... JRiver may well be doing everything correct, but when writing, a condition/event may occur and the OS is writing the wrong bits (resource conflicts, etc).. 

Link to post
Share on other sites


A bit of everything I guess... Although to be fair to them it's hard to determine where the problem lies - e.g. have you do a RAM test as suggested to make sure they're fine?

 

Bug could be anywhere, JRiver, OS, etc... JRiver may well be doing everything correct, but when writing, a condition/event may occur and the OS is writing the wrong bits (resource conflicts, etc).. 

 

But if everything else is the same and iTunes works and J River does not with respect to file conversion what does that suggest? If it was an OS problem or a RAM problem - wouldn't you expect both media players to exhibit the same issue? As it stands J River converts a corrupted file to another corrupted file albeit in a different format - iTunes does not allow the corrupted file to be converted. Both apps installed on the same pc accessing the NAS via the same network etc - everything is the same other than the media players. iTunes works and J River - well J River does work it is converting the file - but i don't want a bad ALAC File to be converted to a bad AIF file - it is nice to be informed that the ALAC file being converted is corrupted - which is what iTunes does. Don't get me wrong i love J River. I use it to listen to my files and to manage my audio files but it is just something i recently noticed as part of this file conversion process.

Link to post
Share on other sites
Guest myrantz

If it was an OS problem or a RAM problem - wouldn't you expect both media players to exhibit the same issue?

I know enough about computers to expect nothing ... :P RAM problem isn't deterministic... Test it and see anyway... A lot of systems are running on bad RAM, and they are 100% working, until they're not...

 

As it stands J River converts a corrupted file to another corrupted file albeit in a different format - iTunes does not allow the corrupted file to be converted. Both apps installed on the same pc accessing the NAS via the same network etc - everything is the same other than the media players. iTunes works and J River - well J River does work it is converting the file - but i don't want a bad ALAC File to be converted to a bad AIF file - it is nice to be informed that the ALAC file being converted is corrupted - which is what iTunes does. Don't get me wrong i love J River. I use it to listen to my files and to manage my audio files but it is just something i recently noticed as part of this file conversion process.

Is there a reason for ALAC? Why not use FLAC? JRiver plays FLAC, and it supports verification on encode, and testing after encode... Best of both worlds?

There's no perfect solution out there.. remember this that stirred up a paranoia a while back? Silent errors are the worst kind of corruption, can't really avoid 'em.

Link to post
Share on other sites

I have one set of wav files, of which I have a copy on an external hard disk.

Any conversions I do (which is very rare), I do to anther internal disk and copy out from there.

 

Has anyone found some comparison software yet?

I'd be keen to compare my two sets of wav files.

 

That said, I don't seem to have come across any dud files so far.........all ripped via dbpoweramp on the most accurate setting.

Link to post
Share on other sites


Guest myrantz

Has anyone found some comparison software yet?

Comparison isn't quite the same as integrity... (you can compare two corrupt files and they will pass because they are bit for bit exact or have the same checksum etc)..

 

integrity checking ensures the file itself has no errors. (like testing ZIP files..), e.g. with flac

flac.exe -t test.flac
 
flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007  Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.
test.flac: ok

Caveat that the integrity check actually comes with absolutely no warranty.. but apparently it's well tested enough to be problem free... So far so good (*touch wood*)

 

  • Like 1
Link to post
Share on other sites

What about scratches and dust? Or the corruption going on in vinyl prices.

 

nah, not a real issue and I use a record cleaner, so all good.  Pricing is getting to be a rip off thou.

Edited by metal beat
Link to post
Share on other sites

I know enough about computers to expect nothing ... :P RAM problem isn't deterministic... Test it and see anyway... A lot of systems are running on bad RAM, and they are 100% working, until they're not...

 

Is there a reason for ALAC? Why not use FLAC? JRiver plays FLAC, and it supports verification on encode, and testing after encode... Best of both worlds?

There's no perfect solution out there.. remember this that stirred up a paranoia a while back? Silent errors are the worst kind of corruption, can't really avoid 'em.

 

I started ripping using ALAC years ago and stuck with that format. After a lot of reading and listening to different formats when i was deciding to look at uncompressed formats i decided to stay with the apple proprietary AIF format to convert all my ALAC files from. So the ALAC decision is more a legacy decision than anything else.

 

I have one set of wav files, of which I have a copy on an external hard disk.

Any conversions I do (which is very rare), I do to anther internal disk and copy out from there.

 

Has anyone found some comparison software yet?

I'd be keen to compare my two sets of wav files.

 

That said, I don't seem to have come across any dud files so far.........all ripped via dbpoweramp on the most accurate setting.

 

I am confident the ripping process has not actually caused the number of files that has become corrupted over time. I can't say for certain that not every file was ripped with 100% accuracy because i don't actually replay in its entirety every album that i have ripped and i guess not many people do. But for the most part i don't put the corruption down to ripping. I suspect it has probably more to do with the number of transfers i have made as i move my library from one disk to another as my library has grown over the years. I also suspect unexpected outages have also contributed to the problem. 

  • Like 1
Link to post
Share on other sites

Ive got two corrupted files of 7,000. I found out when i used Jriver to analyse the entire library for its dynamic range for the r128 volume normalisation feature (which i don't use anymore). No idea what caused it...

Link to post
Share on other sites


Ive got two corrupted files of 7,000. I found out when i used Jriver to analyse the entire library for its dynamic range for the r128 volume normalisation feature (which i don't use anymore). No idea what caused it...

 

Care to elaborate?

Link to post
Share on other sites
Guest myrantz

Sorry, on which point?

Container used: wave/flac/ALAC/ape, etc.

Source of files: Ripped from CD (if yes, which encoder), digital downloads

 

Where are files stored: NAS, USB HDD, local HDD, etc; What filesystem (NTFS, FAT32, etc)

What OS: Mac, Linux or Windows

 

Activities: Did you transfer files back/forth between mediums in the past? What tagging software did you use?

Doubt can find the cause of the corruption now, But if you can provide the above hopefully there'd be some clues...

Link to post
Share on other sites

Sorry, on which point?

 

Also keen to understand what you mean by "dynamic range for the r128 volume normalisation feature" - how can this be an effective tool to determine or pick out corrupted files?

Link to post
Share on other sites
Guest myrantz

Also keen to understand what you mean by "dynamic range for the r128 volume normalisation feature" - how can this be an effective tool to determine or pick out corrupted files?

R128 is a standard, normalisation is sorta like adjusting the volume (can be very simple like a volume control or more complicated)..

It's effective because normalisation will involves decoding the whole file to find the peak and average volumes (and so on).. You can probably do the same by running the dynamic range plugin and update the metadata tags with the DR number.. It's the decoding the file part that helps detect corruption..

Common things between you and him so far appears to be JRiver :P.

Link to post
Share on other sites

Hi jrisles,

 

There is a program called Perfecttunes which is software that you can get as an extra to dbpoweramp; or on its own.

https://www.dbpoweramp.com/perfecttunes.htm

 

Perfecttunes has three components:

 

DeDup: Checks for duplicates in your music library and removes them (with user controls).  This program is quite clever in that it allows you to listen to the tracks in questions and even switch the stream during playback.

 

Album art: Automatically adds missing album covers.

 

Accurate Rip: This will verify previously ripped tracks

 

From the webpage:

“PerfectTUNES AccurateRip verifies previously ripped discs, perfect for validating iTunes or other rippers not capable of checking AccurateRip.â€

 

If you want the ability to check rips that have not been ripped with Accurate Rip, or check on “data integrity†in these terms, then perhaps this may be something that you’re looking for.

 

Milo.

  • Like 2
Link to post
Share on other sites

Container used: wave/flac/ALAC/ape, etc.

Source of files: Ripped from CD (if yes, which encoder), digital downloads

 

Where are files stored: NAS, USB HDD, local HDD, etc; What filesystem (NTFS, FAT32, etc)

What OS: Mac, Linux or Windows

 

Activities: Did you transfer files back/forth between mediums in the past? What tagging software did you use?

Doubt can find the cause of the corruption now, But if you can provide the above hopefully there'd be some clues...

Ok so they were ripped onto ssd via dbpoweramp as a mildly compressed flac (later all compression was removed). I think i had the check feature turned off as i had 300 cds to rip, so the problem might have been right at the beginning. They were then transferred to another ssd when i outgrew the original. I can't remember if i did the original rips on w7 64 or w8 64, i think 7.

The other poster is right, i only became aware of the issue when i scanned all 7,000 files, at the end it said there werwere two left, so i investigated and thats when i realised... Hth.

Link to post
Share on other sites

Ok so they were ripped onto ssd via dbpoweramp as a mildly compressed flac (later all compression was removed). I think i had the check feature turned off as i had 300 cds to rip, so the problem might have been right at the beginning. They were then transferred to another ssd when i outgrew the original. I can't remember if i did the original rips on w7 64 or w8 64, i think 7.

The other poster is right, i only became aware of the issue when i scanned all 7,000 files, at the end it said there werwere two left, so i investigated and thats when i realised... Hth.

 

Did you attempt to replay the two files (lucky you) you found were corrupt? Could you play them? Mine are able to be played but during the song the file makes a static white noise and the file is then skipped with the player moving to the next song. You don't know the file has become corrupt until it is played. The issue has nothing to do with the ripping process for 99.9% of my files. So even with all the best error correction when ripping in the world this is not the actual problem. The problem is with this phenomena called bit decay or bit rot over the years i have had my files.

Link to post
Share on other sites
Guest myrantz

Ok so they were ripped onto ssd via dbpoweramp as a mildly compressed flac (later all compression was removed). I think i had the check feature turned off as i had 300 cds to rip, so the problem might have been right at the beginning. They were then transferred to another ssd when i outgrew the original. I can't remember if i did the original rips on w7 64 or w8 64, i think 7.

The other poster is right, i only became aware of the issue when i scanned all 7,000 files, at the end it said there werwere two left, so i investigated and thats when i realised... Hth.

Hmm. Interesting, and it's just normal copy in Windows?

Now I'm worried/paranoid.. Gonna re-do my integrity check and report back.

Link to post
Share on other sites

I'll try to play them tonight, i can't remember what happened last time i tried.

Myrantz - I'm not sure what you're asking, i use legit windows, jriver (latest) and the rips were nothing out of the ordinary. Tell me if that doesn't answdr your question.

Link to post
Share on other sites
Guest myrantz

I'll try to play them tonight, i can't remember what happened last time i tried.

Myrantz - I'm not sure what you're asking, i use legit windows, jriver (latest) and the rips were nothing out of the ordinary. Tell me if that doesn't answdr your question.

normal copy as in copying via Windows Explorer, which I think you are...You've answered all my questions.. Thanks..

Which is why I'm paranoid, coz your workflow is similar to mine (except I use EAC). Will be good if you can download flac.exe and then just manually run that test and see if it says ok or not.. And then play it back and see..

Hoping the error is a false positive from jriver.

Link to post
Share on other sites

Okay I had a look. Each of the two tracks are 23kb in size, so this must mean they never ripped properly in the first place - as mentioned DBpoweramps checksum feature was turned off so it must have just slipped under the radar.

 

I guess that's not much help to you, sorry dude.

Link to post
Share on other sites

A good primer on the issue http://en.wikipedia.org/wiki/Data_corruption#SILENT

 

ZFS is good for data integrity http://en.wikipedia.org/wiki/ZFS#Data_integrity

If you are running linux as your NAS you can use BTRFS, also good for maintaining data integrity http://en.wikipedia.org/wiki/Btrfs#Checksum_tree_and_scrubbing If there is a problem with the file system there will be an entry in the logs

 

The easiest way I have found to find a corrupt file is using my Linn DS. It will simply stop playing a flac file and move to the next in the playlist when there is an error, this is usually pretty obvious and you will know to check and replace the file if damaged.

 

I use linux and BTRFS and span the volume across multiple spindles. I also use flac and test my library with the flac -t option perhaps once a year and used to find the occasional file, however it was not consistent and was related to other computer hardware not the disk itself. I moved to the setup mentioned and have seen none of it since

 

Cheers, Nick

Edited by nandrzej
Link to post
Share on other sites
Guest myrantz

A good primer on the issue http://en.wikipedia.org/wiki/Data_corruption#SILENT

 

ZFS is good for data integrity http://en.wikipedia.org/wiki/ZFS#Data_integrity

If you are running linux as your NAS you can use BTRFS, also good for maintaining data integrity http://en.wikipedia.org/wiki/Btrfs#Checksum_tree_and_scrubbing If there is a problem with the file system there will be an entry in the logs

You need ECC memory for this to work reliably.. Else the silent errors will propagate when you scrub, and apparently you'd lost the whole volumes...  Not sure how true this is (odds wise), but scared me enough to switch to a machine with ECC RAM... :P

Link to post
Share on other sites

You need ECC memory for this to work reliably.. Else the silent errors will propagate when you scrub, and apparently you'd lost the whole volumes...  Not sure how true this is (odds wise), but scared me enough to switch to a machine with ECC RAM... :P

 

Good call. You would have to be unlucky to have two things go wrong at the same time, a failure in disk and a coresponding failure in RAM whilst executing the volume fix. What did Murphy say?

 

It makes sense though especially as the topic is integrity and the argument I heard was that it is becoming incresingly important as machines have more and more RAM.

Link to post
Share on other sites
  • 9 months later...

I recently come across the same situation of corrupted files, where the files I played have produced patches of noise.

 

In all cases so far it has been problems with the initial rip from CD using EAC, that is basically faulty CD's, the count of faulty CD"s so far is low (~2) and then only a few tracks (~5 out of >10,000).

 

The flac.exe (https://xiph.org/flac/download.html) integrity check will not find this type of initial rip corruption as it checks if the encoded data has changed since encoding, this does not help if the original rip has noise in it, as the noise is encoded as well.

 

Windows users can use WinMerge (http://winmerge.org/) to compare files / directories, handy to check files on different HDD's if there is some suggestion of files being corrupted during copying.

 

Some copiers have a verify option eg. FastCopy (http://ipmsg.org/tools/fastcopy.html.en) and there is hash check software available eg. QuickHash (https://sourceforge.net/projects/quickhash/) for use when copying and data integrity checking.

  • Like 1
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recently Browsing   0 members

    No registered users viewing this page.




×
×
  • Create New...