Jump to content

Do you maintain digital file integrity?


Recommended Posts



If I understand it correctly a corupted file can be copied, so backing up isn't really a solution. Everytime a file is being read, written moved it can corrupt and a HDD can start to fail as well.

Interesting problem here.

 

Yes of course you are right. If you are not aware that the file is corrupted in the first place and let's face it if you don't play all of your files every month or week or whatever frequency you have your backup strategy on you would never know until it is too late because the corrupted files would have already of course been backed up and by the time you realise it would already be too late in most instances having backed up the corrupted files time and time again before you have actually attempted to play it.

 

So this is not about a backup strategy as it does not overcome the problem. A program - as i mentioned - that acts like an antivirus program continuously checking your library files for potential corrupt audio files - a pro active approach rather than a reactive one. In this modern day of storage especially with video i am surprised that such a program doesn't yet exist? Perhaps it does?

Link to comment
Share on other sites



You are right, a real time "on the fly" integrity checker would be great. Not sure whether one exists..until something better comes along testing the FLAC and LAME files with audio tester ( freeware ) or perhaps DBPoweramp's Perfect Tunes ( at a cost ) at an interval of whatever time suits me is all I can think of.

The FLAC lossless does take overnight to fully test all the files and the LAME 320kbps 3 to 4 hours.

Usually do both at once on separate PC's...

If I did end up with a corrupted track or tracks permeating the backups...I would just get the original CD and re rip.

Also it can be a good idea to run perodic tests of the drives using Seagate or WD or ? Tools to make sure the drives are not about to fail..

The alternative all is Tidal, Deeza or Spotify or the like and let them worry about it, I guess....and then hope the your internet doesn't go down or is underperforming

Maybe there is someone out there with something to make it easier hopefully...

Gotta luv technology and its promise of freeing us up for more leisure time 😄

Edited by ummagumma
  • Like 2
Link to comment
Share on other sites

You are right, a real time "on the fly" integrity checker would be great. Not sure whether one exists..until something better comes along testing the FLAC and LAME files with audio tester ( freeware ) or perhaps DBPoweramp's Perfect Tunes ( at a cost ) at an interval of whatever time suits me is all I can think of.

The FLAC lossless does take overnight to fully test all the files and the LAME 320kbps 3 to 4 hours.

Usually do both at once on separate PC's...

If I did end up with a corrupted track or tracks permeating the backups...I would just get the original CD and re rip.

Also it can be a good idea to run perodic tests of the drives using Seagate or WD or ? Tools to make sure the drives are not about to fail..

The alternative all is Tidal, Deeza or Spotify or the like and let them worry about it, I guess....and then hope the your internet doesn't go down or is underperforming

Maybe there is someone out there with something to make it easier hopefully...

Gotta luv technology and its promise of freeing us up for more leisure time

What software do you run to check drives for impending failure?

Link to comment
Share on other sites

I'll refer you to this article on wikipedia http://en.wikipedia.org/wiki/Data_degradation

 

Over time degradation can become a serious problem for certain files with no inherent error checking and can result in small glitches like a skip in audio or video, larger glitches like a file skipping once it reaches a certain unrecoverable point or the file not being playable at all. The CD medium for all it's faults envisaged this type of problem with its error checking and recovery routines.

 

The only solution is to maintain backups so the file you ripped 10 years ago can be recovered after this happens, or keep the original media so you can re-rip (legalities aside this is the only reason I keep all my dvd's and cd's).

 

The larger the disk size, the more chance you have of a file being effected as a potential error rate of 0.0001% for example on a 4tb drive is easily enough to leave 1% of your files unplayable.

  • Like 1
Link to comment
Share on other sites



Be careful! RAID 5 works from at least 3 disks, to allow one to fail.

I have a NAS running RAID1 on a Linux system. A pain to configure, but I was able to read a disk from an older system on a normal Linux PC.

I haven't had the OP's problem though... I'd attribute something like that to my ripping process.

Link to comment
Share on other sites

Be careful! RAID 5 works from at least 3 disks, to allow one to fail.

I have a NAS running RAID1 on a Linux system. A pain to configure, but I was able to read a disk from an older system on a normal Linux PC.

I haven't had the OP's problem though... I'd attribute something like that to my ripping process.

 

I wouldn't say that it is wholly responsible due to the ripping process. I do know that for a fact that some albums i have been able to listen in their entirety just after they have been ripped with no file file degradation - that is to say all the files played played right through to the end. But i cannot rule out a ripping problem with all albums ripped. Given that my pc and laptops have changed over the years and i rarely listen to every single file ripped from an album as soon as it is ripped. So i have not been able to validate every file for every rip. But who can? I mean how many people actually listen to the whole album once you have just ripped it? My current RAID system is RAID1. According to Wiki on RAID systems RAID 5 is all but redundant?

 

Even RAID 6 wouldn't detect this problem as i understand it? RAID arrays are all about managing fault tolerance at a DRIVE level not at a file level.

 

Wiki Quotes;

RAID 5 RAID 5 consists of block-level striping with distributed parity. Unlike in RAID 4, parity information is distributed among the drives. It requires that all drives but one be present to operate. Upon failure of a single drive, subsequent reads can be calculated from the distributed parity such that no data is lost. RAID 5 requires at least three disks.[11] RAID 5 is seriously affected by the general trends regarding array rebuild time and the chance of drive failure during rebuild.[22] Rebuilding an array requires reading all data from all disks, opening a chance for a second drive failure and the loss of entire array. In August 2012, Dell posted an advisory against the use of RAID 5 in any configuration and RAID 50 with "Class 2 7200 RPM drives of 1 TB and higher capacity" for business-critical data.

 

RAID 6 RAID 6 consists of block-level striping with double distributed parity. Double parity provides fault tolerance up to two failed drives. This makes larger RAID groups more practical, especially for high-availability systems, as large-capacity drives take longer to restore. RAID 6 requires a minimum of four disks. As with RAID 5, a single drive failure results in reduced performance of the entire array until the failed drive has been replaced.[11] With a RAID 6 array, using drives from multiple sources and manufacturers, it is possible to mitigate most of the problems associated with RAID 5. The larger the drive capacities and the larger the array size, the more important it becomes to choose RAID 6 instead of RAID 5.[24] RAID 10 also minimizes these problems.

Edited by jrisles
Link to comment
Share on other sites

  • 2 weeks later...

Also given what i have discovered about the JRiver media player i am reluctant to use this to convert any file format from one format to another. There is no integrity checked as part of the file conversion process. I could not find any special setting on the player to provided some form of check sum or integrity check on the file. It was just fortuitous that i used the apples iTunes player in the first instance to convert from ALAC to AIF. When a corrupt file was in the process of being converted the apples iTunes program provided me with a message indicating the file format was not recognised and stop the conversion. So i always knew which file was corrupted in my library allowing me to segment those files. Whilst the J River program provided me with no such information. Something to consider when you are working out what program you must use to convert files from one program to another. I would not use J River for this process.

Link to comment
Share on other sites

Guest myrantz

Also given what i have discovered about the JRiver media player i am reluctant to use this to convert any file format from one format to another. There is no integrity checked as part of the file conversion process. I could not find any special setting on the player to provided some form of check sum or integrity check on the file. It was just fortuitous that i used the apples iTunes player in the first instance to convert from ALAC to AIF. When a corrupt file was in the process of being converted the apples iTunes program provided me with a message indicating the file format was not recognised and stop the conversion. So i always knew which file was corrupted in my library allowing me to segment those files. Whilst the J River program provided me with no such information. Something to consider when you are working out what program you must use to convert files from one program to another. I would not use J River for this process.

When using EAC, I added an extra option "-V" to the command line for encoding (-V stands for verify)

 

You can probably do the same with other GUI apps - e.g. with foobar, just create a new custom converter and supply the -V as one of the arguments. I'm sure you can do the same with JRiver (as long as you're using the flac binary, and the wrapper is able to trap the returned exit codes)

 

 

 

Verify the encoding process. With this option, flac will create a parallel decoder that decodes the output of the encoder and compares the result against the original. It will abort immediately with an error if a mismatch occurs. -V increases the total encoding time but is guaranteed to catch any unforseen bug in the encoding process.

 

after you're done, once in a while you can whip up a script to check the integrity with -t flag... 

 

 

-d, --decode  Decode (flac encodes by default). flac will exit with an exit code of 1 (and print a message, even in silent mode) if there were any errors during decoding, including when the MD5 checksum does not match the decoded output. Otherwise the exit code will be 0.

-t, --test  Test (same as -d except no decoded file is written). The exit codes are the same as in decode mode.

 

 

See here for full options. 

Link to comment
Share on other sites



When using EAC, I added an extra option "-V" to the command line for encoding (-V stands for verify)

 

You can probably do the same with other GUI apps - e.g. with foobar, just create a new custom converter and supply the -V as one of the arguments. I'm sure you can do the same with JRiver (as long as you're using the flac binary, and the wrapper is able to trap the returned exit codes)

 

 

after you're done, once in a while you can whip up a script to check the integrity with -t flag... 

 

 

See here for full options. 

 

And i can do this on either ALAC and AIF files? I am not ripping from or to FLAC

Link to comment
Share on other sites

Guest myrantz

And i can do this on either ALAC and AIF files? I am not ripping from or to FLAC

Not sure about that as I don't use that format.. Google search gives me this: Linky.

So at least with dbpoweramp it can verify the encoding, but that's ripping, not converting.. You have to find out how your converting software (be it jriver, etc) does the encoding, and if it uses a binary executable, see if it supports testing/verifying.. 

Do you know if JRiver is using afconvert? qaac or something else? I'm trying to look for commandline arguments for that but can't find it quickly..

Link to comment
Share on other sites

Not sure about that as I don't use that format.. Google search gives me this: Linky.

So at least with dbpoweramp it can verify the encoding, but that's ripping, not converting.. You have to find out how your converting software (be it jriver, etc) does the encoding, and if it uses a binary executable, see if it supports testing/verifying.. 

Do you know if JRiver is using afconvert? qaac or something else? I'm trying to look for commandline arguments for that but can't find it quickly..

 

Not too sure if there is a command line interface with J River. I have never used it if there is only what is available in the gui app.

Link to comment
Share on other sites

Are you using Mac or PC. Also how did you transfer the files from HD to NAS, just drop a massive folder or use a program to clone the disk

 

I am using PC to rip and convert and  also use a Mac to connect to the NAS to be able to read the files to play.  The file format on the NAS can be read by both PC and MAC but only write with PC.

 

I did a staged drop from HDD to NAS when i moved the files across. By staged this was done by alphabetical order - so i didn't attempt to transfer all of the files in the same transaction - and this was along time ago and would of easily have been less than 50% of the number of files i have on my NAS now. The number of files then were much more manageable. Since then all of the files have been directly ripped directly to NAS using the ALAC format. I have just converted my library to AIF and in the process of correcting all of the corrupt files found. An arduous time consuming process

Link to comment
Share on other sites



Guest myrantz

Not too sure if there is a command line interface with J River. I have never used it if there is only what is available in the gui app.

Looking at JRiver manual: http://wiki.jriver.com/index.php/Encoding_Settings

you just need to pick "external encoder" and set up the encoder manually, just a matter of finding a ALAC encoder that supports verification (if one exists)... If there's no support for this you can only do manual checks afterwards I guess.

  • Like 1
Link to comment
Share on other sites

Looking at JRiver manual: http://wiki.jriver.com/index.php/Encoding_Settings

you just need to pick "external encoder" and set up the encoder manually, just a matter of finding a ALAC encoder that supports verification (if one exists)... If there's no support for this you can only do manual checks afterwards I guess.

 

And that is my point. The iTunes player must have some built-in verification. As i receive a message (ERR: File format not recognised) when it attempts to decode the corrupted file from ALAC to AIF (and i have done nothing special to the iTunes app other than just use the ALAC to AIF file converter utility that comes with the app). Whereas the J River player when using it's conversion utility converts the corrupt ALAC file to a corrupt AIF file with no message what so ever. A bug in their converter if you ask me or a poorly designed piece of utility software?

Link to comment
Share on other sites

Guest myrantz

And that is my point. The iTunes player must have some built-in verification. As i receive a message (ERR: File format not recognised) when it attempts to decode the corrupted file from ALAC to AIF (and i have done nothing special to the iTunes app other than just use the ALAC to AIF file converter utility that comes with the app). Whereas the J River player when using it's conversion utility converts the corrupt ALAC file to a corrupt AIF file with no message what so ever. A bug in their converter if you ask me or a poorly designed piece of utility software?

A bit of everything I guess... Although to be fair to them it's hard to determine where the problem lies - e.g. have you do a RAM test as suggested to make sure they're fine?

 

Bug could be anywhere, JRiver, OS, etc... JRiver may well be doing everything correct, but when writing, a condition/event may occur and the OS is writing the wrong bits (resource conflicts, etc).. 

Link to comment
Share on other sites

A bit of everything I guess... Although to be fair to them it's hard to determine where the problem lies - e.g. have you do a RAM test as suggested to make sure they're fine?

 

Bug could be anywhere, JRiver, OS, etc... JRiver may well be doing everything correct, but when writing, a condition/event may occur and the OS is writing the wrong bits (resource conflicts, etc).. 

 

But if everything else is the same and iTunes works and J River does not with respect to file conversion what does that suggest? If it was an OS problem or a RAM problem - wouldn't you expect both media players to exhibit the same issue? As it stands J River converts a corrupted file to another corrupted file albeit in a different format - iTunes does not allow the corrupted file to be converted. Both apps installed on the same pc accessing the NAS via the same network etc - everything is the same other than the media players. iTunes works and J River - well J River does work it is converting the file - but i don't want a bad ALAC File to be converted to a bad AIF file - it is nice to be informed that the ALAC file being converted is corrupted - which is what iTunes does. Don't get me wrong i love J River. I use it to listen to my files and to manage my audio files but it is just something i recently noticed as part of this file conversion process.

Link to comment
Share on other sites

Guest myrantz

If it was an OS problem or a RAM problem - wouldn't you expect both media players to exhibit the same issue?

I know enough about computers to expect nothing ... :P RAM problem isn't deterministic... Test it and see anyway... A lot of systems are running on bad RAM, and they are 100% working, until they're not...

 

As it stands J River converts a corrupted file to another corrupted file albeit in a different format - iTunes does not allow the corrupted file to be converted. Both apps installed on the same pc accessing the NAS via the same network etc - everything is the same other than the media players. iTunes works and J River - well J River does work it is converting the file - but i don't want a bad ALAC File to be converted to a bad AIF file - it is nice to be informed that the ALAC file being converted is corrupted - which is what iTunes does. Don't get me wrong i love J River. I use it to listen to my files and to manage my audio files but it is just something i recently noticed as part of this file conversion process.

Is there a reason for ALAC? Why not use FLAC? JRiver plays FLAC, and it supports verification on encode, and testing after encode... Best of both worlds?

There's no perfect solution out there.. remember this that stirred up a paranoia a while back? Silent errors are the worst kind of corruption, can't really avoid 'em.

Link to comment
Share on other sites



  • Recently Browsing   0 members

    • No registered users viewing this page.




×
×
  • Create New...
To Top