Jump to content

Recommended Posts

This thread/question is largely directed at Drizt, as he is clearly a true believer in the method of blind testing of hifi components. My question is WHY???

I'm not a firm believer one way or the other and I do my best to evaluate a possible upgrade being evaluated, but to me blind testing is not the holy grail of best practice for identifying differences in what we hear. Actually, I'm not sure what is! Maybe it is just sitting and listenning in familiar surroundings.

Blind testing relies on our memory for comparrison purposes, which makes the method open to failure in my opinion.

I recently read on this forum about wine buff's not being able to identify the difference between red and white wine when doing a DBX test. But there IS a difference between red and white wine. This tells me it is the testing method which is flawed.

What do you all think?

Link to post
Share on other sites
  • Replies 114
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

the simple answer Phil is that we ARE NOT immune to our preconceptions I'm afraid.

Don't think we can be swayed by many subtle means? There is a (god knows how many billions) industry out there called advertising and PR which will prove the opposite.

Properly set up experiments PROVE that we are swayed by appearances, reputation and suggestion. That is a FACT and not in dispute here m'k?

These factors can and DO change our reported AURAL experiences. In other words the exact same component can sound different when it's identity is known/unknown.

So to be able to judge a component on it's audio merits ALONE we need to do it blind, and all that means is we don't know what it is. Or if it is a direct swap between two components we don't know which is which (or preferably not even know the identity of the components).

The double is so that the person doing administering the demo is unable to give you subtle clues or influence your decision one way or the other.

Salesman " Well Ok that was nice, now hear the top of the line from the amazingly well reviewed manufacturer and see what clarity in hi fi really means''.

Floyyd Toole has done the experiments with loudspeakers (we tend to only think of the little things like cables and interconnects etc when we think of DBT, it applies to everything) and the well known expensive brands when tested sighted consistently scored highly, yet when the comparisons were done masked often very much cheaper speakers scored higher in how they actually sound.

Reviews have a lot to answer for, recently there is a thread where an obscenely expensive valve amp was reviewed in stereophile and was raved about despite the measurements.

Of course a few responded saying 'see, measurements are not the whole part of the story, cause in spite of them the reviewer raved' (and unspoken assumptions, the reviewer knows what he's doing, his opinion can be respected etc etc. Heck, he uses Wilson Watts as the reference, so how can he have any judgement??)

But what sort of a scientific review was it in actuality?? Crikey, it looked a million dollars (even though it cost only half that), he already knew exactly hoe much it cost!! And HE was entrusted with the sacred duty of reviewing them (he proudly thinks to himself as his chest expands both with pride and delight), I mean no mere mortal on earth could aspire to hear them ever let alone own them.

C'mon, the entire reviewing process is flawed from the start. Be honest, is it at all even slightly possible that there may have been a different outcome if a reviewer is only given a black box. "here is the thing I need you to review next, it is an amplifier" (as the reviewer needs to know how to put it in his system, it only has input and output lugs visible, there is absolutely no other information available about what's inside).

Don't think a reviewer is above any of us and immune from unconsciuos biases. The ONLY way to review a product is as above.

And down the feeding chain it goes....are you going to tell me you would be totally able to divorce yourself from any influence of any sort when you were to evaluate a half million dollar amp in your system?? that you know got extremely rave reviews from everyone?

I don't think you can honestly say yes.

Drizt seems to have pissed a few off lately, I can understand that as I'm the Godfather of pisser offs, and he is but my mere apprentice heh heh, but c'mon you 're not trying to suggest that your conclusions about the audio worth of any component is lessened by not knowing what it is??????? are you?

On the contrary it could ONLY be contaminated by that knowledge.

Link to post
Share on other sites
I recently read on this forum about wine buff's not being able to identify the difference between red and white wine when doing a DBX test. But there IS a difference between red and white wine. This tells me it is the testing method which is flawed.

What do you all think?

Hello LD,

Yes sure the method could be flawed, but as an example the measuring equipment (the wine buffs) could equally be flawed. The same would apply to any blind test: audio, wine or whatever. As in any experiment the equipment needs to be tested and calibrated for the results to hold firm against criticism. Doing this with people would be interetsing..... taste tests, hearing tests etc..............

Of course no-one needs blind testing to indicate their OWN preference or to dictate what equipment they should buy or upgrade, that is personal and a matter of their own opinion....., BUT if one wants to test a premise like:

"All amplifiers sound the same",

"Cables make no difference to the sound" or

"All wine tastes the same"

AND if one wants to publish the results and have them stand up to scientific and public scrutiny then care needs to be taken with the selection of method, equipment at its calibration.

Best

JA

Link to post
Share on other sites
Hello LD,

Yes sure the method could be flawed, but as an example the measuring equipment (the wine buffs) could equally be flawed. The same would apply to any blind test: audio, wine or whatever. As in any experiment the equipment needs to be tested and calibrated for the results to hold firm against criticism. Doing this with people would be interetsing..... taste tests, hearing tests etc..............

Best

JA

i like that point. Floyd Toole goes into it and the results speak for themselves.

its a pitty that its rarely brought up with the debate about DBT's. just shows how little most ppl understand how DBT's work, both for and against the tests.

LD, perhaps u should read up about Floyd Toole's tests.

Link to post
Share on other sites

A DBT is so much simpler than what people make it out to be. It a method which allows a pretender to try and prove what he/she is saying, is true. A way to shut up the nay sayers. If you can't pass a DBT then you just have to live with the fact that you can't and certain people won't believe what you say because you can't. It doesn't mean you've gone coo coo or anything.

We shouldn't worry about what we CAN'T prove, it's irrelevant, null, void. We just need to worry about what we CAN prove, that's it.

Link to post
Share on other sites

For me, blind testing is useless as my audio memory is less than that of a gold fish and as unreliable as a politician and believe me, I have tried. I can't pick the difference between cables or sources as I can't remember what the previous one sounded like, unless you are able to instantly switch back and forth between them.

So for me, it basically comes down to sometimes I like what I hear, and sometimes I don't. And that bugs the crap out of me as I have an engineering back ground and like to see fact, figures and test data to confirm things.

So for me, blind testing is not flawed, I am .

Link to post
Share on other sites

I agree with some of what you are saying here LD. There must be a certain threshold of what a person can detect or remember. 98% vs 99% sound quality might be very hard to pick. Changing one pixel's colour from green to blue per frame in digital video for example, would probably be impossible to pick in a DBT. But we know there is/was a difference, because we changed it.

This thread/question is largely directed at Drizt, as he is clearly a true believer in the method of blind testing of hifi components. My question is WHY???

I'm not a firm believer one way or the other and I do my best to evaluate a possible upgrade being evaluated, but to me blind testing is not the holy grail of best practice for identifying differences in what we hear. Actually, I'm not sure what is! Maybe it is just sitting and listenning in familiar surroundings.

Blind testing relies on our memory for comparrison purposes, which makes the method open to failure in my opinion.

I recently read on this forum about wine buff's not being able to identify the difference between red and white wine when doing a DBX test. But there IS a difference between red and white wine. This tells me it is the testing method which is flawed.

What do you all think?

Link to post
Share on other sites

Terry has already covered a lot of it. I would like to add a lot more, but im busy at work today.



To me its pretty simple.

People convince themselves of things that aren't really there, its the placebo effect. Some people end up convincing themselves that they hear things that simply are not there. This then comes back to whether or not you are willing to live in ignorance and be happy with that. Thats a personal question.

To be honest I can not for the life of me understand how anyone can say that sighted testing is better than un-sighted (blind) testing... (ps. that doesn't mean you don't use your eyes it just means that you don't know what is being tested.)

The whole point of blind testing is to remove our preconceived ideals so that we can come to conclusions based on what our EARS hear, not what we want them to hear.

If you can not detect a difference (for good or bad) between two pieces of the equipment under proper blind test conditions then to YOU there is no difference in THAT particular test/environment/equipment. If you then do the same test but you know what is being tested and you can tell a difference then its clear that you have FOOLED yourself into hearing a difference. Then again it comes back to the question of is ignorance bliss?

BLIND TESTING MEANS: That you do not know what is being tested. Thats it. Its not magic. It does not ruin the musical experience. I do not understand how anyone can be against it or not understand how its better than sighted testing. You can do a blind test in any way you want for as long as you want, it does not matter. The only thing that matters is that you do NOT know what is being tested.

Link to post
Share on other sites

Blind testing relies on our memory for comparrison purposes, which makes the method open to failure in my opinion.

But what is more reliable? Testing that is done when you know what you are listening to or when you don't know what you are listening to? Can you honestly say that you have never fallen pray to hearing what you want to hear? No one is immune to that. And the only way to remove that is to do blind testing.

And remember blind testing is to PROVE there is a difference, not disprove. You can NEVER disprove something through a blind test. You can only PROVE there is a difference.

Link to post
Share on other sites

And remember blind testing is to PROVE there is a difference, not disprove. You can NEVER disprove something through a blind test. You can only PROVE there is a difference.

NULL hypothesis testing, is kinda interesting to me now. I never use to subsribe to it, but I think my opinion is changing somewhat. A NULL hypothesis for example would be white wine = red wine using LD's example, or 320kps MP3 = CD. You can never prove anything with a NULL hypothesis, but you'll have some statistics.

Link to post
Share on other sites

I guess you can count me amoungst the supporters of blind testing. I think Terry has pretty much covered the reasons ( in his own inimitable style ) in his post above.

I cant see how knowing what you are listening through makes you more impartial and better able to judge a components quality and I cant see how not knowing what you are listening through makes it more difficult to respond to music you are hearing.

Having said this I can see that blind tests are very difficult to do properly and just not practical in the vast majority of situations.

I dont think that there anything wrong with just listening and having an opinion that it sounds good , or bad, or whatever but there can be problems where people want to make generalizations from such a situation.

JA has a valid point about a listener being part of the test equipment in listening tests. Unless a listener can demonstrate a reliable ability to be able to distinguish between two components then I would think they not in a position to make a better/worse call. I would think it is pretty much a waste of effort to run blind tests with listeners who dont hear a difference sighted.

I think it is a real pity that blind tests are so often perceived as something negative used to try and prove people wrong . Blind tests would be better used to investigate the correlations between what we hear and characteristics of components ( as per the Harman group etc ) leading to improved playback equipment. Cheers Mike

Link to post
Share on other sites
NULL hypothesis testing, is kinda interesting to me now. I never use to subsribe to it, but I think my opinion is changing somewhat. A NULL hypothesis for example would be white wine = red wine using LD's example, or 320kps MP3 = CD. You can never prove anything with a NULL hypothesis, but you'll have some statistics.

Yes thats right. You can come to the conclusion that its highly unlikely there is a difference if a large enough sample population can not detect a difference. But you can never disprove it altogether.

Link to post
Share on other sites

Yes Keith, I do think this has been done to death, but as the issue keeps finding it's way into so many threads, I thought, well lets discuss it!

I like the way Rob323 described it, "So for me, blind testing is not flawed, I am". This is what I am relating to. To me it is the audio memory which detracts from blind testing as THE method.

I would also like to remove the most subtle tweaks like cable from the debate, as they are often picked on, rightly or wrongly, but understanding the huge and valid point of pre-conceived predjudices is not under question by me, but many many many times when I have compared CD players, amps and speakers, the differences are at times pretty plainly obvious. So why the further need for a more scientific testing method?

I sometimes think that the pro BD testers maybe pay little attention to the spacial qualities of a system, ie, how the performance is presented in front of you and that they are more concerned with the raw sound comparrison only (hope that makes sense). Could there be some who thinks everything sounds equal?

Also, Drizt has never pissed me off, ever, but I must admit the notion of being in ignorant bliss (or similar) for not using DBT as a method is unfortunate. I appreciate his passion for this pursuit we are sharing.

Link to post
Share on other sites

Yes thats right. You can come to the conclusion that its highly unlikely there is a difference if a large enough sample population can not detect a difference. But you can never disprove it altogether.

It's might not just be about whether it's highly unlikely with a NULL hypothesis. It can be about how many people can actually pass it an how many cannot. It might be 50/50 of the population with the examples above. You definitely can't assume that everyone has the same hearing ability, or the same "hifi trained ears" ability.

Link to post
Share on other sites

Hi LD

What's up? Bored or something? Not enough sport on TV?:)

FWIW, I reckon Terry has said it very well.

I'm happy to acknowledge/accept that my subjective opinions can be swayed very substantially by factors that I'd rather keep out of the equation (price, brand cachet appearance etc etc). So, I like the idea of evaluating sound quality without those influences. Of course they can be taken into account after the sound has been assessed.

But I think there are some things that need to be said (again):eek:

1. The proponents of objective testing have no desire to shove it down the throats of those who are opposed.

2. The "measurement" vs "what it sounds like" argument is a totally different argument. Anyone who thinks that is what is being proposed hasn't been paying attention.

3. Listening in the dark or with blindfolds is NOT what this is about. It's simply about not knowing exactly what is being listened to so that the pre-conceived expectations are taken out of the sound evaluation process. How this is achieved is irrelevant to the concept.

4. Most of the proponents have probably done very little in the way of DBT......at least in relation to audio. Because it ain't as easy as some people think. However, the proponents are simply suggesting it might be worth a go and it could be both fun and educational.

5. There is no expectation of "proving" anything or swaying the opinions of others.....especially those opposed to the concept.

6. Blind testing is not necessarily all about quick listens. For example, it's possible for someone to have two identical-looking components, but one has been highly modified internally. They can be labelled A and B. The person can then listen to them for weeks.....swapping one for the other (or having the swapped by another person) at random. If, at the end of as long as they need to come to a conclusion, they are convinced there is no difference, then they may decide the expensive modifications in one of the components are not worth the outlay to them in that system. Or they may decide they are because they can afford them and they provide "peace of mind" or they want the "bragging rights" or.......... The important point is that they will at least know that the unmodified component is pretty much on a par with the modified one under those conditions. Of course, it's also possible that they will find their listening enjoyment is greater at some times than others and this co-incides with component A being in the system. This suggests component A genuinely sounds better, but does not "prove" it as it could conceivably be all down to coincidence. This is only an example of the principle. It's a truth-seeking quest. An anti-hype quest.

7. There is nothing to fear.

It both saddens and amazes me that this topic can cause bitter division. That makes no sense. Why not accept that some people like the idea of evaluating pure sound quality devoid of outside influences and some people are uncomfortable with that concept and don't want to know about it. Maybe they aren't as easily swayed by price, appearance etc, maybe they are not willing to accept that possibility or maybe they are happy for that to be the case. Whatever, if it works for them, that's OK.

Fair enough, I say. It's all part of the "live and let live" philosophy most of us would support. It's not "live and let live as long as you agree with me":rolleyes:

Cheers

Tony

Link to post
Share on other sites

1. The proponents of objective testing have no desire to shove it down the throats of those who are opposed.

Well I've read many who have tried to shove it down the throats of those who are opposed. And there are those who might be trying to sell you something too.

Link to post
Share on other sites

1. The proponents of objective testing have no desire to shove it down the throats of those who are opposed.

Then why is it mentioned in every thread?

Link to post
Share on other sites

I like the way Rob323 described it, "So for me, blind testing is not flawed, I am". This is what I am relating to. To me it is the audio memory which detracts from blind testing as THE method.

But there are many DBT's that you and Rob323 will pass. So are you both flawed?

Link to post
Share on other sites

This is what I am relating to. To me it is the audio memory which detracts from blind testing as THE method.

but many many many times when I have compared CD players, amps and speakers, the differences are at times pretty plainly obvious.

your contradicting urself.

on one hand ur saying that DBT isn't the be all and end all because we cant remember how something sounds.

but then u go and say that sighted tests are somehow different in that u can remember how something sounds?

Link to post
Share on other sites
Hi LD

FWIW, I reckon Terry has said it very well.

I think terry j and Drizt are getting too caught up in the reasons why people might fail or why we need to do DBT's. It's irrelevant and if a person fails you have NO reason why they have failed. Linking failure with the placebo effect or whatever is just speculating.

Link to post
Share on other sites
Don't drag Luckydog down to my level, there's only enough room in this gutter for one :eek:

LOL, it's nice to see we can discuss this without guns being pulled.

Link to post
Share on other sites
I think terry j and Drizt are getting too caught up in the reasons why people might fail or why we need to do DBT's. It's irrelevant and if a person fails you have NO reason why they have failed. Linking failure with the placebo effect or whatever is just speculating.

Well said Dr X :eek:

Link to post
Share on other sites
Then why is it mentioned in every thread?

Hmm, a bit of hyperbole there, Keith?:eek:

I certainly would not go near the topic now unless I was responding to a question in a specific thread;)

Cheers

Tony

BTW, any mention of "proving" and individuals "failing" causes me some concern as neither word sits comfortably with my concept of this subject.

Link to post
Share on other sites
Hmm, a bit of hyperbole there, Keith?:eek:

Not at all. Do a search for the posts made by our most well known double blind testing proponent and see how many different threads you come up with.

I am not a Tremella aurantialba by any means. Guess that excludes me from DBT's :)

Link to post
Share on other sites
your contradicting urself.

on one hand ur saying that DBT isn't the be all and end all because we cant remember how something sounds.

but then u go and say that sighted tests are somehow different in that u can remember how something sounds?

Excellent question!

I think my answer is because there might be OTHER factors at play in a DBT. They could include, looking like a fool if I don't pick the expensive one - this is pressure which is not present in normal listenning.

Is the person in the right frame of mind to listen? To me this has a huge bearing on the enjoyment level able to be reached.

My point was that many comparrisons are pretty obvious. Repeatable and obvious.

Link to post
Share on other sites
Excellent question!

I think my answer is because there might be OTHER factors at play in a DBT. They could include, looking like a fool if I don't pick the expensive one - this is pressure which is not present in normal listenning.

Is the person in the right frame of mind to listen? To me this has a huge bearing on the enjoyment level able to be reached.

My point was that many comparrisons are pretty obvious. Repeatable and obvious.

If you umm.......DONT PASS the test, it doesn't mean that you can't have another go at some later date. Maybe then you'll get over your fears and have had more time to "ear learn" your material.

Link to post
Share on other sites
Excellent question!

I think my answer is because there might be OTHER factors at play in a DBT. They could include, looking like a fool if I don't pick the expensive one - this is pressure which is not present in normal listenning.

Is the person in the right frame of mind to listen? To me this has a huge bearing on the enjoyment level able to be reached.

My point was that many comparrisons are pretty obvious. Repeatable and obvious.

i still cant see how there is any difference between the DBT and sighted tests tho with those points? they are both exactly the same. the only difference i can see is that the person cant just choose the expensive one because it is expensive. he has to choose it because of what it sounds like. he still might have that pressure on him to choose the expensive one, but i cant see how its relevant to the test.

My point was that many comparrisons are pretty obvious. Repeatable and obvious

but the question is why are they obvious. is it because they sound different, or is it because the price tags are different?

thats the whole point of a DBT

Link to post
Share on other sites
If you umm.......DONT PASS the errr......TRIAL, it doesn't mean that you can't have another go at some later date. Maybe then you'll get over your fears and have had more time to "ear learn" your material.

I either hear a difference or I don't. The difference, if heard, will either make me want to buy or it wont. Couldn't give a hoot if the components being compared have a 'Levinson' or 'Wintal' badge on them.

Link to post
Share on other sites
I either hear a difference or I don't. The difference, if heard, will either make me want to buy or it wont. Couldn't give a hoot if the components being compared have a 'Levinson' or 'Wintal' badge on them.

I wasn't trying to make fun of you LD. I was trying to be funny. :eek: Someone previously said they didn't like the word "failing".

Link to post
Share on other sites
Not at all. Do a search for the posts made by our most well known double blind testing proponent and see how many different threads you come up with.

I am not a Tremella aurantialba by any means. Guess that excludes me from DBT's :eek:

Well, Keith, I must admit I had to google that.......and the initial info re a jellyfish didn't help much, until I noticed the common name of that critter.:)

Anyway, I'm convinced you're definitely more of a TA than I am and you're probably capable of listening more objectively than most :)

Perhaps the concept of objective testing is more attractive to those of us who are less skilled......I don't know:confused:

Cheers

Tony

Link to post
Share on other sites
I either hear a difference or I don't. The difference, if heard, will either make me want to buy or it wont. Couldn't give a hoot if the components being compared have a 'Levinson' or 'Wintal' badge on them.

The problem is all the factors at play. What if one CD player has 3V output compared to another which only has 2V. The one with 3V should sound louder. And yes I admit that I'm speculating, that those kind of things could cloud your judgment.

Link to post
Share on other sites
i still cant see how there is any difference between the DBT and sighted tests tho with those points? they are both exactly the same. the only difference i can see is that the person cant just choose the expensive one because it is expensive. he has to choose it because of what it sounds like. he still might have that pressure on him to choose the expensive one, but i cant see how its relevant to the test.

Fair call. How many DBT's have you been involved in. I suspect the pro DBTers would not have done many. Do you do them in the audio shops you visit? Do you do them with your car audio?

but the question is why are they obvious. is it because they sound different, or is it because the price tags are different?

thats the whole point of a DBT

They are obvious because the differences stand out. They are not the same!

Link to post
Share on other sites

This is going to sound bad, but its only an observation.......

Has anyone ever noticed that the ones that are most heavily against Blind testing are those that have already spent a lot of money on their hifi toys?

Link to post
Share on other sites
Fair call. How many DBT's have you been involved in. I suspect the pro DBTers would not have done many. Do you do them in the audio shops you visit? Do you do them with your car audio?

i've probably done 10 or 15 and that was all soley about source material. ie, differences in mp3, and at what bitrate etc. not many obviously.

why do u suspect that the pro DBTers havent done many?

They are obvious because the differences stand out. They are not the same!

but the point of a DBT is to determine if those differences are actual audible differences or if the differences were something else like the 'placebo' effect.

Link to post
Share on other sites

They are obvious because the differences stand out. They are not the same!

LuckyDog, could I make a suggestion. Perhaps put yourself through a couple proper blind tests (read level matched etc... with the help of a friend) to see if these obvious differences are detectable.

The true benefit of blind testing is generally not realised until you have gone through the process yourself.

I've done lots of blind testing... cables, CDP's, DAC's, Integrated amps etc. Some of those 'obvious' differences i thought I heard under sighted tests were simply not there under proper test conditions.

Link to post
Share on other sites
but the point of a DBT is to determine if those differences are actual audible differences or if the differences were something else like the 'placebo' effect.

Not quite, it's just to allow someone the chance to prove what they claim, that's it.

If they fail you have NO proof they are:

(1) Lying

(2) Sufferring from the placebo effect

(3) Crazy

(4) Everything else in the known universe

Link to post
Share on other sites
Guest
This topic is now closed to further replies.
  • Recently Browsing   0 members

    No registered users viewing this page.



×
×
  • Create New...