Jump to content

Sign in to follow this  
Phil G

Blind Testing

Recommended Posts

This thread/question is largely directed at Drizt, as he is clearly a true believer in the method of blind testing of hifi components. My question is WHY???

I'm not a firm believer one way or the other and I do my best to evaluate a possible upgrade being evaluated, but to me blind testing is not the holy grail of best practice for identifying differences in what we hear. Actually, I'm not sure what is! Maybe it is just sitting and listenning in familiar surroundings.

Blind testing relies on our memory for comparrison purposes, which makes the method open to failure in my opinion.

I recently read on this forum about wine buff's not being able to identify the difference between red and white wine when doing a DBX test. But there IS a difference between red and white wine. This tells me it is the testing method which is flawed.

What do you all think?

Share this post


Link to post
Share on other sites

the simple answer Phil is that we ARE NOT immune to our preconceptions I'm afraid.

Don't think we can be swayed by many subtle means? There is a (god knows how many billions) industry out there called advertising and PR which will prove the opposite.

Properly set up experiments PROVE that we are swayed by appearances, reputation and suggestion. That is a FACT and not in dispute here m'k?

These factors can and DO change our reported AURAL experiences. In other words the exact same component can sound different when it's identity is known/unknown.

So to be able to judge a component on it's audio merits ALONE we need to do it blind, and all that means is we don't know what it is. Or if it is a direct swap between two components we don't know which is which (or preferably not even know the identity of the components).

The double is so that the person doing administering the demo is unable to give you subtle clues or influence your decision one way or the other.

Salesman " Well Ok that was nice, now hear the top of the line from the amazingly well reviewed manufacturer and see what clarity in hi fi really means''.

Floyyd Toole has done the experiments with loudspeakers (we tend to only think of the little things like cables and interconnects etc when we think of DBT, it applies to everything) and the well known expensive brands when tested sighted consistently scored highly, yet when the comparisons were done masked often very much cheaper speakers scored higher in how they actually sound.

Reviews have a lot to answer for, recently there is a thread where an obscenely expensive valve amp was reviewed in stereophile and was raved about despite the measurements.

Of course a few responded saying 'see, measurements are not the whole part of the story, cause in spite of them the reviewer raved' (and unspoken assumptions, the reviewer knows what he's doing, his opinion can be respected etc etc. Heck, he uses Wilson Watts as the reference, so how can he have any judgement??)

But what sort of a scientific review was it in actuality?? Crikey, it looked a million dollars (even though it cost only half that), he already knew exactly hoe much it cost!! And HE was entrusted with the sacred duty of reviewing them (he proudly thinks to himself as his chest expands both with pride and delight), I mean no mere mortal on earth could aspire to hear them ever let alone own them.

C'mon, the entire reviewing process is flawed from the start. Be honest, is it at all even slightly possible that there may have been a different outcome if a reviewer is only given a black box. "here is the thing I need you to review next, it is an amplifier" (as the reviewer needs to know how to put it in his system, it only has input and output lugs visible, there is absolutely no other information available about what's inside).

Don't think a reviewer is above any of us and immune from unconsciuos biases. The ONLY way to review a product is as above.

And down the feeding chain it goes....are you going to tell me you would be totally able to divorce yourself from any influence of any sort when you were to evaluate a half million dollar amp in your system?? that you know got extremely rave reviews from everyone?

I don't think you can honestly say yes.

Drizt seems to have pissed a few off lately, I can understand that as I'm the Godfather of pisser offs, and he is but my mere apprentice heh heh, but c'mon you 're not trying to suggest that your conclusions about the audio worth of any component is lessened by not knowing what it is??????? are you?

On the contrary it could ONLY be contaminated by that knowledge.

Share this post


Link to post
Share on other sites
I recently read on this forum about wine buff's not being able to identify the difference between red and white wine when doing a DBX test. But there IS a difference between red and white wine. This tells me it is the testing method which is flawed.

What do you all think?

Hello LD,

Yes sure the method could be flawed, but as an example the measuring equipment (the wine buffs) could equally be flawed. The same would apply to any blind test: audio, wine or whatever. As in any experiment the equipment needs to be tested and calibrated for the results to hold firm against criticism. Doing this with people would be interetsing..... taste tests, hearing tests etc..............

Of course no-one needs blind testing to indicate their OWN preference or to dictate what equipment they should buy or upgrade, that is personal and a matter of their own opinion....., BUT if one wants to test a premise like:

"All amplifiers sound the same",

"Cables make no difference to the sound" or

"All wine tastes the same"

AND if one wants to publish the results and have them stand up to scientific and public scrutiny then care needs to be taken with the selection of method, equipment at its calibration.

Best

JA

Share this post


Link to post
Share on other sites
Hello LD,

Yes sure the method could be flawed, but as an example the measuring equipment (the wine buffs) could equally be flawed. The same would apply to any blind test: audio, wine or whatever. As in any experiment the equipment needs to be tested and calibrated for the results to hold firm against criticism. Doing this with people would be interetsing..... taste tests, hearing tests etc..............

Best

JA

i like that point. Floyd Toole goes into it and the results speak for themselves.

its a pitty that its rarely brought up with the debate about DBT's. just shows how little most ppl understand how DBT's work, both for and against the tests.

LD, perhaps u should read up about Floyd Toole's tests.

Share this post


Link to post
Share on other sites


A DBT is so much simpler than what people make it out to be. It a method which allows a pretender to try and prove what he/she is saying, is true. A way to shut up the nay sayers. If you can't pass a DBT then you just have to live with the fact that you can't and certain people won't believe what you say because you can't. It doesn't mean you've gone coo coo or anything.

We shouldn't worry about what we CAN'T prove, it's irrelevant, null, void. We just need to worry about what we CAN prove, that's it.

Share this post


Link to post
Share on other sites

Don't you think this has been done to death already LD? :eek:

Share this post


Link to post
Share on other sites

For me, blind testing is useless as my audio memory is less than that of a gold fish and as unreliable as a politician and believe me, I have tried. I can't pick the difference between cables or sources as I can't remember what the previous one sounded like, unless you are able to instantly switch back and forth between them.

So for me, it basically comes down to sometimes I like what I hear, and sometimes I don't. And that bugs the crap out of me as I have an engineering back ground and like to see fact, figures and test data to confirm things.

So for me, blind testing is not flawed, I am .

Share this post


Link to post
Share on other sites

I agree with some of what you are saying here LD. There must be a certain threshold of what a person can detect or remember. 98% vs 99% sound quality might be very hard to pick. Changing one pixel's colour from green to blue per frame in digital video for example, would probably be impossible to pick in a DBT. But we know there is/was a difference, because we changed it.

This thread/question is largely directed at Drizt, as he is clearly a true believer in the method of blind testing of hifi components. My question is WHY???

I'm not a firm believer one way or the other and I do my best to evaluate a possible upgrade being evaluated, but to me blind testing is not the holy grail of best practice for identifying differences in what we hear. Actually, I'm not sure what is! Maybe it is just sitting and listenning in familiar surroundings.

Blind testing relies on our memory for comparrison purposes, which makes the method open to failure in my opinion.

I recently read on this forum about wine buff's not being able to identify the difference between red and white wine when doing a DBX test. But there IS a difference between red and white wine. This tells me it is the testing method which is flawed.

What do you all think?

Share this post


Link to post
Share on other sites


Terry has already covered a lot of it. I would like to add a lot more, but im busy at work today.



To me its pretty simple.

People convince themselves of things that aren't really there, its the placebo effect. Some people end up convincing themselves that they hear things that simply are not there. This then comes back to whether or not you are willing to live in ignorance and be happy with that. Thats a personal question.

To be honest I can not for the life of me understand how anyone can say that sighted testing is better than un-sighted (blind) testing... (ps. that doesn't mean you don't use your eyes it just means that you don't know what is being tested.)

The whole point of blind testing is to remove our preconceived ideals so that we can come to conclusions based on what our EARS hear, not what we want them to hear.

If you can not detect a difference (for good or bad) between two pieces of the equipment under proper blind test conditions then to YOU there is no difference in THAT particular test/environment/equipment. If you then do the same test but you know what is being tested and you can tell a difference then its clear that you have FOOLED yourself into hearing a difference. Then again it comes back to the question of is ignorance bliss?

BLIND TESTING MEANS: That you do not know what is being tested. Thats it. Its not magic. It does not ruin the musical experience. I do not understand how anyone can be against it or not understand how its better than sighted testing. You can do a blind test in any way you want for as long as you want, it does not matter. The only thing that matters is that you do NOT know what is being tested.

Share this post


Link to post
Share on other sites
Don't you think this has been done to death already LD? :)

Only as much as "I bought a new TT and it's my new sonic revelation" thread :eek:

Share this post


Link to post
Share on other sites

Blind testing relies on our memory for comparrison purposes, which makes the method open to failure in my opinion.

But what is more reliable? Testing that is done when you know what you are listening to or when you don't know what you are listening to? Can you honestly say that you have never fallen pray to hearing what you want to hear? No one is immune to that. And the only way to remove that is to do blind testing.

And remember blind testing is to PROVE there is a difference, not disprove. You can NEVER disprove something through a blind test. You can only PROVE there is a difference.

Share this post


Link to post
Share on other sites

So for me, blind testing is not flawed, I am .

Im not flawed!..............KIDDING :eek:

Share this post


Link to post
Share on other sites


And remember blind testing is to PROVE there is a difference, not disprove. You can NEVER disprove something through a blind test. You can only PROVE there is a difference.

NULL hypothesis testing, is kinda interesting to me now. I never use to subsribe to it, but I think my opinion is changing somewhat. A NULL hypothesis for example would be white wine = red wine using LD's example, or 320kps MP3 = CD. You can never prove anything with a NULL hypothesis, but you'll have some statistics.

Share this post


Link to post
Share on other sites

I guess you can count me amoungst the supporters of blind testing. I think Terry has pretty much covered the reasons ( in his own inimitable style ) in his post above.

I cant see how knowing what you are listening through makes you more impartial and better able to judge a components quality and I cant see how not knowing what you are listening through makes it more difficult to respond to music you are hearing.

Having said this I can see that blind tests are very difficult to do properly and just not practical in the vast majority of situations.

I dont think that there anything wrong with just listening and having an opinion that it sounds good , or bad, or whatever but there can be problems where people want to make generalizations from such a situation.

JA has a valid point about a listener being part of the test equipment in listening tests. Unless a listener can demonstrate a reliable ability to be able to distinguish between two components then I would think they not in a position to make a better/worse call. I would think it is pretty much a waste of effort to run blind tests with listeners who dont hear a difference sighted.

I think it is a real pity that blind tests are so often perceived as something negative used to try and prove people wrong . Blind tests would be better used to investigate the correlations between what we hear and characteristics of components ( as per the Harman group etc ) leading to improved playback equipment. Cheers Mike

Share this post


Link to post
Share on other sites
NULL hypothesis testing, is kinda interesting to me now. I never use to subsribe to it, but I think my opinion is changing somewhat. A NULL hypothesis for example would be white wine = red wine using LD's example, or 320kps MP3 = CD. You can never prove anything with a NULL hypothesis, but you'll have some statistics.

Yes thats right. You can come to the conclusion that its highly unlikely there is a difference if a large enough sample population can not detect a difference. But you can never disprove it altogether.

Share this post


Link to post
Share on other sites

Yes Keith, I do think this has been done to death, but as the issue keeps finding it's way into so many threads, I thought, well lets discuss it!

I like the way Rob323 described it, "So for me, blind testing is not flawed, I am". This is what I am relating to. To me it is the audio memory which detracts from blind testing as THE method.

I would also like to remove the most subtle tweaks like cable from the debate, as they are often picked on, rightly or wrongly, but understanding the huge and valid point of pre-conceived predjudices is not under question by me, but many many many times when I have compared CD players, amps and speakers, the differences are at times pretty plainly obvious. So why the further need for a more scientific testing method?

I sometimes think that the pro BD testers maybe pay little attention to the spacial qualities of a system, ie, how the performance is presented in front of you and that they are more concerned with the raw sound comparrison only (hope that makes sense). Could there be some who thinks everything sounds equal?

Also, Drizt has never pissed me off, ever, but I must admit the notion of being in ignorant bliss (or similar) for not using DBT as a method is unfortunate. I appreciate his passion for this pursuit we are sharing.

Share this post


Link to post
Share on other sites


Yes thats right. You can come to the conclusion that its highly unlikely there is a difference if a large enough sample population can not detect a difference. But you can never disprove it altogether.

It's might not just be about whether it's highly unlikely with a NULL hypothesis. It can be about how many people can actually pass it an how many cannot. It might be 50/50 of the population with the examples above. You definitely can't assume that everyone has the same hearing ability, or the same "hifi trained ears" ability.

Share this post


Link to post
Share on other sites

Hi LD

What's up? Bored or something? Not enough sport on TV?:)

FWIW, I reckon Terry has said it very well.

I'm happy to acknowledge/accept that my subjective opinions can be swayed very substantially by factors that I'd rather keep out of the equation (price, brand cachet appearance etc etc). So, I like the idea of evaluating sound quality without those influences. Of course they can be taken into account after the sound has been assessed.

But I think there are some things that need to be said (again):eek:

1. The proponents of objective testing have no desire to shove it down the throats of those who are opposed.

2. The "measurement" vs "what it sounds like" argument is a totally different argument. Anyone who thinks that is what is being proposed hasn't been paying attention.

3. Listening in the dark or with blindfolds is NOT what this is about. It's simply about not knowing exactly what is being listened to so that the pre-conceived expectations are taken out of the sound evaluation process. How this is achieved is irrelevant to the concept.

4. Most of the proponents have probably done very little in the way of DBT......at least in relation to audio. Because it ain't as easy as some people think. However, the proponents are simply suggesting it might be worth a go and it could be both fun and educational.

5. There is no expectation of "proving" anything or swaying the opinions of others.....especially those opposed to the concept.

6. Blind testing is not necessarily all about quick listens. For example, it's possible for someone to have two identical-looking components, but one has been highly modified internally. They can be labelled A and B. The person can then listen to them for weeks.....swapping one for the other (or having the swapped by another person) at random. If, at the end of as long as they need to come to a conclusion, they are convinced there is no difference, then they may decide the expensive modifications in one of the components are not worth the outlay to them in that system. Or they may decide they are because they can afford them and they provide "peace of mind" or they want the "bragging rights" or.......... The important point is that they will at least know that the unmodified component is pretty much on a par with the modified one under those conditions. Of course, it's also possible that they will find their listening enjoyment is greater at some times than others and this co-incides with component A being in the system. This suggests component A genuinely sounds better, but does not "prove" it as it could conceivably be all down to coincidence. This is only an example of the principle. It's a truth-seeking quest. An anti-hype quest.

7. There is nothing to fear.

It both saddens and amazes me that this topic can cause bitter division. That makes no sense. Why not accept that some people like the idea of evaluating pure sound quality devoid of outside influences and some people are uncomfortable with that concept and don't want to know about it. Maybe they aren't as easily swayed by price, appearance etc, maybe they are not willing to accept that possibility or maybe they are happy for that to be the case. Whatever, if it works for them, that's OK.

Fair enough, I say. It's all part of the "live and let live" philosophy most of us would support. It's not "live and let live as long as you agree with me":rolleyes:

Cheers

Tony

Share this post


Link to post
Share on other sites

1. The proponents of objective testing have no desire to shove it down the throats of those who are opposed.

Well I've read many who have tried to shove it down the throats of those who are opposed. And there are those who might be trying to sell you something too.

Share this post


Link to post
Share on other sites

1. The proponents of objective testing have no desire to shove it down the throats of those who are opposed.

Then why is it mentioned in every thread?

Share this post


Link to post
Share on other sites

I like the way Rob323 described it, "So for me, blind testing is not flawed, I am". This is what I am relating to. To me it is the audio memory which detracts from blind testing as THE method.

But there are many DBT's that you and Rob323 will pass. So are you both flawed?

Share this post


Link to post
Share on other sites

Don't drag Luckydog down to my level, there's only enough room in this gutter for one :eek:

Share this post


Link to post
Share on other sites

This is what I am relating to. To me it is the audio memory which detracts from blind testing as THE method.

but many many many times when I have compared CD players, amps and speakers, the differences are at times pretty plainly obvious.

your contradicting urself.

on one hand ur saying that DBT isn't the be all and end all because we cant remember how something sounds.

but then u go and say that sighted tests are somehow different in that u can remember how something sounds?

Share this post


Link to post
Share on other sites
Hi LD

FWIW, I reckon Terry has said it very well.

I think terry j and Drizt are getting too caught up in the reasons why people might fail or why we need to do DBT's. It's irrelevant and if a person fails you have NO reason why they have failed. Linking failure with the placebo effect or whatever is just speculating.

Share this post


Link to post
Share on other sites
Don't drag Luckydog down to my level, there's only enough room in this gutter for one :eek:

LOL, it's nice to see we can discuss this without guns being pulled.

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.
Sign in to follow this  

  • Classifieds Statistics


    Currently Active Ads

    Total Sales (Since 2018)

    Total Sales Value (Last 14 Days)

    Total Ads Value (Since March 2020)
  • Recently Browsing   0 members

    No registered users viewing this page.


×
×
  • Create New...