Jump to content

Blind Testing


Recommended Posts



  • Replies 114
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

On reflection, I initially thought you (Drizt) were taking aim at the contributors of this thread who were questioning the validity of DBT's. Sorry for jumping the gun there, but I still feel your point was bound to offend some. Anyway, moving on. All is good.

Has my question been answered about whether DBT's are the undisputed king of comparison? Nah, not really. Still has me doubts :eek:

Nuff about me, over to the rest to comment.

Link to comment
Share on other sites

Has my question been answered about whether DBT's are the undisputed king of comparison? Nah, not really. Still has me doubts :eek:

It's not the king of comparison. An ABX test in particular is not testing comparisons. You do should do the comparisons BEFORE the test. Then come up with a hypothesis. An ABX test is testing whether you can DIFFERENTIATE under controlled conditions, not testing whether you can COMPARE.

Link to comment
Share on other sites

Firstly, a disclaimer that I 100% agree on the opinion that aural memory can be so small as to make any tests of dubious quality. My own aural memory is disgustingly poor, I think. That said, I think if you were going to 'test', DBT is more accurate/worthwhile than sighted testing

YS, what someone else fails to hear has no bearing on you. Someone else picking a difference with repeated accuracy where you cannot may or may not have a bearing. Now that does mean a difference exists. But if you can't pick the difference, then why worry about it?

As for your personal analogue/digital example - it's unlikely that you won't be able to hear a difference. But if you were to do the test and couldn't hear a difference... It'd give you something to think about. But as has been stated, a negative result doesn't 'prove' anything. Only a repeatable positive result.

LD, way back to your original post and the reason you started this topic - I think a few others such as Terry, Scorp, Dr X have made some good points to address your question, but as you didn't agree with them, I'll offer my interpretation.

As Terry said, you're a victim of your preconceptions. Visually seeing the units you're testing has already put an idea in your head of where they stand. DBT is not ideal for a plethora of reasons, but what it does manage to do is eliminate some of the shortcomings of a sighted test. Not all. But some. It's simply a more accurate test than a sighted test, with less of the existing shortcomings and no new ones. That's the short answer.

Is it necessary? That's for you alone to answer. You could arbitrarily say that sighted tests are 45% accurate and DBT's are 43% accurate, and they're both so inaccurate that it doesn't matter. You could come up with your own numbers. The problem is that we're dealing with something subjective, and we're comparing subjectively. Music is not something where measurements are going to give a better indication than the ear. If you get to the point where you need measurements to tell the difference... you've already passed the point at which the difference was of any consequence. And it's difficult to measure enjoyment...

And then, you've got the mother of all qualifiers - when you're listening to music for the sake of listening to music, you're listening differently than when you're listening critically to the system. This is why the ideal auditions probably aren't really rapid switching tests but relaxed listening over long periods.

Anyway, had to get my 2c in before they revoked it as legal tender. Oh... too late!

Wilson

Link to comment
Share on other sites

Hi PWilson

That was very well summed up. I can live with that fine summary of the already excellent points raised by others in the 'for' camp.

I particularly like the way you describe neither sighted or blind testing is perfect. I agree with that.

Link to comment
Share on other sites



I find blind testing too stressful, and my philosophy for most if not all purchases I make is if they look good, and the noise they make is pleasing to my ears then I’m usually happy for 5mins or so.:)

Seriously though a lot of you guys should get out and hear live music…although I haven’t been out for some months, I like to listen to live music, it gives me some perspective on what I try to achieve at home. I’m sure bands would love you DBT them and their equipment :eek:. For me it’s always about the music, and if you don’t listen to live music on a regular basis what is your reference. Listening to one system and comparing it to another, which one is right, and which is wrong, you are basically comparing what appeals and what doesn’t IMO.

Link to comment
Share on other sites

Young Skywalker what is your hypothesis? Remember that mixing/mastering can not only differ between CD and Vinyl for the same album, but different releases of the same album just on CD or just on vinyl alone.

Hi Dr X,

You make a very valid point about the mixing/mastering being different between the CD and vinyl releases of a particular album and indeed different releases on each format of the same recording, for example the new Analogue Productions reissue program of selected Blue Note jazz titles on 45 rpm vinyl compared to the original releases.

For this reason I feel that it is impossible to compare the two formats on equal footing, be it under controlled DBT conditions or under everyday listening conditions.

They are what they are and should be evaluated and enjoyed on their own terms. A true music lover will gorge themselves on music from any available source and the audiophile would take that a step further by trying to maximise the performance of those sources within his/her particular system. I don't think too many music lovers who are also audiophiles would reject one format for reasons other than convenience or lack of interesting software both of which may fall under the personal cost/benefit ratio argument proposed elsewhere.

I am happy that, to my ears and under everyday conditions, my turntable sounds better than my CD player, as it should for the considerable financial investment it represents, not because I have daggers out for digital or CD but because the vast majority of the music I like to listen to is available on very well mastered and produced LP reissues from people that care about the music and sound quality.

I am all for the best possible digital sound because some of the music I like is only available on CD, SACD, etc. For example, I can't listen to a commercial release from U2 on CD because the sound is so bad, not because my CD player sucks but because the CD was produced with a mass market audience in mind. When I play audiophile approved CDs (FIM, Reference Recordings, etc) they sound very good indeed. Unfortunately, the number of audiophile approved CDs pales in comparison to the number of audiophile approved LPs out there which contain music that I actually enjoy.

My point is that comparing formats designed, in the modern era at least, for very different target audiences is like comparing apples to oranges, both have inherent advantages and disadvantages and the impact of each will differ depending on the context of the system. Both live or die by the software, for without this we have no sound to begin with.

What would be interesting, and this has been tried before, is comparing a high resolution digital recording from very high end vinyl to the original vinyl playback. Unfortunately, this is a purely academic situation unless we intend to market the otherwise indistinguishable high resolution digital recordings of vinyl as a superior sounding product to commercial CD releases.

At the end of the day, if you like the music that is available on vinyl and don't mind (or even enjoy) the ritual then buy a turntable and knock yourself out. While you're at it, get the best CD player, computer music server/DAC and FM tuner you can afford and collect/listen to as much great music as you can lay your hands on. This is what hi-fi is all about afterall and it is all too easy to forget it in the heat of battle.

Aaron

Link to comment
Share on other sites

My comment regarding the point that a large % of the people against blind testing are those that have already spent a large sum of money on hifi. Do a quick look at the people on all different forums and have a look at the equipment list of the people who are against blind testing. This is just an observation. Its not an accusation. What I should have said, was to elaborate my point with a hypothesis on why this would be.

sooo....

If you have spent $50000 or more on brand new equipment, and there is no chance of you recouping your money on the second hand market, what would be the incentive for you to even do a blind test. In the possibility that you find you like something much cheaper after doing a blind test, then you will potentially lose a lot of money selling your expensive gear. So I can understand why one who has spent a lot of money on hifi having an unwillingness to do a blind test.

Hi Drizt,

Being amongst those who have spent an awful lot of money on audio, and music and concerts ...

I am not opposed to blind testing - it is occasionally helpful. My experience with audio and the scientific world I inhabit has lead me to the view that it is one of the tools I can utilise to make a decision.

I think you may also be mistaking association and causality. It is entirely possible that those of us who have spent a lot of money on equipment are all collectively susceptible to suggestion and placebo. It is also entirely possible that on average people such as myself are older, have made some expensive mistakes along the way and have come to a series of ideas that are quite different to the ones we may have started out with.

One of the curious, reactionary ideas that has come to me is that dbt are near useless in forming opinions as the conditions that were artificially created during the testing only apply during the test.

I will say it again - I am not opposed to blind testing but extending the results of a blind test to another situation is intellectually irresponsible.

Kevin

Link to comment
Share on other sites

I'm glad this was allowed to continue.

I kinda get the idea there are two possible approaches to blind testing, the strong blind test and the mild blind test.

As the differences between the items under test become smaller, the need for the strong approach becomes more urgent, to be able to tease the 'signal out of the noise floor'. This is obviously the approach required when testing things like cables.

The ultimate blind test (I think) is as I mentioned before, not even knowing the identity of the amp being tested, I gave the example of the review of the valve amp earlier.

I mean, lets try a little experiment right here.

"I'm going to test an amp in my system tonight".

OK, we get some sort of concept from that statement.

"I'm going to test a valve amp in my system tonight".

Would anyone agree that straight away we have a different concept? the mere mention of it being a valve amp introduces all the past ideas, concepts, conclusions and idle wonderings we've ever had about valve amps. Immediately the experiment has been coloured, and we have not even conducted the test yet!

Indeed, if it could be arranged, the ULTIMATE blind test is that you are not even aware that a change has occurred. You hear a track , then w/out your knowledge an amp is changed (and with a suitable pretext) you hear the track again. "gee, I liked that track, would you mind playing it again for me?' or whatever.

I'm sure I don't need to spell it out any further. Tony M touched on this in his post, when if by whatever mechanism it could be arranged that on random days an amp was switched in and out and over longer term listening (hopefully) your enjoyment increased/decreased in step with a particular change.

Those examples illustrate the difficulty of the strong blind test.

The soft blind test is simply that we forget about level matching etc etc, and we audition the new amp (say) in our system as we always have done, BUT we don't know the identity of the amp. Forget level matching and all the rigmarole associated with the strong test.

I'll ask you Phil simply cause you were the curious one, AND I know in the last year you bought a new amp.

In what way would the outcome have been compromised, different etc if when you were auditioning the amp against your old one the ONLY change in the procedure was that at no time did you or whomever was involved in the auditioning process with you know the identity of the amps you were listening to?

In other words, rather than "this is the rowland" and "this is the meridian" (I'm ashamed that I don't recall what your last amp was) instead you had

"this is amp A" and "this is amp B".

For the life of me I don't understand how this would change any conclusion you made if the conclusion you made was reality.

But on the flip side, by this simple expedient an awful lot of baggage has been removed.

it is NO more stressful, NO more open to wrong conclusions, I simply cannot see anything disadvantagous about it.

Link to comment
Share on other sites



YS.... mate... please let me know what is un-natural about a blind test? You do know that the only thing different between normal sighted testing and blind testing is the fact that you do not know what is being tested. (what ever method used to conceal what is being tested does not matter).

Are you insinuating that without knowing what is being played you can not evaluate if it makes a sonic difference or not?

Just so that I am clear on the whole DBT thing.....

If I participated in a double blind test in my system and couldn't reliably pick the difference between my very expensive analogue setup and my relatively modest but still excellent digital setup then I am imagining things when I find vinyl playback vastly superior under normal sighted real world conditions? The placebo effect is really that powerful?

If an untrained listener/non-audiophile enters the room, listens without bias or knowledge of what is being played back (digital or analogue) and prefers the analogue without any hesitation is that a close approximation to a real world double blind test?

Are DBTs flawless and utterly conclusive or would the proponents of such concede that there is some wiggle room?

When the results do not correlate with the real world observations does that imply in some way that the DBT may be flawed in addition to suggesting that some degree of placebo or preconceived bias is present?

Aaron

Link to comment
Share on other sites

:DJust remind me guys DBT did stand for "Draught Beer Testing" of equipment. I bought 5 cartons of Draught Beer when this thread started and have tested my audio equipment - just finished the 5 cartons and I must say the music sounded much better - luv ya's all:rolleyes:

Link to comment
Share on other sites

Hi Drizt,

One of the curious, reactionary ideas that has come to me is that dbt are near useless in forming opinions as the conditions that were artificially created during the testing only apply during the test.

Kevin

How is a 'blind' test un-natural (artificially created) ? All that is required is that you do not know what is being played. The sound is exactly the same. All that has been done is your bias' have been removed from the equation.

You can still sit in your comfy chair. You can still listen to your favourite music while drinking a lovely red. All that is needed is that you are un-aware of what is being tested?

This is an honest question, im not being a smart ass. What am I missing?

Link to comment
Share on other sites

Come on Aaron, that is just crap. Exagerating a bit I think. So you now begrudge people's enthusiasm for a newly found upgrade. Why not share their joy. Go back and read some of your highly enthused posts over upgrades you have done. You're no different to the rest of us.

Never said I was. My subjective reviews are only that, subjective. And now that I have learnt more I would like to have another go at all of my previous reviews to revisit them under proper blind testing conditions to get a gauge on whether or not I had been fooling myself or not.

I'm man enough to admit where I have been wrong. And I know that I have been wrong a lot, and I'm going to wrong a lot in the future. The point being is to learn from ones mistakes.

Link to comment
Share on other sites

Hey Terry

There is no negative to my amp trial a year or so ago, except I didn't have access to a helper who could change the cables to the differing amps. Yes I did do a serious comparison, where I was always frowning while listening to identify a benefit, one way or the other.

I never said I am against blind testing, but this thread today came about as it was being preached from up on high that those who don't do rigorous blind testing are living in a state of ignorance. I wanted that issue fleshed out today and I think we have achieved a bit today.

Getting back to the amp demo, the Rowland was up against my then Elektra Ref. No blind testing was done at this time. I would not shy away from a DB test when comparing anything like this. From memory, I think I had the amp for about 3 days to evaluate it's performance. Even if someone was super fast, there would be at least a minute required to change speaker cable and interconnects, tuning the amps off while changing cables to be bloody careful to not short anything and adjust volume if needed. This is a strain for music memory in a DB test, as it is for a sighted one.

I was convinced of a difference/benefit so I bought the Rowland.

I felt my system took a good leap forward with that buying choice.

Could I have been swayed by knowing which amp was in play? Sure. But at the time I had never heard of Rowland before this demo, I was not specifically looking for an amp at the time (the demo was offered to me by the ever helpful Tony C) and it price was way more than I had available at the time. Yes I was eagerly wanting to hear a benefit, but I think I would've been quite delighted if the Elektra held it's own and I was able to keep whatever funds I had in my bank.

I am confident of being able to pick the difference between these two amps in my system and in blind conditions. Extremely confident. "Temptation" by Diana Krall would be the demo track, as I clearly remember the Elektra producing a harshness during that track, where the Rowland does not, amongst other benefits.

General comment; I guess I take issue with the thinking that people just like me, who spend plenty of $$ on this hobby, who agonise over searching for improvements, who has made wrong choices along the way and paid dearly for them, learned by their mistakes, would then buy components purely based on the price tag or brand name.

Again, I am not against blind testing, not one bit, but it is not the be all end all of testing.

Link to comment
Share on other sites



You guys still here...? :eek:

OK speaking of blind testing....

One day, a blind man and his dog are walking down a street, they come to a busy intersection, and the dog, ignoring the high volume of traffic zooming by on the street, leads the blind man out into the thick of traffic. This is followed by the screech of tires and horns blaring as panicked drivers try desperately not to run the pair down.

The blind man and his dog finally reach the safety of the sidewalk on the other side of the street, and the blind man pulls a cookie out of his coat pocket, and offers it to the dog.

A passerby, having observed the near fatal incident, can't control his amazement and says to the blind man, "Why on earth are you rewarding your dog with a cookie? He nearly got you killed!"

The blind man turns partially in his direction and replies, "To find out where his head is, so I can kick his ass."

cheers

2sheds

Link to comment
Share on other sites

I've only seen the word objective a couple of times in this thread but it has reared it's ugly head. This is my main problem with DBT. It claims to be objective when enjoyment of music and preferences for equipment are subjective.

I'm not really opposed to DBT but I really don't put as much value on the perceived advantages as some seem to. I don't think that excluding biases and preconceptions is particularly useful unless you continue to listen in a DBT scenario after you have bought the equipment. I mean, what if you do a DBT, buy the product you perceive as "better" under DBT conditions but later regret it because your perceptions are different when you take the gear home and listen to it outside of a DBT scenario.

I have always railed against the supposed automatic superiority of objectivity as I think that we often make better decisions when we have a stake in the outcome. Life is a subjective experience, listening to music is one of the more subjective aspects of life, I'm quite happy with the subjective outcomes of sighted testing.

Also I don't like the implied superiority of the DBT crowd. Those of us who are quite happy with sighted comparisons are not fooled by the comparison (or fools as is implied), we are not being seduced by a placebo, nor are we ignorant. We just know that a sighted test gives us a more than satisfactory comparison of equipment we are subjectively evaluating with what we would subjectively term "good music".

DS

Link to comment
Share on other sites

Interesting thread.

It certainly arouses some passion doesn't it.

In summary it seems that DBT's look interesting for confirming if you can detect a perceived difference accurately (unsighted).

So if you were try to decide between two components and wanted to know if on sound alone you could reliably select component A from component B, a DBT would help.

If you couldn't, wont all the sighted things still come into play? Would you decide to go with the cheaper, or the one the looks the best, or the one that is reviewed the best, or the one that has a longer warranty or the one that has a better resale value?

If you could pick reliably that there was a difference, don't you still have to decide which you like better? What if you like the bass on A and the midrange or top end on B? Decisions are still required.

Whilst I like the idea of doing DBTs - I think in practice they would be tedious to conduct and I am not sure that they prove anything - except maybe if you can reliably notice a difference between A and B.

I'm not really an audiophile. If it plays music I'll listen to it. If I like the music, I'll be happy.

Link to comment
Share on other sites



How is a 'blind' test un-natural (artificially created) ? All that is required is that you do not know what is being played. The sound is exactly the same. All that has been done is your bias' have been removed from the equation.

You can still sit in your comfy chair. You can still listen to your favourite music while drinking a lovely red. All that is needed is that you are un-aware of what is being tested?

This is an honest question, im not being a smart ass. What am I missing?

Sorry Drizt I was not being very clear. I am thinking more towards the meaning of any results. You could conduct a very well controlled blind test in your room and system but the results have no validity outside your system and situation. Interesting for us all to read about but to apply the same thinking in a different situation may well not achieve the same results.

I personally find a body of observations made in a variety of situations by a variety of people allied with my personal observations more useful than a dbt.

Over time I have found assessing the sound versus my accumulated concert experience the most useful in making decisions. I realise there are faults with this approach too but it tends to very quickly reveal faults.

Kevin

Link to comment
Share on other sites

Terry, Last time I bought an amplifier here is how I did it. I borrowed three amplifiers that I thought would work well in my system. I tried listening to them sighted ,disconnecting/ reconnecting each after shortish time intervals of two or three tracks . I got pretty confused and was going nowhere.

Looking for a better method I put them out of sight and had my wife connect different amps at different times over the next couple of weeks without me knowing when she had changed connections. When I was listening and had a thought like " this sounds exceptionally enjoyable " or "I never noticed that bit before " or whatever I got up and had a look what amp was playing and around 80 % of times it was the same amp so I bought it. This was flawed as a blind test as the amp I bought was significantly more sensitive than the others so I may just have preferred it because I probably played it a little louder than the others but I needed to make a decision and this was the best I could come up with.

I think a critical component of a good blind test is that you first establish that you can reliably determine a difference. I suspect that a blind test where you know "this is component A " or "this is Component B " is somewhat compromised because it doesn't check on consistency of judgment as having decided A is better you will just keep repeating this opinion. If you don't know which is which and make judgments like A is better than A then I think its fair to say you not really sure which is better.

I think its good to have this thread so people who don't want to read about blind testing don't have to. I think it would be very sad if people were put off reading and contributing to the forum because blind testing kept rearing its ugly head everywhere.

I am still interested in the concept of limited auditory memory if anyone has any good references .Cheers Mike

Link to comment
Share on other sites

YS.... mate... please let me know what is un-natural about a blind test? You do know that the only thing different between normal sighted testing and blind testing is the fact that you do not know what is being tested. (what ever method used to conceal what is being tested does not matter).

Are you insinuating that without knowing what is being played you can not evaluate if it makes a sonic difference or not?

Hi Drizt,

I understand the concept of blind testing perfectly well. I think caution needs to be applied in choosing what is being compared to what under these blind conditions. For instance, in my mind it is impossible to directly compare analogue playback to commercial CD since the differences between available software are often huge. In this case it either lights your fire or it doesn't and both can happily co-exist.

Cables and various tweaks is an interesting one since the effects may be subtle or dramatic depending on the system context and sometimes it takes a period of adjustment before meaningful opinions can be formulated. In much the same way that a loudspeaker that is all leading edge and no sustain or decay might initially impress with its apparently hyper-detailed presentation and yet grows fatiguing over time. Maybe a poor analogy on my part but the best I could come up with after a 14 hour day.

Comparing one CD player to another with identical discs would be interesting as would a blind comparison between two analogue rigs (optimised turntable, arm, cart, phono stage) using identical pressings of an particular LP. Of course other factors besides sound may enter the equation in the real world (aesthetics, ergonomics, affordability, etc) and these should be factored into any purchase decision in my opinion.

Naturally when someone has just installed a new component there is an expectation that some change will be audible and obvious and I can accept that blind testing removes that bias. The problem as I see it is that it is a test and therefore, while it stands on its own as an experiment, it doesn't reflect a balanced opinion formed over an extended period of time which may be necessary in some cases (see loudspeaker example above).

Why don't we see hi-fi review magazines based around the concept of double blind shootouts? Possibly because the results would not make for entertaining reading, although some may find it highly informative.

I know the type of sound that appeals to me when I hear it and the price, topology or whatever has no bearing on the opinion that I formulate. I have worked at a store selling obscenely expensive hi-fi gear and for the first few days I was in heaven but after that I was bored by the sameness of it all, the connection to the music was seriously lacking in many cases. That is not to say that there aren't wonderfully performing expensive products but I am totally convinced that price does not necessarily equal realistic sound. Did I need a blind test to reach these conclusions, no, but I can see that others might in which case the methodology is perfectly valid for that person.

Would anyone agree that straight away we have a different concept? the mere mention of it being a valve amp introduces all the past ideas, concepts, conclusions and idle wonderings we've ever had about valve amps. Immediately the experiment has been coloured, and we have not even conducted the test yet!

Hi Terry,

I think that statement would be true for the listener who has never heard a valve amp that is free from the old cliche of warm midrange, rolled off at the frequency extremes, etc.

The type of gear/sound that I gravitate towards, be it valve or solid state, tends to sound more similar than different. Having said that when paired with a suitable loudspeaker the valves do certain things that create a stronger connection to the live experience for me, although these may very well be an artifact of the technology. Whether I could pick out these things under a DBT I don't know since it is a subtle but personally significant effect over the long term for me.

Apologies in advance for my ramblings.

Aaron

Link to comment
Share on other sites



Guest
This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...
To Top