Jump to content

Are Audio Double Blind Tests (DBT) To Be Doubted (TBD) ?


Message added by Marc,

This thread will be allowed to continue, but each post will need to be approved by moderators. We understand this slows the discussion down, but it is the only way to ensure that posts are on topic, helpful and relevant to the thread. Many posts have now been removed that were way off topic or considered to be trolling.

Recommended Posts

13 hours ago, MLXXX said:

 

I'd have thought that for audiophile purposes averages for a group would be less important to establish than the performance of high listening acuity individuals.  What use would a statistically "average" measure be put to?  For example it were found that the upper frequency hearing acuity for a population of adults had a mean value of 14kHz, would it be wise to use that value figure for the design and testing of microphones and loudspeakers?  Certainly not.

 

Years ago I was frustrated with a car radio gain control that provided gain changes in minimum steps of 1dB. I prefer to be able to adjust gain down to a 0.5dB increment.  A design engineer when queried as to why they chose 1dB increments for the car radio might state that studies had shown that the minimum audible gain change for music is around 1dB for people generally. 

 

What is fine for a hypothetical "average acuity" listener may not satisfy those with greater acuity than a test average.

 

Conversely what is fine for an "average acuity" listener may not satisfy those with hearing loss. I am a member of an orchestra that includes retired musicians in their 60s and 70s. Within that group of retirees there are a few who often ask for instructions to be repeated. Their hearing loss is sufficiently advanced that the orchestra conductor to be understood needs to raise their voice by a considerable margin.  I note that people with noticeable hearing loss may prefer to listen to recorded music with a restricted dynamic range.

 

In short, there can be no such thing as a precise repeatable "average" level of hearing acuity, given that individuals vary so much from each other, and on top of that vary individually themselves from day to day and hour to hour. And even if after extensive investigation one did establish a "typical" or "common" range hearing of acuity, it would not necessarily be wise to design equipment to cater only for those whose hearing acuity fell within that range.

We are left with the result that DBTs administered to large groups are likely to provide only very broad results. So for example the people comprising a large group may almost all appear to be able to hear a 3dB change.  But once you get down to a 1dB level or lower change in gain,  results will vary a lot from individual to individual within the group.  Also there is no guarantee the mean result of the group overall for a 1dB level change will closely match the mean result of a second randomly selected  group of human test subjects for a 1dB level change.  That doesn't mean the double blind testing method is flawed.  It just means that the DBT results have to be interpreted as broad indications only of [apparent] human listening acuity.

All correct. So let’s face up to another truth about this. With a few exceptions many of whom have come up in this thread and we can call “industry” researchers, they aren’t concerned with running an extreme audiophile listening contest, nor are they interested in confirming our prejudices around particular technologies or products. 
 

Much as we like to think we’re special... are we?

Link to post
Share on other sites
  • Replies 268
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

I completed perceptual testing of audio products in my PhD thesis and got reliable results. Enough to make design decisions on. This was supervised by a research audiologist professor (specialist in p

Dear @Grant Slack, thank you for your letter dated 3 hours ago. Its nice to get a civilised letter, you wouldnt believe some of the uncouth communique's I receive.    I have considered

Hi frednork   I get the impression that you you opened this thread because you resent people drawing strong conclusions, that certain things are inaudible, from shoddy DBT listening tests. 

14 hours ago, Eggcup the Dafter said:

Other differences I heard were probably "bias" to use the perjorative term. Thinking about it, we could do with a more neutral term ("extra-audible response"?). I think we are allowed as humans to have those responses.

 

Hi John,

 

please, please, please don’t take my use of the term “bias” as perjorative. It is a completely neutral, technical term to describe an inevitable, natural human perceptual process. That is how I am using it, without fail.

 

I know it is used in the vernacular in a negative way to say “you’re just biased”, i.e. prejudiced, but in discussions of sighted listening effects, it needs to be allowed as a purely technical descriptor. Whether we initially see the Rubin vase as a vase or as two faces nose-to-nose is an example of personal bias without value judgement: it is in this sense that we need to be able to discuss sighted listening while using the term in a technical sense, not perjorative.

 

I would be somewhat horrified if you have been reading my posts with the word as a slur on people who perceive things differently in sighted listening compared to blind listening. ? I promise you I would never do that. I am just using the same term as the research uses when discussing certain natural, inevitable and frankly necessary human perceptual mechanisms.

 

regards

Grant

  • Like 2
Link to post
Share on other sites
1 hour ago, Grant Slack said:

please, please, please don’t take my use of the term “bias” as perjorative. It is a completely neutral, technical term to describe an inevitable, natural human perceptual process.

 

This can't be overstressed but I admit I forgot that someone might take it this way.   Good point.

Link to post
Share on other sites
15 hours ago, Eggcup the Dafter said:

When I did sighted tests before my latest upgrade, my reasoning is that I will react to things like the ergonomics, look, maybe the cachet of ownership, the degree of confidence I have in the result, and my wife's concerns and inclinations (maybe not her impaired hearing so much)... doesn't that all count?

 

I was ignoring this useful side of sighted testing, so thanks for including it.

 

I guess I do it too (but it has very little sway in my evaluation of something), and I don't even really regard it as testing in the audiophile/technical/sound sense.  It certainly isn't what is usually meant when people on this forum claim 'x' is better, sounds better, easily heard, it's a big improvement etc.

 

15 hours ago, Eggcup the Dafter said:

But if your reaction is not to the soundwaves, then you have to even take some of that with a pinch of salt when deciding what speakers to buy (where you will definitely hear real differences sighted) or what cables to buy (where the benefits are less obvious in terms of sound, but are definitely real to the buyers

 

Some of my examples would be things like cables.  I am a cynic about cables in the general sense, but I don't mind paying more for something that looks sexy  and is solid and well made.

 

Also, records/vinyl.   Sure, I like the look of a record playing.  I love the ritual of cleaning and polishing, and carefully putting one on the turntable.

Link to post
Share on other sites


2 hours ago, sir sanders zingmore said:

I’m not sure I understand what you are saying. 
My guess (and pls correct me if I’m wrong) is you are saying that the reason obvious differences get smaller once I introduce even casual controls, is that those controls somehow affect my ability to discern difference?

 

 

When you blind yourself, you put yourself under some sort of different psychological pressure. This blinding is a cue to you that this is different somehow to your normal listening (and at a concious/subconcious level you may need to respond differently).   You may be aware or unaware and more susceptible or less susceptible but there is some level of pressure on you to perform that was not there in normal listening conditions.  Whether or not this can be completely eliminated in all people I dont know for sure but would expect, some sort of distribution with totally eliminated and not eliminated at all with practice on either extreme).  If you think about any sport endeavour, once a required level of physical strength and skill is attained at a similar level to other competitors, what is the magical factor that allows champions to perform more consistently than others. One important one is their ability to perform under extreme pressure.

 

I am not trying to say this invalidates blind testing, as, I have mentioned before have used it successfully to confirm or otherwise subjective  assessment.  But being unaware of this means that many will just try doing a blind test that appears on the face of it to tell them what they want to know and expect the result to be in direct comparison and correlation to their sighted listening however one situation is unfamiliar and the other very familiar. This is another (important) variable that is not being accounted for. If you have been masochistic enough to read my unhinged ramblings you will see that I mention training as the indicator of a test that may be worth examining more closely and no training + no discernability makes it TBD.

 

My personal experience of being on sensory panels and training to be on expert panels has shown me that without the training my ability to discern was impaired under blind conditions but almost gets back to what I expect I can perceive under sighted conditions once training has been performed as long as you are interested in doing the panel.   Regardless of how good you may think you are under sighted conditions it doesnt cut it for peer reviewed journals so the researchers may include subjective analysis as a proposition and then confirm that to a greater or lesser extent with blind testing. In that situation it was really only used to definitively say something that was trending through the subjective and instrumental data. It was never used to discover new things that werent popping up elsewhere in the subjective and instrumental testing.  DBT was done as little as practically possible as it was highly expensive, resource intensive and sometimes not successfully showing what was hoped for. 

 

Hope that helps to explain.

Link to post
Share on other sites
7 hours ago, Grant Slack said:

 

Hi John,

 

please, please, please don’t take my use of the term “bias” as perjorative. It is a completely neutral, technical term to describe an inevitable, natural human perceptual process. That is how I am using it, without fail.

 

I know it is used in the vernacular in a negative way to say “you’re just biased”, i.e. prejudiced, but in discussions of sighted listening effects, it needs to be allowed as a purely technical descriptor. Whether we initially see the Rubin vase as a vase or as two faces nose-to-nose is an example of personal bias without value judgement: it is in this sense that we need to be able to discuss sighted listening while using the term in a technical sense, not perjorative.

 

I would be somewhat horrified if you have been reading my posts with the word as a slur on people who perceive things differently in sighted listening compared to blind listening. ? I promise you I would never do that. I am just using the same term as the research uses when discussing certain natural, inevitable and frankly necessary human perceptual mechanisms.

 

regards

Grant

I know this. It's quite clear that among the "anti DBT" or "anti science" factions, the term bias is an insult. (Especially among, well, the biased).

It would make our lives easier here to separate known prejudice ("all class D amps are crap"), the areas where actual differences become hard to discern, and the areas where our other senses get in the way (our eyes see a 12" driver and we hear more bass).

 

And the vernacular is important in a discussion forum like this.

 

Though maybe, I'm just biased ?

  • Like 2
Link to post
Share on other sites
On 08/04/2021 at 12:29 PM, frednork said:

 

Your science consensus manifesto is very clear and precise (as it should be) and says "High res audio is not detectably different to standard res" 

 

Stuart's and Amir's results would argue this statement is false and the previous work done which indicates otherwise was done poorly.  This would also appear to add weight to the conflict ridden (but well executed) Stuart work.

 

There appear to be some holes in the SS consensus and how it goes about doing things. Will say it again, what you have put forth is not a scientific approach.

 

Let's be clear about one thing... anyone with the right equipment can measure a difference between 16 bit and 24 bit playback. No matter what amounts of upsampling technology you throw at them, playback of a 16 bit signal will have at least 0.18% distortion at -60dB, and distortion increases from 0dB down to reach that figure. Better CD playback over the decades has reached this point. Back when CD was first introduced, that -60dB figure was close to 2% for the converters used.

 

To put that into perspective, end of side LP distortion is still higher. We need to know more than the raw figure to understand if we can hear that distortion. But for me it remains a question. I don't see that we "hear" the distortion of CD playback consciously in the majority of situations, and that explains the general null results when people search for it. I would suggest that we may subconsciously perceive it on rare occasions  - and it may give us a slight feeling of unease, though I don't have evidence for that, obviously. I'd also guess that you could design a playback solution where you could tell the difference, because this is about playback technology rather than the format. You may also be able to teach yourself to discern the difference, though I suspect that if Amir was subjected to somebody else's DBT on their choice of equipment his ability might disappear.

 

The equivalent distortion figure for 24 bit playback has lots of zeros after the decimal point... so from an engineering perspective, it would make sense to be certain and go to 24 bit recording and playback whether or not the average person can pick it in a DBt/

Link to post
Share on other sites
36 minutes ago, Eggcup the Dafter said:

To put that into perspective, end of side LP distortion is still higher.

 

Probably right.  But I am curious as to whether that takes into account modern cartridges and stylus shapes, like microline?

 

btw.  there are people on some vinyl oriented forums who claim this distortion does not exist :)  

Link to post
Share on other sites


1 hour ago, frednork said:

When you blind yourself, you put yourself under some sort of different psychological pressure. This blinding is a cue to you that this is different somehow to your normal listening (and at a concious/subconcious level you may need to respond differently).   You may be aware or unaware and more susceptible or less susceptible but there is some level of pressure on you to perform that was not there in normal listening conditions. 

 

The last part of that (pressure to perform) happens in sighted testing anyway.    Also, if pressure is a problem, this lends weight to the validity of informal DBTs, rather than the contrary.

 

Link to post
Share on other sites
2 minutes ago, aussievintage said:

 

Probably right.  But I am curious as to whether that takes into account modern cartridges and stylus shapes, like microline?

 

btw.  there are people on some vinyl oriented forums who claim this distortion does not exist :)  

I should have been more specific. End of side distortion is related specifically to arm geometry. If a record is properly cut to allow for the lower available space per revolution, and you use a linear tracking arm, it will be much less than for your typical nine inch arm. Just as with CD, the distortion is playback technology. And even with a nine inch arm, you can minimise end of side distortion by alignment, but only at the expense of more distortion when playing back the rest of the side.

 

I guess I should never use a generalised example. Too many pedants here.

Link to post
Share on other sites
15 minutes ago, Eggcup the Dafter said:

I should have been more specific. End of side distortion is related specifically to arm geometry. If a record is properly cut to allow for the lower available space per revolution, and you use a linear tracking arm, it will be much less than for your typical nine inch arm. Just as with CD, the distortion is playback technology. And even with a nine inch arm, you can minimise end of side distortion by alignment, but only at the expense of more distortion when playing back the rest of the side.

 

I guess I should never use a generalised example. Too many pedants here.

 

Apologies - not trying to be a pedant.  Just a pet peeve of mine that some don't think it even exists.  It isn't a big problem.  Even with a short arm, if you align a microline cart properly on it, you aren't going to be annoyed by it, even with the most energetic orchestra in full bellow at the end of side two. :) 

Link to post
Share on other sites
  • Administrators

Back to the topic please.

Link to post
Share on other sites


Posted (edited)
38 minutes ago, aussievintage said:

 

The last part of that (pressure to perform) happens in sighted testing anyway.  

 

Yes, good point! but that is where the flaw of sighted listening is revealed the most, as people will use other senses/perceptions to make up for the shortfall of discerning ability.

 

38 minutes ago, aussievintage said:

Also, if pressure is a problem, this lends weight to the validity of informal DBTs, rather than the contrary.

It depends on what you call informal. But assuming there are no other fatal flaws in the methodology, the more comfortable and familiar a person is will give them a better chance of not being as affected. And the other essential part regardless of familiarity of surroundings is being familiar with doing DBT's.  A really unhinged idea may be to start up a DBT club with your mates, meet weekly, and do some really simple DBTs' , introduced level of distortion vs none, volume difference, etc ie Things that we are pretty sure we can discern. And get used to performing the DBT and being tested and see how you go over time, whether your ability to discern is improved. And share your progress to inspire and inform others (and keep yourselves honest).  DBT's done by a group such as this, after some time, where there had been sufficient training would be a much better indication of what might be discernable or not under blind conditions when compared with the vast majority of tests I just went through.

Edited by frednork
Link to post
Share on other sites
2 hours ago, frednork said:

When you blind yourself, you put yourself under some sort of different psychological pressure. This blinding is a cue to you that this is different somehow to your normal listening (and at a concious/subconcious level you may need to respond differently).   You may be aware or unaware and more susceptible or less susceptible but there is some level of pressure on you to perform that was not there in normal listening conditions.  Whether or not this can be completely eliminated in all people I dont know for sure but would expect, some sort of distribution with totally eliminated and not eliminated at all with practice on either extreme).  If you think about any sport endeavour, once a required level of physical strength and skill is attained at a similar level to other competitors, what is the magical factor that allows champions to perform more consistently than others. One important one is their ability to perform under extreme pressure.

 

1. If one ceases to be able to hear a difference because one is "under pressure to perform" that suggests that the perceptual difference is small to begin with. In other words, one's ability to hear the difference is only marginal.  In these circumstances it is arguable that one's ability to hear the difference is so tenuous that the difference made by the change (whatever it happens to be) is not worth worrying about! For practical purposes, the perceptual difference is so small as to be negligible.

 

2. One can make a converse argument, that only when put under pressure to perform will a slight difference become noticeable (because of intensified concentration and focus).  Instead of listening to and enjoying the musical interpretation of the performance one tries to hear subtle deficiencies or differences in the reproduction technology.  Ordinarily, with relaxed listening, such differences could go unnoticed.

 

3. Further to 2, with an efficient psychoacoustic codec at a high bitrate (e.g. AAC stereo at 320kbps) the only way, generally, that  I can tell an uncompressed passage of music from a compressed version of that passage is by very careful listening to a short segment of the music, about 2 or 3 seconds' worth, played repeatedly so that I can have multiple opportunities to hone in on a difference.  I can use the ABX plug-in for foobar 2000 to do that. Without that assistance it is unlikely I'd be able to tell the original and compressed files apart.

 

4. I think it is  worth emphasizing that a person may fail to deliver a statistically persuasive DBT result if there is a large number of trials involved and the person is not allowed to rest. If I am sitting for an automated ABX test session using foobar 2000 and I feel I can no longer confidently answer whether X is A or B, I will take a 5 or 10 minute break, or I will choose another short segment of the file.  I will not proceed further with the ABX test session unless and until I regain my confidence I can really hear a difference. I refrain from providing random responses or "guesses". 

  • Like 1
  • Thanks 1
Link to post
Share on other sites
50 minutes ago, frednork said:

Yes, good point! but that is where the flaw of sighted listening is revealed the most, as people will use other senses/perceptions to make up for the shortfall of discerning ability.

 

to say nothing of peer pressure and group think.

 

51 minutes ago, frednork said:

And the other essential part regardless of familiarity of surroundings is being familiar with doing DBT's.

 

very good point

 

52 minutes ago, frednork said:

A really unhinged idea may be to start up a DBT club with your mates, meet weekly, and do some really simple DBTs' , introduced level of distortion vs none, volume difference, etc ie Things that we are pretty sure we can discern. And get used to performing the DBT and being tested and see how you go over time

 

close to where I am coming from  and also, yeah maybe a bit unhinged :) 

 

53 minutes ago, frednork said:

DBT's done by a group such as this, after some time, where there had been sufficient training would be a much better indication of what might be discernable or not under blind conditions when compared with the vast majority of tests I just went through.

 

 

only I wouldn't stress too much on the "sufficient training" aspect.  I think it's scientifically wrong to train people to pass tests of any kind.

 

Link to post
Share on other sites
11 hours ago, Eggcup the Dafter said:

Much as we like to think we’re special... are we?

Yes and very much so!  We are all special or unique.   No other human is the same as any other human living or now dead.  Therefore our all our various senses including hearing will all be unique to our respective self and nobody else. 

John

Edited by Assisi
Link to post
Share on other sites


While DBT is recognised as the gold standard, it may not be practical in most situations. I have suggested in the past that assistants running the test wear face masks, and this rise some chuckles from people ?

 

If not practical, single blind tests are almost as good in eliminating the unwanted effects of 'biases'. Does anyone know quantitatively just how much of an improvement one gain from using DBT instead of a single blind test? For example if the results from a single blind test is 60% reliable, would a DBT improve that to a 80% reliable (percentages are just made up for discussion purpose).  

Link to post
Share on other sites
3 hours ago, frednork said:

When you blind yourself, you put yourself under some sort of different psychological pressure. This blinding is a cue to you that this is different somehow to your normal listening (and at a concious/subconcious level you may need to respond differently)

 

The question of pressure is interesting. I would be surprised if it always worked the same way. Some people perform better under pressure (heightened concentration, more attention to detail), so you would reasonably expect that the effect would average itself out.

 

I have done blind tests of a few weeks duration with very low pressure. In fact for much of the time I think I forgot I was even doing the test.

I had a friend hook up two supposedly very different DACs to my system. DACs that were very very easy to tell apart sighted. They didn't tell me which DAC correspond to which input on my preamp.

I listened as I normally do, relaxed and enjoying the music and only switched between DACs every two or three days if I even remembered to… I thought there was maybe the tiniest difference between them when I switched. I thought I knew which was which but this always disappeared after a day or so of listening to one DAC and then I honestly couldn't tell any more.

 

The counterfactual test is something I've done on my kids who have far better hearing than I do (one of whom is a very accomplished singer and has a brilliant musical ear). I told them which DAC was playing each time and they described the same (big) differences every time I switched. They even described the same differences when I told them I had switched but in fact had not changed anything.

 

If I think of the possible explanations for hearing an obvious difference when the DACs were known and hearing virtually no difference when that knowledge was missing, it seems plain to me that the significant variable is the knowledge itself. Particularly when that knowledge can lead to completely incorrect conclusions (as was the case with my kids)

 

 

 

 

 

 

 

Link to post
Share on other sites
47 minutes ago, Assisi said:

Yes and very much so!  We are all special or unique.   No other human is the same as any other human living or now dead.  Therefore our all our various senses including hearing will all be unique to our respective self and nobody else. 

John

I agree, but in context I was actually intending to ask whether audiophiles are special when it comes to hearing differences, and whether that is a more important question than what the average human can hear. 
 

And for what it’s worth - beyond knowing about some setup issues such as out of phase wiring, I don’t think audiophiles have any special ability.

Link to post
Share on other sites
1 hour ago, LHC said:

If not practical, single blind tests are almost as good in eliminating the unwanted effects of 'biases'.

 

Can be very true.

 

1 hour ago, LHC said:

Does anyone know quantitatively just how much of an improvement one gain from using DBT instead of a single blind test? For example if the results from a single blind test is 60% reliable, would a DBT improve that to a 80% reliable (percentages are just made up for discussion purpose).  

 

You can't quantify it like that.  If the person running the test is skillfull and proficient in running tests without providing clues, then maybe there is no benefit at all.

  • Like 1
Link to post
Share on other sites
5 hours ago, aussievintage said:

only I wouldn't stress too much on the "sufficient training" aspect.  I think it's scientifically wrong to train people to pass tests of any kind

 

The training generally  is to improve their performance broadly under DBT and also to become more precise in their description of differences, However it depends on what the question is. If it were the question of the moment, Do ethernet cables make any difference.? Then any difference no matter how unrelated to normal music listening would be the first step.

  • Like 1
Link to post
Share on other sites
5 hours ago, MLXXX said:

1. If one ceases to be able to hear a difference because one is "under pressure to perform" that suggests that the perceptual difference is small to begin with. In other words, one's ability to hear the difference is only marginal.  In these circumstances it is arguable that one's ability to hear the difference is so tenuous that the difference made by the change (whatever it happens to be) is not worth worrying about! For practical purposes, the perceptual difference is so small as to be negligible.

 

I dont disagree that the differences may be perceived by some as marginal in the big picture (and probably should be acknowledged as being so by more) however I think the reason for the night and day type descriptions is that people get so fixed on the sound of their (often very good sounding) setup that they have tweaked painstakingly  to a certain level, that they are very happy with , and then they try something , it improves or changes things marginally, but to them it is a huge difference as they are highly accustomed to the sound of the system prior to that.  Whether these changes are "worth worrying about" in the end are up to the individual. I think a lot of arguments regarding this  are due to this mismatch of terminology.  Personally I am comfortable in saying that many of the changes  I hear in the more esoteric "tweaks" performed are nowhere near a night and day situation but however are striking at the time. To someone else less familiar with the setup , they may hear very little or no difference and I would not be surprised or disappointed. As unpalatable as it can be to talk about the cost of things, I think the very same people who hear night and day  differences with ethernet cables and similar would not be using them if their budget was more highly constricted as there are (much) bigger fish to fry at that point.

Link to post
Share on other sites
4 hours ago, LHC said:

While DBT is recognised as the gold standard, it may not be practical in most situations. I have suggested in the past that assistants running the test wear face masks, and this rise some chuckles from people ?

 

If not practical, single blind tests are almost as good in eliminating the unwanted effects of 'biases'. Does anyone know quantitatively just how much of an improvement one gain from using DBT instead of a single blind test? For example if the results from a single blind test is 60% reliable, would a DBT improve that to a 80% reliable (percentages are just made up for discussion purpose).  

 No, as long as care is taken that the "switcher" (for lack of a better word) does not indicate consciously or subconsciously what sample is being tested then it makes no difference to reliability. 

Link to post
Share on other sites
7 hours ago, frednork said:

I think the reason for the night and day type descriptions is that people get so fixed on the sound of their (often very good sounding) setup that they have tweaked painstakingly  to a certain level, that they are very happy with , and then they try something , it improves or changes things marginally, but to them it is a huge difference as they are highly accustomed to the sound of the system prior to that. 

 

 

This is a good explanation for the over-the-top descriptions for the "latest tweak"  that we often read.

 

7 hours ago, frednork said:

Personally I am comfortable in saying that many of the changes  I hear in the more esoteric "tweaks" performed are nowhere near a night and day situation but however are striking at the time.

 

Yes, a common one.  Night and day.  I have seldom heard anything that deserves that.

 

 

Link to post
Share on other sites
11 hours ago, sir sanders zingmore said:

The counterfactual test is something I've done on my kids who have far better hearing than I do (one of whom is a very accomplished singer and has a brilliant musical ear). I told them which DAC was playing each time and they described the same (big) differences every time I switched. They even described the same differences when I told them I had switched but in fact had not changed anything.

I have never seen this "technique" used in sensory analysis and would the expect the misdirection aspect of the test would make any result instantly invalid. If we accept that sighted listening can be affected through external (to the sound in this case) stimuli then this method ramps up that aspect.

Link to post
Share on other sites
13 hours ago, MLXXX said:

2. One can make a converse argument, that only when put under pressure to perform will a slight difference become noticeable (because of intensified concentration and focus).  Instead of listening to and enjoying the musical interpretation of the performance one tries to hear subtle deficiencies or differences in the reproduction technology.  Ordinarily, with relaxed listening, such differences could go unnoticed.

 

The state of play with many of these controversial topics is whether differences are audible to anyone, ever , that is the first thing to be proven. Then, we can move towards understanding how audible they are. to the "average" listener.

Link to post
Share on other sites
Posted (edited)
14 hours ago, MLXXX said:

3. Further to 2, with an efficient psychoacoustic codec at a high bitrate (e.g. AAC stereo at 320kbps) the only way, generally, that  I can tell an uncompressed passage of music from a compressed version of that passage is by very careful listening to a short segment of the music, about 2 or 3 seconds' worth, played repeatedly so that I can have multiple opportunities to hone in on a difference.  I can use the ABX plug-in for foobar 2000 to do that. Without that assistance it is unlikely I'd be able to tell the original and compressed files apart.

 

It all depends on what the question you are trying to answer is. If its "Is there an audibe difference?" then it is appropriate.

 

14 hours ago, MLXXX said:

4. I think it is  worth emphasizing that a person may fail to deliver a statistically persuasive DBT result if there is a large number of trials involved and the person is not allowed to rest. If I am sitting for an automated ABX test session using foobar 2000 and I feel I can no longer confidently answer whether X is A or B, I will take a 5 or 10 minute break, or I will choose another short segment of the file.  I will not proceed further with the ABX test session unless and until I regain my confidence I can really hear a difference. I refrain from providing random responses or "guesses". 

 

Totally agree, as simple and easy as these tests appear, there is definitely a point at which fatigue will affect the results. Training sessions are normally used to determine appropriate limits. The shortcut answer is , do as little as possible, but, usually testers wish to pack in as much as possible once the test has been organised, so, there is usually some tension there.

Edited by frednork
Link to post
Share on other sites
1 hour ago, frednork said:

I have never seen this "technique" used in sensory analysis and would the expect the misdirection aspect of the test would make any result instantly invalid. If we accept that sighted listening can be affected through external (to the sound in this case) stimuli then this method ramps up that aspect.


Its validity depends on the purpose of the test. Misdirection does not automatically invalidate a test - far from it.  Many tests in psychology fundamentally rely on it in some form (participants think they are being tested for one thing when they are actually being tested for something quite different). 

 


In a casual, relaxed sighted test with none of the pressure associated with “blinding”, it seems to me to show very clearly that the effect of sight (knowledge) can easily mislead. 
It also leads me to think that the reason for sighted tests being more “successful” is more to do with knowledge and far less to do with the pressure of blinded tests. 

Edited by sir sanders zingmore
Link to post
Share on other sites
35 minutes ago, sir sanders zingmore said:

Its validity depends on the purpose of the test

Please explain how using this technique would be valid in the testing of audibility of a change to an audio system.

Link to post
Share on other sites
3 minutes ago, frednork said:

Please explain how using this technique would be valid in the testing of audibility of a change to an audio system.


It shows that visual cues can influence the outcome of audibility tests to a far greater extent than audibility itself. 
That provides some very useful information about possible reasons for differences that are detected in sighted tests 

Link to post
Share on other sites
7 minutes ago, sir sanders zingmore said:


It shows that visual cues can influence the outcome of audibility tests to a far greater extent than audibility itself. 
That provides some very useful information about possible reasons for differences that are detected in sighted tests 

Well, that would be a study on the effect of counterfactual stimuli on audibility. To extend your argument further, one could also say that testicular electrodes that were charged with 10000v on a random basis are also a valid technique for testing audibility.

Link to post
Share on other sites
2 hours ago, frednork said:

I have never seen this "technique" used in sensory analysis and would the expect the misdirection aspect of the test would make any result instantly invalid. If we accept that sighted listening can be affected through external (to the sound in this case) stimuli then this method ramps up that aspect.

 

I think the test is aimed at demonstrating how susceptible we are to this kind of thing.   I think the test works well and clearly demonstrates what it set out to do.  That intention of the test is NOT invalidated.

  • Like 1
Link to post
Share on other sites
1 hour ago, sir sanders zingmore said:

Its validity depends on the purpose of the test. Misdirection does not automatically invalidate a test - far from it. 

beat me to it.  Indeed!

 

1 hour ago, sir sanders zingmore said:

In a casual, relaxed sighted test with none of the pressure associated with “blinding”, it seems to me to show very clearly that the effect of sight (knowledge) can easily mislead. 

 

I agree.   Not that I needed convincing.  It' something that is only needing to be demonstrated to those who deny it's power to persuade. 

  • Like 1
Link to post
Share on other sites
3 hours ago, sir sanders zingmore said:

In a casual, relaxed sighted test with none of the pressure associated with “blinding”, it seems to me to show very clearly that the effect of sight (knowledge) can easily mislead. 
 

I think this is a given and everyone is ok with this.

 

Now its the blind test that so many clamp on to, even though this is full of pit holes it seems if not done properly. Which I think most are done this way from the casual listeners.

So I think this is a great thread that frednork started that gives us some more perspective of some of the pitfalls of DBT tests. I think far too many make dubious conclusions from these that are not well implemented DBT's.

 

  • Like 4
Link to post
Share on other sites
1 hour ago, aussievintage said:

 

I think the test is aimed at demonstrating how susceptible we are to this kind of thing.   I think the test works well and clearly demonstrates what it set out to do.  That intention of the test is NOT invalidated.

 

Perhaps I was not clear enough.  If you are trying to determine whether a difference due to some change in a system can be heard, misdirection or testicular electrodes , both would not be part of a recommended protocol and (presumably) would reduce the effectiveness of the test. 

 

PLease remember this thread is about what makes up a good blind test, not about all the problems associated with sighted testing.

Link to post
Share on other sites
2 hours ago, frednork said:

To extend your argument further, one could also say that testicular electrodes that were charged with 10000v on a random basis are also a valid technique for testing audibility.

Can you explain how this extends my argument? 

Link to post
Share on other sites
1 hour ago, rocky500 said:

Now its the blind test that so many clamp on to, even though this is full of pit holes it seems if not done properly. Which I think most are done this way from the casual listeners.


As I see it, the biggest issue with poorly conducted blind tests is that they are far more likely to give false positives.

 

 

 

 

Link to post
Share on other sites
4 hours ago, frednork said:

 

The state of play with many of these controversial topics is whether differences are audible to anyone, ever , that is the first thing to be proven. Then, we can move towards understanding how audible they are. to the "average" listener.

I was considering mainstream cases where there is a difference in the audio that is apparent enough for at least some people to hear it unsighted.  My suggestion was that in that situation the challenge of listening out for the difference in a formal  DBT test could very possibly increase the likelihood of the difference being heard by a person (rather than decrease it because of "performance anxiety"). This would because the person might normally be so interested in listening to and appreciating the music that minor changes in the performance of the reproduction technology could easily pass unnoticed by them.

 

I was not considering those controversial cases where no one can pass a DBT and (i) there is no measurable difference in the audio to begin with, or (ii) a measurable difference so subtle it is at least an order of magnitude less than any human being has shown an ability to perceive unsighted.  Spending time refining DBT protocols for such unpromising situations is likely to go unrewarded.

 

As a general comment, I think it is likely to be of limited usefulness  to carry out a fully fledged formal DBT session with an individual unless an informal unsighted test of that person has shown promising results. Before testing myself formally I will always do a quick informal unsighted test. If I cant get a 100% correct result ín a quick informal test I see no point in embarking on a formal test involving something like 10 to 20 trials!  By the same token, if someone asks me to help them sit for a formal DBT, I first of all invite them to listen unsighted and tell me what they hear. Sometimes at that point they say they have no idea which version is being presented to them. It all "sounds the same".  In such circumstances there seems little point to me in going further at that time.  Of course it is possible they could start to hear a difference given time to practise or to "warm up", or if they tried again on another day, when their hearing was more acute.

  • Like 1
Link to post
Share on other sites
50 minutes ago, sir sanders zingmore said:

Can you explain how this extends my argument? 

Sure, the addition of misdirection, which is unnecessary and totally avoidable and is likely to have a negative impact on the subjects ability to discern an audible difference created by a change in a stereo system would not be used in any blind study about determining an audible difference created by a change in a stereo system , unless the test was about the (negative) effect of misdirection.  Testicular electrodes , although perhaps more unusual , would also not be used, as they are unnecessary and totally avoidable and are likely to have a negative impact on the subjects ability to discern an audible difference created by a change in a stereo system unless the study was about the effect of them.  you could happily pop anything in there but we already know that this should be avoided when actually testing for differences as opposed to  testing the effect of different stimuli on the ability to discern (which the blinding is intended to reduce)

 

I dont think I have said anywhere that misdirection wont change the result. It most likely will and should be avoided at all costs.

Link to post
Share on other sites
1 hour ago, frednork said:

PLease remember this thread is about what makes up a good blind test, not about all the problems associated with sighted testing.

 

One needs to keep the pros and cons in proper perspective.  I am sorry, but a list of bad DBTs must be viewed in context with even worse sighted testing.

Link to post
Share on other sites
  • Recently Browsing   0 members

    No registered users viewing this page.




×
×
  • Create New...