Surely the reference point is not the master tape or some other contrivance, but instead 'sounds made in nature'.
By this I mean birdsong, a human vocal, a traditional musical instrument, the purity of a triangle being struck (just the ringing of metal on metal contact - afterall) water cascading along a mountain stream, the magnificent scale and authority of the sound of thunder! ...coming to you direct from the source straight to your ears, no electronics involved.
If your sound system can approach sounds made in nature, then everything else will fall into place in terms of music listening. That is why Peter Walker's claim in 1957 that his Quad electrostatic loudspeaker is the closest approach to natural sound, has stood the test of time. I suspect horns have this knack of producing a natural acoustic, which leads many to believe they are the pinnacle of realism.
That's where I beg to differ 'greatly' (about un-amplified sounds) being the most accurate reference!
Because we mainly DON'T listen to 'un-amplified' sounds on our systems and the engineers (who have produced the music we listen to, onto a master ), have geared the resulting sound to be played through 'amplified systems'!
An recording engineer 'doesn't' use the most 'revealing and natural' system to 'produce' his music (assuming we are talking a band situation). Whatever system (including speakers (or headphones) he uses is the most accurate reference point 'because' he is the 'only' one to determine 'exactly' what he wants the end result to sound like! And without knowing what he has used (as 'his' reference point, then we are only guessing as to what we think is the most 'accurate' reproduction situation.
And as no two albums have the same 'mastering' ( therefore 'master tape'), (therefore 'reference point'), then EVERY album will have a 'different' reference point so that means we can never really aim for a 'right' (or even more accurate) reference point for a recording because every recording has been mastered 'uniquely' by a different engineer using different equipment to 'evaluate' his personal preference of the end result!
IMHO a 'reference point' is like looking for the 'holy grail' (which does not exist)!
The 'One Talk' system of distributing wealth earned among the whole family 'discourages' Capitalist thinking!