Shootouts With Meaning: Here's How
Everyone wants to know which loudspeaker is best for their applications. What could be more natural than to line several models up and listen to them? This is called a shootout.
EVERYONE wants to know which loudspeaker is best for their applications. What could be more natural than to line several models up and listen to them? This is called a shootout.
The idea is a sound one: line them up and have a listen. I always have advocated that the only way to account for all of the variables in the way that a loudspeaker sounds is to listen to it. You can't have a shootout with spec sheets. In fact, this is the very reason that spec sheets should be approached with a bit of skepticism. It's not that manufacturers are being untruthful, but there are many ways to come up with specifications such as power rating or sensitivity rating, and marketing encourages the use of methods that produce the biggest numbers.Logistical Problems
The problem with shootouts is not with the concept, but with the execution. Werner Heisenberg is famous for formalizing the idea that you cannot observe a system without disturbing it. Nowhere is this more true than when listening to loudspeakers. The response — and therefore the sound — of a loudspeaker is profoundly affected by its immediate surroundings. The same is true for the listener. A fair comparison would require that each loudspeaker be located at the same position (not possible without a time interval to change loudspeakers) and listened to from exactly the same position in space. This negates the idea of a listener group, because the presence of the others affects the outcome, and all the listeners can't be at the same place.
Loudspeakers interact with their surroundings in ways that change their sound. If you line up a group of loudspeakers in a row, the one in the center will benefit from the acoustic loading provided by the adjacent boxes. The ones on the ends will not. This is why some sound system designers put “wings” on their low-frequency systems. A row of loudspeakers means that each one has different wings, and this alone will affect significantly the low-frequency performance.Human Factors
While the human auditory system is a wonderful analyzer, it is not calibrated or consistent. You know what you are hearing, but you don't know why you are hearing it. Subjective judgments such as “Brand A sounds warmer,” can be describing a higher distortion level. A conclusion such as “Brand C sounds more open,” can stem from the acoustic comb filter produced by its interaction with the floor. The conclusion “Brand D has more kick,” can come from a mistuned cabinet that produces a low-frequency response peak.
A meaningful shootout requires measured data, and I don't just mean a real-time analyzer running with a mic connected. The complete transfer function of a loudspeaker must be assessed with complex data that includes both magnitude and phase information. The immediate environment around the loudspeaker and test microphone profoundly affect such data. You can't have a shootout in a test chamber.
Humans can only remember what something sounds like for a short time. If you listen to one box, then take it down and replace it with another, the time lapse in between ruins the comparison. If you set up both loudspeakers and switch between them, you have affected the result by listening to sound coming from different positions in space with different surroundings.
You can't do it with your ears alone, no matter how good you think your ears are. It is not physically possible to listen simultaneously to a half-dozen loudspeakers located at the same point in 3D space from multiple listening positions. Yet this would be the only way to do a truly meaningful comparison.Vision Factor
Even the way that a loudspeaker looks can affect the outcome. A techie trap box with a metal grill just looks like it should sound better than a white box with grill cloth. If you don't think that looks affect the sale, wander into the car stereo department of the local Circuit City. An objective shootout would require invisible loudspeakers, or at least a scrim that prevents the listeners from seeing them.
So to do it right, you have to be able to move each loudspeaker to the same point in space, make it invisible to the observer, and instantly switch to the next model. This has been done for small loudspeakers in a small listening room. JBL invested hundreds of thousands of dollars into such a setup to evaluate studio monitors. Imagine what it would take to do it in a full-size auditorium.What About Coverage?
Any loudspeaker in a sound reinforcement system will be covering an area. A loudspeaker may be perceived as being superior to another when listened to on-axis, but the outcome may be the different if evaluated from several different positions.
Loudspeakers will have differences in their “working distance” based on their directivity. It can be meaningless to compare different directivities at the same distance or at a distance that is too far or too near for either device. A studio monitor will always sound better than a large format touring box in a dead environment. But, move them both into a large reverberant space and the touring box will win every time.Which One Goes the Loudest?
If you run both loudspeakers up to 85dBA slow-reading, how do you know how hot each is getting, and how reliable it will be over time? You don't. The one that sounds “best” to you may be near failure. You had better include a laser thermometer in your demo kit to see how hot each is running, and that can only be done with a controlled program source like pink noise. The only way to determine the maximum SPL available from each is to take it to the threshold of failure, and doing so will likely permanently alter the response of the loudspeaker.
Evaluating loudspeakers is a lot like evaluating an HVAC or lighting system. The only way you can really do it is to install it and then evaluate its performance in all parts of the room. Little could be determined from a temporary setup, yet that is the only practical way to evaluate a loudspeaker.
If the design requires an array of the loudspeakers, listening to one box will reveal precious little about the performance of the array. When you cluster loudspeakers together you are forming a new loudspeaker. Its radiation pattern will be grossly modified from the single box. Some loudspeakers are more arrayable than others, but you would have to conduct a shootout between various arrays to determine this.