Reviews and reviewers - are they necessarily bad?

[jandl100]

... your reviews then you are very welcome to post them here, we obviously don't have the same level of readership, but we have quality instead of quantity

Thanks Doc. Hey, you do approve of me!
I fully intend to do so - that's why I started my system blog thread on this forum.

[Dr Bunsen Honeydew]

Its the compromises of professional reviewing, or amateur reviewing for a commercial entity, that I object to. The more members that review on forums, the more members attend bake-off with other members, the more we share our love of our hobby, the more we try to help the newby, without dictating to them, to find their own path, then the happier I am that this industry is on the way to a better place than it has been for a very long time.

[zebbo]

Well said that man.
The first problem though seems to be getting people to be at all interested in hi-fi in the first place. We had a bit of a party here over the weekend for my other half's 50th birthday with about 25 friends & family. The hi-fi sits in full view in the lounge with my lovely Anniversary having pride of place all nice and shiny so I thought I might get at least a cursory question or two about it, some interest but no, nothing. I've even sat people down and had them listen to some tunes and they're always impressed, (although possibly only being polite), but basically they couldn't care less. Just two of my friends are into hi-fi and we have good evenings in each others homes with tunes and beer but everyone else just calls us the hi-fi boys like we're some kind of wierdos!

[Dr Bunsen Honeydew]

Go into the news agents and look at how many mags there are devoted to Steam Locos, far more than for hi-fi. And yet there couldn't be a worse opinion than for train spotters in the general publics mind. Yet it is an industry that survives and flourishes on its enthusiasts, and so will we. The general public will always look on us as train spotters with ears

[Alan Sircom]

My take on this, point by point:

Should there be a role for reviews and/or reviewers in the idealised hifi environment of the future?

As things currently stand, I'm not sure. We are undergoing radical cultural change. In some parts of the world, hi-fi is still a subject of interest, but regrettably the UK currently does not appear to be one of those parts. Only one-seventh of our audience these days is based in the UK. Appealing to the home and away crowds is not easy, because they go in radically different directions (the UK readers want it small, cheap, minimalist, black and quite possibly made 25 years ago... but these traits are not universally shared any more). The UK has the advantage of speaking the world's lingua franca, and UK reviews are praised for being more concise than our American counterparts. However, we've all noticed that on the international front, our reviews are often read as PDF downloads from a manufacturer's website (or as part of its promotional literature) rather than as a magazine as a complete entity.

It seems that hi-fi enthusiasts in 2012 want to read independent, professional reviews (often from published magazines, because the printed page still seems to have greater gravitas on the buying decision than e-words – although this is changing)... but they want to read those to support their interest in a product, rather than regularly engaging with a magazine in its entirety.

1. What factors are out there in the real world that might tend to corrupt the reviewing process. What can be done to mitigate against this?

Many things. Partisanship. Personal relationships with manufacturers, distributors or dealers. Personal taste. Time constraints preventing due diligence. Failed and/or frustrated designers taking it out on what they perceive to be rivals. Self-serving reviews for freebies or accommodation prices. Strong personal agendas toward or away from measurement or particular product lines. Conservative reviewers squeezing out anything that doesn't maintain the status quo. Anti-heroes who praise broken boxes just because they are made in a shed. People failing to understand the science behind the subject. People hiding behind the science instead of expressing their preferences. Basically, all the 'human' things that can twist any critical process out of shape, whether through malign intent or benign naivety (and yes, you get both).

The process is never going to be fully correctable, because a subjective process cannot be 100% accurate, by definition. A subjective process is subject to the tastes of the critic and there will always be people who cry foul for a critic expressing their personal preference. The checks and balances of having a small pool of reviewers who know one another prevents 'dirty' reviews from taking place (for example, there are no cash bungs taking place - in the UK, at least - because if there were, my six year old Honda Jazz wouldn't be the shiniest, newest car in the car park when we all pitch up to a press conference). This has weeded out some of the more unsavoury elements in the reviewing fraternity, but the problem now is pool is currently not growing.

2. Are reviewers claiming to have Golden Ears, or do they simply see themselves as regular folk who want to write reviews because of their enthusiasm for hifi?

That depends on the reviewer. More specifically, it depends on the country the reviewer works in. American reviews are 'Golden Ear' driven (I am frequently requested to be less 'man of the people' in my reviews by my superiors, and to write as a Golden Eared kingmaker instead of an enthusiast), but the UK is heavily driven by a more enthusiast-led approach. I try to get away with both by adding a touch of humour to the reviews, which gets me off the Golden Ear demands and still just about appeals to the enthusiast-led approach. Sadly, I think the move is inevitably toward the Golden Ear style of review.

3. If a review is a worthwhile or useful thing, what should it be like and who should write it?

From a buyer's perspective, a review should be a combination of extended news item and short-list creator. There are so many audio products out there, a good reviewer should sift through the market multiplicity and highlight what they think is interesting and worthy of selection. Some think reviewers should provide the final answer rather than the short-list, but unless your musical tastes, room, audio system, ears, mains, environment and myriad other factors are identical to the reviewers, all you can hope for is some kind of commonality, thereby making a short-list. If you find yourself regularly agreeing or disagreeing with the tastes of a critic, you can probably take their value judgments as speaking to you. While you should never take that as a guarantee, it does mean that what works for them may well work for you.

The nature of reviews are changing, and I'm not sure what the end result will be and whether what it 'will' be like is necessarily what it 'should' be like. Each person has his or her own take on what it should be like. My take is they should discuss more about functionality, system matching and descriptions of performance based on real-world recordings and less about specmanship and gushing over audiophile recordings.

As to who should write them, I feel they should be written by those who love music and the sound it makes. Musicians and electronics engineers bring their own things to the party and these are equally valid, but run the risk of creating reviews that are almost useless to the end user, just as the input of a pattern cutter and a fabric designer will bring useful elements to a fashion magazine, but people don't read Vogue to know about stitches and wefts.

4. Should a review be "peer reviewed" like in a scientific journal?

Why? A consumer publication (in print or online) is not a scientific journal. Demanding peer review would be special pleading for hi-fi. Occasionally, we resubmit a product for review to two reviewers. And they rarely contradict one another, even if the conclusions end differently (as in "I like it because it does X/I don't like it because it does X").

We have a peer reviewed journal in the JAES.

5. Most folk simply don't get the opportunity to hear much gear. Are reviews based on a strictly limited experience really worth that much in terms of assessing what is out there in terms of relative value for money.

Generally, I'd say if you've tested three things at any given price level, you've got a handle on relative value for money in that sector. The more you hear, the more finessed those critical faculties get. However, I'd also say if you test more than three of the same thing in a month - as some box swappers do - you run the risk of only hearing the fireworks and missing out on the subtleties. Even as a full-timer (like most pro reviewers, I've developed a personal set of what I call 'palette cleansing' tools - like taking several day breaks from listening) it's easy to fall into the trap of listening for the sake of criticism, rather than listening for the sake of enjoyment.

6. Is there an important distinction between professional and amateur reviewers?

The dress code.

Pro reviewers tend to have a greater understanding of the wider market (as in, where a product sits on an international standing) that is difficult for an amateur to parse. This has upsides - a brand might be a world-leader with outstanding performance but local myopia means everyone here dismisses it as 'foreign' - and downsides - heaping undeserving praise on a so-so product because of the company's reputation. Pros should have a greater understanding of the legal implications of what they write - a part of the reason why pro reviews tend to be so positive is since the 1990s our knee-jerk reaction to bad reviews is to spike them rather than risk expensive libel actions, so we pre-select and quietly drop the products that don't make the grade. There is also Malcolm Gladwell's theory that 10,000 hours make the expert, and pretty much all the pro reviewers I know have confidently spent more hours than that in the execution of reviews.

Pro reviewers should have access to the engineers and designers should the need arise, however this is often not so clear cut because PR intervenes. It's possible that an amateur reviewer would reach the designer where a 'name' writer would be caught in the net by the marketing team. But generally, if a pro reviewer needs to dig deeper to describe a product under test, they have wider access to the people who designed the thing than an end user.

[Dr Bunsen Honeydew]

Well I am in a state of shock I am actually agreeing with 90% of what a reviewer has written

Also that took time and care to write and we should all thank you for that, so

[Alan Sircom]

Well I am in a state of shock I am actually agreeing with 90% of what a reviewer has written

Also that took time and care to write and we should all thank you for that, so

Sorry about that. Normal service will be resumed shortly.

[Dr Bunsen Honeydew]

I am sure it will :D

Now my 10% of disagreement may well be due to my current old fogey state. I have had nothing to do with, or even read, any magazine or met any reviewer since 2000, apart from very occasionally chewing the fat with Paul when we go to Broadstairs on a hot summer day and pop in on the way back for Nadia to sample his collection of strange drinks he has been given over the years sadly I can't as I am always driving, but it makes for an entertaining journey home.

Anyway too much flannel, back to the point. Maybe the publishing industry / reviewers is better now, in some ways it was getting better toward the end of the 90's, but for me it has lost all credibility having had to live and trade through the flat earth bullshit of the 80's and 90's. For me the whole process is blatantly corrupt as a human process. Only occasionally financial but more to do with the desire to be in the Salisbury / Glasgow good books or similar. I remember a conversation with a well known reviewer, no names no pack drill, where when asked why do you only ever use Naim in your reference system, the reply was well if I don't use Naim I don't get the level of work I need. What a sad state of affairs which corrupted this industry for about 20 years.

On top of that having to sue Hi-Fi Choice for a blatant deliberate hatchet job, which was then repeated when I won an out of court settlement, so at that point my opinion of the magazines and their minions reached rock bottom and was a major part in my decision to walk away from the industry for five years.

My opinion is obviously just that, my opinion. And it is clearly tarnished by observed reality, I hope it is changing, but to be cynical I doubt it, as commercial imperatives / consideration will always tarnish good intention.

[Dr Bunsen Honeydew]

Reviewing is just so simple "it sound great / horrible / boring in this system (description), with this music (list), in my room, to my ears". Obviously you can flannel and entertain, but that is the basis and helps spread the word on a product around the hi-fi enthusiast sphere. That can be done by anyone, and now we have forums should be done by everyone. We just need to control and filter out the spammers and shillers.

[jandl100]

Many thanks for your input, Alan - that was enlightening and refreshing in its honest appraisal.

I've been an amateur reviewer on some of the forums for a few years now and seem to have built up a fair following of folk who like to try the things I have liked. The kit I have reviewed has generally just been obtained as part of my "boxswapping journey". I find it enjoyable and rewarding (emotionally, not financially) - and it's also good to help small businesses, who often sell direct and who are providing very good value product, but who find it a real uphill struggle in a market place dominated by the 'majors'.