Confessions of A Diary Keeper
Each survey day generates great heat and emotion, as revenue and reputations go on the block.
But what’s often forgotten in the rush, is how the cold numbers actually come from real people.
Here’s just one Australian diary-keeper’s story, along with some technical background, and CRA comments to help shed light on the mystique and myths of the ratings …
I was recently contacted by someone with an industry connection as a multi-media service provider, but eligible under GfK rules to participate in the survey. She had offered not to accept the diary, but was told it would be OK.
She completed her diary as objectively and meticulously as possible, and wanted to provide Radio Today with her insights into the experience.
Best that her identity is anonymous, but I can personally vouch for this person’s integrity in wanting to tell her story, to genuinely help the radio industry in an often controversial area.
Measuring radio listening is an elusive beast, and all methodologies face great challenges in reliably capturing behaviour that is both habitual and changeable as the wind. GfK runs a tight ship with very high standards of quality control to meet the challenges.
It’s critical for commercial radio’s credibility to have a measurement system trusted by advertisers and marketers.
But it all comes down to people, and how, in their busy, non-radio lives, they engage with the survey process.
So as a One-on-One research exercise, here are my question prompts, and the honest Confessions of a Diary Keeper. I’ve also added further background notes in green after some of the comments.
You completed a paper survey – were you given a choice? Would you have preferred an e-diary, if so why?
There was no choice – I was given the paper booklet, and the consultant asked me some questions about which were likely to be the most listened to stations, and put those stickers in the book for me, ready to fill in. I have no idea whether I would have preferred an e-diary because I have no idea what they’re like.
20% of the Australian sample is recruited separately to complete online diaries. Other systems such as the UK’s RAJAR allow a choice – which is now up to 50% of respondents just on desktops/laptops, not yet on mobiles. Usually based on extensive trialling in each country.
How did you find the instructions, both verbal and printed?
The instructions were clear, but they are very detailed. There is a LOT to be filled in. For each 15 minutes you have to identify which station, where you are, what device you used … and if you listened on a mobile phone or tablet, that’s a 4th box you have to cross.
Just the simple fact you have to put a cross means two pen strokes for everything you’re filling in, instead of only one.
The whole ‘cross instead of tick’ is really old fashioned thinking and harks back to the days when everything was manual, and ticks weren’t allowed because they stray out of the box areas. But when you have to fill a minimum of three boxes for every 15 minutes, it’s really tedious and annoying. It’s only radio ratings, not a federal election.
The use of online diaries is growing in many countries, because it’s much easier to tap on location of listening, platform, device etc, once you’ve selected your quarter-hours listened.
What day of the week were you asked to start recording your listening?
Started Friday and finished Saturday the week following. And it was Melbourne Cup week. So that’s three anomalies: two Fridays, two Saturdays and a public holiday (only in the Melbourne metro area.) How they weight for that I don’t know.
The large samples and length of survey periods help smooth out individual anomalies.
How was your engagement level as the week progressed?
I didn’t fill in as I went, because the boxes are so small I had to wear my reading glasses every time I filled in the diary. So I would catch-up every couple of days.
There were a couple of periods of in-car listening that I know I missed, but you know, when I get out of the car, I’m usually doing something straight away – it wasn’t my first thought to go and fill in a ratings diary.
By the third day I was really, truly over it. I persisted only because I have friends and clients who work in radio and the Networks they work for, worship the radio ratings. I have no idea why though. I always used to think the ratings were old-fashioned.
Now I’ve actually participated, I’m also highly dubious about how reliable they are. I also wonder how much variance there is between online and manual results – and how GfK thinks they’re weighting or adjusting the results to take that into account. Even I don’t know what I might have recorded if it were easier to do.
What was it like trying to accurately record your listening at each time of day, while presumably doing other activities …and then juggling stickers and x-ing each quarter hour?
The GfK rep asked me about the stations most listened to, and put the stickers on for me. Every quarter hour was a pain in the ass.
The diary-keeping period is kept to one week so it doesn’t become too much of a burden.
If you were listening in-car, how would you reflect flicking between stations?
They’re very lucky I’m not a flicker – I’m an old fart so I actually concentrate on driving while I drive. My daughter controls the radio dial – and she usually sticks to one station.
The question makes me think though – if my daughter flicked,, and I’m not really paying attention to the radio – what SHOULD I put in the diary?
Should it be ‘no listening’ because it wasn’t my choice? Should it be ‘no listening’ because I wasn’t paying attention, or should the station and duration be recorded regardless? Human nature being what it is, if I wasn’t sure, I would have entered ‘no listening.’
So that makes me wonder how much in car listening is not being recorded?
My understanding is that anything under 8 minutes of listening that could be called flicking, is not needed to be recorded. Apparently minimum listening benchmarks can also be set in a meter system.
How much of your recorded listening do you think was accurate right at the time of listening vs. having to rely on recall of an earlier occasion?
Only about two days while I was cooking and doing housework. The radio went on in the morning and stayed on the same station until the kids came home from school, when it was turned off. So it was easy to fill in that day (well after the fact).
What would have been your longest recall period between a listening occasion, and the first opportunity to record it in your diary? Three days.
In the diary there are 8 options for Where and How you listened next to each quarter-hour – what was it like keeping track of those options?
I only listen in car, and on DAB+ so I didn’t pay any attention to the other options at all.
Based on all this, how would you feel about the option of carrying a people-meter or agreeing to have a “meter-app” on your smart-phone? Much better idea. Should be EASY for the listener.
This is the current great debate in radio research circles comparing paper/online diaries with meters. There are trade-offs on both sides, and industry bodies have to make hard choices.
Any other comments about your experience?
There was nowhere on the ratings diary that specifically asked about the process itself.
There were at least 24 ‘lifestyle’ questions that asked about other media, intended purchases, Internet use and various other things. They reminded me of old fashioned ‘intention to purchase’ shopper research.
No mention of Pandora, Spotify or Rdio – by name or even by reference to ‘streaming’ services. Or of music listening habits. Or where I turn to for news, traffic, weather. It did ask about social media but not why.
There were no additional questions about radio in general – nothing about advertising – too much, too little, quality of. Nothing about what I might rely on radio for? No questions about overall quality of announcers, team shows vs partnerships, nothing about general listening habits i.e. more breakfast, more drive. Nothing about have I ever entered a competition, why or why not.
I was spending a whole week (supposedly) with radio in my brain while I was filling in the diary – and they didn’t ask me anything else about radio, or specifically about radio’s competitors?
The Gfk service includes Audienscope, a separate online survey that addresses these sort of “why” issues in more depth here.
This is strictly a “sample of one”, but it does highlight the large commitment needed from diary-keepers.
We’re fortunate in Australia of having one of the higher research participation rates in the world, and that people still care enough to track their radio listening in great detail for a week, to the best of their ability.
Certainly reinforces how top-of-mind recall is critical in the diary system, and will logically favour a listener’s “habit-menu” stations, formed over the long-term, by cut-through content, branding and marketing.
But inevitably poses questions about the sustainability of diaries in the listener’s world of time-stress and fragmented analogue and digital media choices.
Joan Warner – CEO of Commercial Radio Australia has provided Radio Today with further interesting comments and insights on the choice of methodology …
“The ‘diary keeper’ like almost all of our radio audience measurement respondents sounds passionate about their radio listening and is keen to have their feedback heard. Which is why GfK data collectors are able to get more than 60,000 respondents a year to participate in radio audience surveys.
The Australian diary system is one of the most robust radio measurement systems in the world. The new enhancements by GfK including the introduction of 20% online diary completion plus the automated fieldwork capabilities, has modernised the system of data collection while still ensuring that the integrity of the currency is maintained.
As an industry we continue to monitor the inclusion of 20% online diary completion, as we do all innovations in radio audience measurement. While there are some promising electronic technologies, none are currently capable of providing robust and reliable data as the sole measurement tool to take into account the mobility and ubiquity of radio.
The diary methodology is world class and is the global currency except in a very small number of markets. Meters are three times as expensive and have their own issues and result in a much smaller number of people in the sample. The fact radio is anywhere anytime means one methodology is not likely to catch all listening in the future which is why we have added an e-diary and will be exploring with GfK over the next few years any relevant and cost effective electronic options to add to our suite of tools but not replace diaries.”
Thank you Joan! And for more information on how the ratings work, the Gfk Radio Ratings Toolkit is available here.
This is the common experience when radio faces the “meter question”. Against the positives of the “granularity” of minute x minute data, matching with TV, and no reliance on recall, are weighed the major issues of cost and sample size.
There’s no one black & white answer. And as marketers and product teams will more and more need a 360° view of listeners’ media consumption, the search for the Holy Grail will continue for a long time yet.
Eriks Celmins is Managing Director of Third Wave Media and InsiderFocus, consultant for research, strategy and content. Full Member of Australian Market& Social Research Society (AMSRS). Find him on LinkedIn