Researchers have a term for results that not
only run contrary to what they expect but seem to fly in the face
of reality as they know it. Such results, they say, lack face
value.
I recently received a
press release with the results of a survey that ranked consumer
magazines, including consumer travel magazines. Im no researcher,
but my first reaction when looking at the results was to question
their face value.
According to the 2005
Publication Readership Satisfaction Survey, produced by Monroe
Mendelsohn Research, the top three consumer travel magazines are,
in order: Departures, National Geographic Traveler and Budget
Travel. Departures was also fifth among all consumer publications
in the Overall Excellence category.
If you havent ever read
Departures -- or even seen it -- youre not alone. It only goes to
people who carry American Express Platinum and Centurion cards and
is not available on newsstands.
I called Robert
Shullman, the architect of the survey and a senior vice president
at MMR, to ask how a limited-circulation magazine could land at the
top.
Shullman provided me
with details about the methodology, some sample questions and some
background. He said Departures circulation, though controlled, is
more than 800,000, which puts it in the same size category as some
of the major consumer travel magazines found on
newsstands.
He volunteered that
Departures is a client of MMR (and in the spirit of full
disclosure, Ill add that Travel Weekly, too, has hired
MMR).
One aspect of the
methodology that jumped out at me -- and likely has some impact on
Departures prominence among general circulation magazines -- was
the description of the sample. Of the 16,000 consumers
participating in the survey, 9,000 were drawn from all income
groups, but an additional 7,000 are labeled as affluent, which is
defined (rather generously) as having a household income of
$85,000-plus.
By including those
extra 7,000 affluent consumers (in addition to whatever affluent
were among those drawn from all income groups), was the deck
stacked in Departures favor?
Shullman said the
results were sample balanced: Data were reweighted after they were
tabulated to relevel the playing field.
It was necessary to
oversample the affluent, he said, because a survey of respondents
who reflect only incomes of Americans in proportion to the general
population will leave a publication like Departures shortchanged --
it would not have enough participating subscribers to get a valid
result, even though its circulation is comparable to travel
publications found on newsstands. Unless its potential readership
is forced into the sample, itll never register enough volume to be
included in readership surveys.
Because publications
that sell the most travel ad pages cut across different
classifications (travel, food/wine, outdoors, personal
finance/investing, etc.), I asked Shullman to compile an Overall
Excellence ranking -- with Departures added on -- for the 10
publications that sell the most travel ads. That ranking is listed
below.
Departures competitors
that did not do well in the MMR survey may take the position that
the results lack face value. But given the industrys growing
emphasis on high-end markets, its unlikely they will attack the
studys fine-print limitation -- that the results will be of most
interest to advertisers striving to reach affluent
readers.
To
contact Arnie Weissmann, send comments to [email protected].