This project propelled itself in one-month cycles.
Each phase involved gathering both quantitative and qualitative
data pertaining to questions generated from the previous phase,
analyzing the data, and thengenerating new paths to explore in the
next phase. Each phase usually consisted of about 4 questionnaires
focused on different aspects of the game. Data from different phases
was linked using a participant's email address so that important
data, such as a participant's scores on a set of personality scale,
need only be gathered once. More importantly, data gathered separately
could be analyzed together. The resulting database of information
is useful because it allows access to both the big numerical picture
and individual qualitative responses. Participants were encouraged
to complete as many questionnaires as they felt comfortable doing,
and the underlying database structure was described to them briefly
so that they understood their continued participation was important.
Participants were recruited over the Internet for
this project. At the beginning of each phase, participants already
in the database were invited to participate in the new phase through
email. At the same time, a standardized message was posted in online
message boards and forums that are frequented by EverQuest players.
Participants were also encouraged to tell their
fellow EverQuest gamers about the project and to spread the link
to the project main page. It is not known however whether this helped
with recruitment significantly.
This project collected data from 5 phases
in the period between September 5th 2000 and May 5th 2001. These
included 13 multiple-choice forms, 7 free-response forms, and 3
Flash-implemented experimental designs. Approximately 4000 individuals
participated in the study, and filled out about 25,000 forms altogether.
These estimates are generated after the exclusion criteria have
been applied. Most multiple-choice forms had at least 1500 responses.
There was a participant carry-over rate of about 25% from one phase
to the next.
Below are the criteria used for excluding submissions
from the analysis:
--Duplicated/Repeated submission, by comparing the email address
--Submission with missing email address field, even if all other
fields are completed
--Submission with more than 20% of fields left blank
--Submission with obviously impossible information, ie. age=2, email@example.com
Sampling biases and representativeness are important
issues to consider in any kind of empirical research. In the months
since the first release of this report, I have noticed concerns
about representativeness surfacing in several message boards which
linked to this study, and also in several email correspondences.
One individual presented a well-articulated critique of the representativeness
of my sample:
When referring to the results of your study, you
say "x% of EQ players are <blank>". However, in
reading your methodology, I notice that your study does not choose
EQ players via a random sampling method. Instead, you primarily
rely on EQ websites and to a lesser extent word of mouth to find
your study's participants.
It seems to me that this skews the data heavily in favor of the
most devoted EQ players. The casual EQ population who don't bother
to read EQ websites will be almost entirely overlooked in your
data, even though I'm sure they represent a significant portion
of the EQ population. Also, by using the volunteer method to get
your participants, you automatically filter out all the EQ players
who don't bother to fill out surveys.
Thus I find it odd to derive from your study any implications
about the EQ populous as a whole. So the phrase "x% of EQ
players" simply doesn't apply; rather, your results reflect
"x% of EQ players who read EQ websites and who like to fill
out surveys". There are probably some big differences between
those two statements.
I would answer this particular critique and similar
ones with the following arguments:
1) EQ is one of the few games where going to these
websites will help a lot, to the extent where a lot of casual gamers
probably frequent these sites as well. For example, consider scheduled
server downtime, quest info, class strategies etc. So it's not clear
whether this creates a heavy skew at all. In fact, since playing
the game necessitates a connection to the Internet on a decent computer,
it is probably only a tiny percentage of players who have never
gone to an EQ website.
2) Through a correspondence with Sony Online/Verant, I found that
my basic demographics match theirs very closely. For example, the
percentage of players who are female was 16% for both our data.
And average playtime per week was "around 20" for them
and mine was 22.
3) Gender and age differences probably don't interact with whether
someone goes to an EQ website or not. So we know that female players
are more likely to feel that their EQ friendships are comparable
to their RL friendships, but this difference probably doesn't only
occur among people who go to EQ websites. And in a sense, these
age and gender differences are the more important part of the study.
4) And the same for people who don't bother to fill out surveys.
They probably don't differ in important ways from people who fill
out surveys, in the sense that almost all of the findings would
remain the same even if we did somehow take them into account.
5) Finally, since "hours played per week"
was one of the collected variables, I can check whether this impacts
the other variables I am measuring. So even if my sampling included
a very skewed proportion of heavily-devoted EQ players, I am not
oblivious to the effects of this skew. On the contrary, I have a
good way of determining how severe these effects are.
In essence, I feel that most EQ gamers frequent
EQ websites, and that a non-random sampling of these gamers who
go to websites does a good enough, though clearly not perfect, job
of representing the entire EQ player population. And that while
slight skews do exist, that my findings would not differ significantly
from a study that was able to take random samples.