A Brief Primer on Methods and Critiques
Match with aggregate WoW census data. Perhaps the most compelling support comes from a data source that isn't based on the survey methodology. The PARC PlayOn data is based on census snapshots from within World of Warcraft. Every 10-15 minutes, an automated script collects information from every single character online. In other words, this data source is almost as perfect as we can get in terms of actual behavioral data.
For example, from the survey data we were able to get the average achievement motivation of players who prefer different classes in WoW. From the PlayOn data, we were able to get the average time it take different character classes to make each level. The classes who score low on the achievement motivation (survey data) are also the ones who level the slowest (census data).
Of course, not every data point from the survey can be supported by the census data. After all, the census data can't track RL age or gender information. Nonetheless, this congruence goes a long ways to showing that the survey data is able to reflect actual behavioral data in a real game.
On the other hand, there are no perfect methodologies. Every methodology comes with its strengths and weaknesses. But the weaknesses of a methodology must be understood in their proper context and scope. Thus, while some argue that the sampling bias invalidates the study, it in fact has very little to do with the bulk of the findings presented at The Daedalus Project. Furthermore, their critiques often miss the important point that there are seldom optimal solutions to many problems. It would be nice if we could survey all MMO players. It would be nice if everyone were willing to fill out surveys. But social science researchers are not omnipotent and the world is not perfect. Thus, sometimes the most practical and sensible way to answer a question isn't perfect. But it would be unfair to judge The Daedalus Project against a perfect world of omnipotent researchers. Finally, there really isn't that much empirical information about MMO players – who they are, their preferences, and what they do in the game. And here's what I've always believed and what led to the genesis of the Daedalus Project. Knowing something is better than not knowing anything at all.