A few weeks ago I made a post about playtesting my ttrpg prequel with the question:
“how early can you test a game?”
In this post, I ‘ll talk about a time I actually tested before I even had a game.
How is this possible? Why would you do this? What materials are required?
I’ll answer all these questions in this post.
Last year I was working on another of my projects, Level: Adventures in a Virtual Fantasy, a ttrpg about playing out adventures in a futuristic fantasy mmo like those scene in Ready Player One or Sword Art Online. I was still working on the core systems but wanted to get insight into possible player interest and expectations about the product.
To do that I turned towards my UX (User Experience) background and decided to conduct a User Exploration Study.
What is a user exploration study?
A user exploration study is methodology focused gaining insights and expectations about a product from prospective customers. They are usually conducted early in development to help validate a product need or gain an understanding of possible unknown competitors. An exploration study can be organized into different formats but most commonly takes the form of a survey or focus group. Insights gained around an Exploration study are usually qualitative, focusing around exploring the potential user’s relationships with products in that fill a similar market.
Study goals
The goal of the study was to identify the relationship potential users to the game’s references (fictional media about being in a hyper-realistic fantasy mmo and “mmo style” gameplay patterns overall) and their impressions Level’s premise. The hope was to be able to use the findings to identify any unknown competitors and insight into what features to prioritize.
Study methods
The study was organized into multiple parts
a consent form for every participant to complete before the study
a pre-study survey to answer some establishing and ranking-based questions
a series of discussion focused questions
A feature shopping exercise
Participant Consent Form
[Example Material: Participant Consent Form]
The participant consent form is an important first step for any user focused study or playtest. Especially if you plan on seriously collecting data (quantitative or qualitative) it’s important to give participants clear guidelines on how their information will be used and the chance to opt out before or during the course of the study.
Pre - Study Survey
[Example Material: Pre-Study Survey]
The pre-study survey was a series of questions focused on getting data on the user’s background and preferences before the study began. In a more robust study, I would probably have used the pre-study survey as a screening tool, to either make sure I got a more specific audience of participants or to create follow up questions to dig into the group’s background in the focus group session.
If you were running a large study, you could also use the pre-study data to tailor the audience make up of each group (for example if you wanted to group together participants with similar player motivations.) The pre-study can provide extra background that may give context to feedback taken during the study.
I used the study to get a baseline for the participants background in tabletop roleplaying games and MMOs.
I then followed up with asking them to rank their familiarity with a collection reference media and their relationship with the “crunchy” in games.
Focus Group Questions
[Example Material : Focus Group Script]
The focus group questions focused on talking about the player’s impressions and related around experiences. The conversation was intentionally open and deliberate, giving every participant and opportunity to speak and asking follow up questions. Having players only presented with the game’s basic premise allowed for me to dig into their raw expectations, asking question about what they would expect without any context.
Q: What comes to mind when I say “virtual fantasy”? Do you get excited or discouraged by the idea? Are there any properties (games, tv, books, movies) that you feel embodies that fantasy?
Participant A: “99% of the time is European fantasy, hoping to see fun fantasy in other cultural”
Participant B: “Something Low tech with interesting creatures”
During the conversation I was actually able to have an open discussion around someone of the possible problem areas I see for Level, the player’s experience “leveling” and “grinding”.
We talked about the participants experience with MMO’s and the related media listed in the pre-study. While most of the participants were familiar with a number of the popular anime basically no one was familiar with the references from Korean manhwa and lite novels.
Q: Were you familiar with any of the fictional properties listed in the Pre-Questionnaire? Was there anything you were surprised to see there? Was there anything you expected to see but didn’t?
Participant B: “Alot that i haven’t heard of before”
Participant C: “I didn’t know about half those either”
I also learned about few unknown indie tabletop roleplaying games and actual plays with a similar premise.
Feature Shopping Exercise
[Example Material: Feature Shopping Sheet]
Once the we had gotten through the focus group questions, I ran the participants through a “feature shopping” exercise. The goal of “feature shopping” is to provide the participants with a list of possible game features and a fictional “budget”. By providing more options than one could possibly buy, participants are forced to make choices about what items they would prioritize.
We went around the table and had each participant run through their “shopping cart”. It was extremely interesting to see the choices and impressions the participants had of each feature with only the context of their name to go off of.
To follow up we went through a series of further discussion questions where participants had the opportunity to discussions what options surprised or excited them. One option that surprising came up repeatedly were features related to building a guild and claiming land which in terms of development was a something I was thinking of as a minor or stretch goal feature.
Players were given the opportunity to first add 1 more item to their shopping cart and finally the task to remove 1 item from their shopping cart. Curiously 4 out of 5 participants removed “Creating Custom Skills” from their cart which actually was a core mechanic to the Level’s concept.
The process was illuminating in shining the types of possible features players would want to see from the complete game.
What I Would Do Differently
Overall the experiment of blending a UX exploration study with concepting a tabletop roleplaying game was a success. The materials were easy to create and going through exercises provided unique insight that illuminated the assumptions and media reference points participants had to Level’s concept.
In what I would do differently, first I would take more time to going through the study’s notes and recording to pull additional insights. Unfortunately, over the course of the year I’ve lost the study’s recording and my remaining notes from the day are sparse (probably because I was relying on the recording lol). Having extra care or adding an additional moderator to serve as session note taker would improve the quality of recorded findings and the breadth of potential insights gleamed.
Secondly, I would increase the number of potential participants and focus groups. Although I was lucky enough to have relatively qualified (as in fit the demographic of a potential player) participants, using the pre-study survey as a proper qualifying tool or too sort the focus groups could open the door to adding additional focus group questions or seeing how different groups homogenize around topics.
In Conclusion
Whether this peaked your interest in running your own Explorative Study or just bringing some UX principals into your tabletop development processes, I hope you got some value from this topic.
If you liked this post, on other early stage develop posts on competitive analyses and early playtesting.
I would super encourage you to dig into the work and writings of actual UX professionals like Ambrose if you want to learn more!