This is the last in series of posts about rapid prototyping for game development with high school students. I will use one of our games, Teen Angst, as a case study for what to do if things don’t go according to plan.

Teen Angst had the broadest scope of any of our games. The plan was to use game mechanics to shape decision making about three topics that most interested teenagers: relationships, substance abuse, and nutrition. Players answered a series of questions based on scenarios that were presented via PowerPoint (Figure 1). Points for four resources (Health, IQ, Friends, and Money) were gained or lost depending on the most likely consequences of the decision.

There were several design flaws in the game. First, there was no means of controlling flow. Task difficulty was not adjusted based on performance, which typically results in either boredom or frustration for the player. Second, we adopted a linear narrative that limited choices to only a few options. While the psychology literature indicates that too many choices can paralyze a person with indecision, the game world is full of examples where presenting players with more choices increases their engagement with and enjoyment of the game. In the game world, having a choice means the player is in control of the outcome, and thus they are more likely to engage with the system. As educators we should capitalize on this phenomenon and reconcile it with what we already know – ownership of the learning experience is critical to learning outcomes. Third, the game was unbalanced. In a perfectly balanced game, all the probable outcomes have an equal likelihood of occurring. A perfectly balanced game (e.g., “tic-tac-toe” or “rock-paper-scissors”) is also called a zero-sum game because opponents have an equal opportunity to win. The reward-punishment contingencies in a game, the player resources, or other factors that affect the final outcome can also be out of balance. In our game, the reward-punishment contingencies were not evenly distributed across resources. Even though the number of questions pertaining to the topics and resources were balanced, the point allocation for each topic-resource combination was not balanced. For example, players had more opportunities to gather resources for Health than the other resources.

We were aware of all of these design issues going into data collection, but we didn’t realize they would have such a strong impact on the data. Six high school students participated in the experiment, but the data for one was removed because the instructions weren’t followed properly. Subjects played the game and provided responses on answer sheets. As I mentioned in a previous post, there was an error in creating the answer sheets, which made it difficult to relate individual answers to their corresponding questions. However, we were able to compare the points earned from the first half of the game to points from the second half. The prediction was that players would earn more points in the second half because of practice effects.

Interestingly, performance exceeded chance levels during the first half of the game (Figure 3), which suggests that subjects were attentive and understood the rules. Nevertheless, contrary to our expectations, performance decreased during the second half of the game (Figure 2). Data were combined across all subjects and all categories (i.e., Relationships, Drugs, and Nutrition). For each resource (i.e., Health, Relationships, IQ and Money), performance during the second half of the game was worse than for the first half of the game, c² (1, N = 157) = 80.067, p < 0.0001). Similarly, performance during the second half of the game was worse when data were combined across resources (Figure 3), c² (1, N = 157) = 25.28, p < 0.0001). This decrease in performance might be attributed to fatigue. However, it is more likely that this effect is the result of an imperfect game design. The data were consistent with post-game interviews where the players reported being bored.

After six weeks of hard work, a result like this could be devastating to a student. At worst, the student might doubt the scientific method and loose interest in science. It is critical to spend as much time with the student as possible to confirm that they understand the value of impartiality, learn from failure, and persist in their quest for truth. I found it useful to recount my personal experiences with failed experiments as well as examples from famous scientists. Shifting the focus to improving the game was also helpful. However, it was particularly interesting to find that the student found some solace in knowing that her results were important because they provided evidence for the lab’s overarching hypothesis, namely, that properly employed game mechanics are useful for education. In her case, an imperfect design resulted in a baseline to which future iterations of the game will be compared. We both learned a lot from each other, and the student is sure to benefit from this experience in the future.

Figure 2

Figure 3

Face Finder is designed to teach students about their own cognitive biases for ethnicity and gender. Players work to solve a murder mystery by guessing the identity of five characters (Killer, Accomplice, Witness, Bystander, and Victim). Players update their guesses during each round of play, and they cast a final guess after the 40^th round. Players base their guesses on clue cards, face cards, and character cards. Only character cards are required to solve the crime, but we predicted that players would choose face cards that demonstrate an in-group bias for benevolent characters (i.e., Victim and Bystander) and an out-of-group bias for malevolent characters (i.e., Killer and Accomplice).

Nine high school students participated in the study, and they reported their ethnicity in alignment with one of the five ethnicities in the game (Caucasian, Asian/Pacific Islander, Indian/Middle Eastern, Hispanic/Latino, and Black). Players were informed that categories were loosely defined and to choose the one they identified with the most. The rules of the game were described in a previous post. Before describing the data, I should mention that there were errors made during data collection. Rather than making a guess for all five characters on each round, players only made a guess for one character. Consequently, feedback for each guess was too specific and players guessed the identity of the characters more quickly than we envisioned. To compensate, we collected data from the second round of play, where players were more likely to select any ethnicity they wanted for the characters.

Data were analyzed relative to a baseline. Because there were five ethnic categories, a player had a 20% chance of selecting their own ethnic category as the Victim or Bystander. There was an 80% chance that they would identify a Killer or Accomplice as out-of-group. Subjects’ guesses were categorized as supporting our prediction or not. The number of guesses that agreed with our prediction (24 out of 36) were counted and compared to the number expected by chance. A non-significant trend was observed for each category (Figure 3), where subjects were making more in-group biases for Bystander and Victim and more out-of-group biases for Killer and Accomplice (Chi-squared, p > 0.10). A significant trend was observed for all guesses summed across categories, and this trend was greater than expected by chance (Figure 4)(Chi-squared, p < 0.05). We observed ethnic biases for judgments of character in our game, and this data was further supported by qualitative reports from the subjects themselves. Players reported making judgments based on the ethnicity of the Face Cards when they were irrelevant to the task.

Despite the errors we made in collecting data, our game proved to be a valid tool for exposing ethnic bias, and it may serve to educate students about cognitive biases in general. Clearly, a replication with more data is required. While the current iteration of this game focused on ethnicity, we have yet to analyze the data pertaining to gender. Also, this game could also be adapted in the future to expose other biases. The student designer recently revealed that she had an interest in exposing biases associated with sexual orientation and gender identity. I’m excited to see if she will develop this variation of the game on her own.

Figure 1

Figure 2