Decision Maker, Part 1

Now that I’ve explained our lab’s method of game development and rapid prototyping, I’m going to briefly explain the rationale for some of the games in development and describe the progress we’ve made in the past two weeks. Keep in mind that we only have six weeks to design the games, collect data, and present our results at a local conference. It’s a sprint.

Decision Maker is designed to teach students about decision-making and to help them make better decisions under uncertain conditions. Decision-making was thoroughly studied by Danniel Kahneman and Amos Tversky, who are considered the fathers of behavioral economics. Kahneman received a Nobel Prize for their work in 2002 (after the passing of Tversky in 1996). Their model of decision-making was called Prospect Theory and, in the classic paradigm, subjects must choose between a sure bet (e.g., $100) and the prospect of winning a lottery (e.g., 50% chance of winning $214). The utility of each prospect is defined as the probability multiplied by the value. In our example, the prospect would be the wisest choice because the overall utility of the bet, $107, is higher than the utility of the sure bet, which is $100.  What Kahneman and Tversky discovered, however, was that people often behaved irrationally when presented with bets that had extreme probabilities or values. The astronomical prize money and the misperception of extremely low probabilities explain why people would pay $5 for a state Lotto ticket when the odds of winning are overwhelmingly against the player. While Prospect Theory has been around for years, there have been few efforts to shape behavior given this knowledge. Merely telling people how to make decisions is not enough to alter their behavior. Consequently, Decision Maker is designed to train students to improve their decision-making skills for extreme probabilities and values.

Designing games that are fun, educational, and able to collect data for scientific purposes is challenging. After the educational objectives of our games are established, we develop experimental protocols that will allow us to assess the behavior we wish to shape. Then, we develop game mechanics that compliment those experimental protocols. Our lab uses Tracy’s Fullerton’s Game Design Workshop and Jessie Schell’s Game Design: A Book of Lenses to ensure we address the most critical elements of game design. While it’s not necessary to adhere to this particular order, I recommend starting with a good scientific experiment. However, it is sometimes useful to develop the experiment and game mechanics in separate “sandboxes.” I often ask students to simultaneously design an experiment, a game mechanic, and a fun toy/interface to play with. Combining the results of independent endeavors often produces interesting and unexpected surprises.

Fortunately for Decision Maker, there are several experimental paradigms from behavioral economics to consider. We adopted a method of adjustment paradigm where players indicate the sure bet they would accept in lieu of a given prospect. Players will be presented with positive and negative prospects, and prospects will vary widely in probability and value. The set of probabilities and values were randomly generated to make the mental computation of utility difficult (e.g., 0.2% chance of winning $10,147). Using the staircase method typically employed in psychophysical experiments, the difficulty of the decisions will increase when players make more correct decisions, and the difficulty will decrease if they make errors. The staircase allows us to find the threshold where decisions become less reliable, and it keeps players in a state of flow (where the task is optimally challenging without being too frustrating or too boring). While multiple ascending and descending staircases can be interleaved, we decided to go with a simple descending staircase with consistent step sizes between trials. Our prediction is that subjects who play our game will preform better on a post-test of decision-making relative to control subjects who received equivalent practice in decision making without using games.

After the experiment was designed, we developed the game mechanics. Some of these mechanics are more clearly defined in Tracy’s book, but I’ll briefly define them here. Objectives describe what the player is trying to achieve in the long run. They can describe intermediate goals or the ultimate win/fail states of the game. Resources are items in the game that you are either acquire or get rid of to achieve your objective (e.g., the pieces in Chess or the money in Monopoly). Feedback mechanisms are implemented in games to inform the player about their performance (e.g., point totals or badges). Reward/Punishment Contingencies describe how and at what rate the game will react to a player’s decisions. Different contingencies can have dramatically different effects on behavior. For example, more work is typically elicited from pigeons if rewards are intermittent rather than consistent. With effective contingencies and feedback mechanisms, behavior can be shaped quickly to help the player achieve their objective. Of course, games don’t always have to be friendly. Unreliable feedback mechanisms can be used to achieve a different effect. Flow has been defined in detail elsewhere. Students should focus intensely on how to use all the other mechanics and standard psychophysical procedures to elicit a state of flow in the players. The staircase method should be the starting point when considering flow. Finally, boundaries are rules that prevent players from acting in a particular way. While limiting player behavior might appear to be a fun killer, boundaries often have the opposite effect. For example, soccer is really only fun because players are not allowed to use their hands. This game mechanic is best employed when you are not making progress on a design. If a student is functionally fixed on a particular design that is not working, introduce a boundary to change the designer’s frame of reference.

The core game mechanic for Decision Maker rests on how prospects are presented to the player and how the player evaluates those prospects. Prospects will be presented on playing cards along with scenarios related to student life. Players must write down the sure bet they would accept in lieu of the prospect. The correct utility of the prospect will appear on the back of the card. Players are rewarded for correct decisions by receiving a card and they are punished by not receiving a card. The objective of the game is to accrue more cards than your opponent before the deck of cards is exhausted. After a choice is made, the player spins the dial of a spinner and watches the gamble play out in real time. It’s important to note that, just like the state Lotto, players can win on rare occasions even if the choice to gamble is incorrect. Immediate feedback is implicit when the player flips the card to see the correct answer, but feedback is also available by comparing how many cards each player has collected relative to the opponents. We didn’t really feel the need to impose boundaries in this game because the behavior is fairly controlled and we didn’t want to further limit our players. Because we are developing this game as a board game, implementing a psychophysical staircase procedure was a little trickier. We decided to introduce levels into the game. Each level will have it’s own spinner and deck of cards. During early levels, the player will evaluate relatively easy prospects (i.e., within the range of reliable decision making). If the player successfully answers a certain number of questions, they are advanced to the next level where more difficult decisions have to be made. If a player doesn’t accrue a significant number of cards at the end of a level, they must go back and replay that level. Thus, student must improve on their evaluation of difficult prospects in order to win the game. Spinners for the early levels will be marked to indicate probabilities from 1 to 100%. Spins for the advanced levels will represent extreme probabilities by requiring the player to make several spins in a row within a target zone (e.g., the bet for a 0.25% prospect pays off if the player gets the needle to land between 0 and 5 twice in a row). The physical task of spinning the spinner several times for low probabilities will hopefully reinforce the notion that it’s unwise to bet on rare outcomes! Players will keep track of their own score. Players must maximize their winnings and finish with the most number of cards to win the game. If a player finishes the game without having the largest total on their scorecard, all players purge their cards and the final level will be repeated until there is a winner.

To make the game more fun, we plan to add a number of physical tasks that also must be completed before advancing to the next level (e.g., stack all the blocks that come with the game Jenga). Players will have a limited time to complete mini-games, physical feats, or puzzles to pass to the next level. The addition of these puzzles will add variety to the game and allow players to take a break from performing mental calculations. Additionally, some game cards will introduce “windfalls” or “calamities” into the mix. Windfalls might spontaneously grant the player an extra turn, extra cards, or allow them to circumvent the physical task. Calamities might require the player to loose a turn, give up cards, or move back a level. To maintain balance between competitors, players with fewer cards are more susceptible to windfalls and players with more cards are more susceptible to calamities.

Our plan is to have a working prototype of this game in the next week and play test all our games so that we can collect data the following week. Subsequent posts will describe our other games and the progress made in this game.

York College Summer Research Program

Welcome to the first in a series of posts related to rapid prototyping and game development with students. York College hosts the CUNY Summer Undergraduate Research Program (C-SURP) for high school students and undergraduates. The program is competitive, pays a stipend, and culminates in paper and poster presentations that can be applied toward college admissions or scholarship applications. I currently have nine students in my lab this summer, six of which are participating in C-SURP. All of the students are developing games of their own, but each student works on every game in the lab. The design process is collaborative, but students serve as project manger for their own game. To support this collaborative environment, I have adopted the principles described in the Valve Handbook for New Employees, which encourages collaboration without authoritative leadership. My goal is to create a sandbox where students can develop ideas without fear of criticism. I view my role as a facilitator who only intervenes to ask critical questions if a student is veering toward hazardous territory. If things are going well, I shouldn’t be talking at all.

The primary research objective of the lab is to study the efficacy of game-based learning in various disciplines. We incorporate principles from cognitive psychology, neuroscience, education, and game design to create effective learning systems. To that end, each student’s project is a study of learning in itself. Students are both the creators of a learning experience and students of the design process. My goal is to engage students in the subject matter of their choosing by involving them in design. By designing games, they must master the material related to their subject and then create a tool to teach those lessons to other students.

There is clearly no single best practice for educational game design. Even so, game designers have agreed that a certain number of elements are commonly found in successful games, and successful games are more likely to result from certain design processes. I’ve found that it’s critical to have frequent (but not necessarily daily) scheduled meetings and then allow students to break into groups as they see fit. Weekly meetings are not enough because students typically forget what they were supposed to accomplish by the time they get around to working. Cognitive psychology informs us that spaced learning is most effective, and professionals in the creative arts will attest that having a regular work regimen sets the stage for those “Eureka” moments in the creative process. Fortunately, for educators, this process is easy to implement in a class that meets two or three times a week.

During our group meetings, I ask students to review any progress they made or problems they encountered since the previous session. This practice is common to lab meetings. However, there is one critical difference. When solutions to problems are explored, it is important not to criticize any proposed solutions. Every member must be able to freely express any wacky idea that comes to mind if the appropriate solution is going to find it’s way to the surface. Reserve critical review of the game until after it has reached a stage where it can be prototyped and tested. As a facilitator, you run the risk of discouraging student participation if you start dropping logic bombs on brainstorming sessions. If you’re looking for a fun way to get this idea across to your students, play a round of “yes-and,” an improv game where students take turns building a story using the phrase “Yes, and” while avoiding the phrase “Yes, but.” It’s harder than it seems.

As a facilitator your primary job in the brainstorming sessions is to make sure the discussion is focused on the problem while remaining impartial. Having a dry-erase board in the room is critical for keeping track of all the ideas that emerge during the session and, more importantly, for making associative links between seemingly unrelated ideas. Student participation will run the gamut from incredibly-insightful-but-shy to extremely-loquacious-but-unfocused. The dry-erase board is a wonderful tool for focusing the unfocused. Ideas can be rapidly sketched as they come to light without the fear of running off the rails. Asking biographical questions is a great way of drawing out the shy students. Even the most reluctant students I’ve had in the lab will respond to personal questions about their interests, opinions, or experiences. You can use the answers to those questions to start a dialog about how to apply those attitudes toward the design of their game. At the end of each session, each student agrees on what needs to be accomplished before the next meeting. I’ve found that taking a picture of the dry-erase board is a quick and dirty means of backing up the day’s efforts without having to take additional notes.

There are a couple of challenges you will undoubtedly face when working with students in game development. First, students will come to you with an idea, together you will work to refine that idea into something tenable, and then they will come back to you the next day with a completely different topic. Many students are not academically resilient. When they encounter a problem that results in an appreciable amount of effort, it is easier for them to choose a new problem to solve. This is not a failure! This is a fine example of lateral thinking. Normally, I would encourage students to explore all the possibilities. However, most projects are under a time constraint so, you must help them not to exceed their “time budget.” Students switch from topic to topic because they are blissfully unaware of the vast amount of information they must absorb to complete the project (aren’t we all?). In my experience, when left alone, students who can’t settle on an idea will switch topics until they run out of time (sometimes this lasts an entire semester). As a facilitator, you have to intervene. I recommend introducing game elements to keep them on track. Give them a task list so they can monitor their own progress. Introduce a boundary that will hasten their work and make it more game-like. In game design, boundaries are used to limit a particular behavior, but it’s the act of limiting that behavior that makes the game fun (e.g., not using your hands in soccer). For example, to keep my students from spending too much time on an idea, I might give them a time limit for each stage of the process. As an alternative, you can also introduce friendly competition between students in the form of a race, or provide rewards for completing stages on time.

Students will also get frustrated at the difficulty of the readings and the amount of time it takes to sift through a seemingly endless pile of literature. Do you remember what it was like to read your first peer-reviewed journal as an undergraduate? Imagine being given a stack of readings like this as a high school student! As a facilitator, one of your other important jobs is to cull the reading list. However, it’s critical that you let the students discover information on their own. No matter how well we curate and scaffold the reading list, students do not like being handed a pile of papers. It makes them feel like they are being forced to do work. Let them start with searches on the Internet, even if it leads to dubious sources. When they find sources they like, augment those readings with a highly refined list of your own. Explain outright that you are giving them this list to save them time. Now that they have done some work on their own, they will appreciate the list. I typically prescribe a chapter from an undergraduate text, a good review article, and less than three peer-reviewed articles that are directly related to the subject matter.

Finally, the students might not understand or identify with the logic of game-based education. Fortunately, this is an easy lesson to teach. Start by having students talk about the games they’ve played, what games they like, what games they didn’t like, and what they may have learned from the games. You’ll probably have to step in and stop the discussion! Then, have them play some state-of-the art educational games at Gamesforchange.org. Explain that these games may or may not be the best examples of how to teach a lesson. They should have fun playing the games, but also encourage them to review the games with a critical eye for improvements. Follow the game playing session with a crash course in game design. Explain the critical elements like objectives, win-loose states, resources, boundaries, and reward-punishment contingencies. I try to explain these elements from the perspective of behavioral psychology because, ultimately, all behavior in games can be explained using well-tested models of behavior from psychology. Try to keep in mind that we are using games as tools to shape behavior. If we do our job, our games will produce measureable changes in behavior that can be quantified. When you have finished, introduce a few readings on game design, particularly Tracy Fullerton’s Game Design Workshop or Jessie Schell’s The Art of Game Design: A Book of Lenses.

Now that I’ve introduced my method of introducing game-based learning to students, I’ll follow up with a few posts on the games we’re working on and the day-to-day progress on those games as they develop.

 

York College Transformative Games Initiative

I’m very happy to announce that there is plenty of interest and support among the faculty, students, and administration to officially launch the York College Transformative Games Initiative. The objective of this committee will be to provide information about game-based learning, organize local efforts to incorporate games into the classroom, and analyze the results of these efforts in order to make improvements. The committee will operate under the direction of Xin Bai from Education, Michael Smith from Performing and Fine Arts, and myself. While our roles may be overlapping, I will be primarily handling the science underlying game design, issues related to in-game assessment of student performance, assessment of game efficacy, and administration of the committee. Xin Bai will oversee issues related to educational technology. And Michael Smith will manage issues related content, asset, and media creation by students and faculty. If you want to be included as a member of this committee or if you want to learn more about how games can be easily incorporated into your classroom, please send me an email. In a future post, I will provide the rationale for game-based learning and provide a list of peer-reviewed articles and books that document the science behind game-based learning. Announcements of the first committee meeting will take place here and via York College e-mail.  In the meantime, I’m happy to answer any questions you might have here.

Learning by design