Feedback: Boys and Girls Club

Started at 4:20PM
1 7th grader and 3 8th graders
Playtesting round 1 Got distracted with each others questions and answers period
Interview time seems too long- they seemed confused with the extra time
o Thought that they could only ask the one question on the mission card
o Were confused with what else they should ask
o Reverted to random questions such as whats your favorite shoe brand? which had
nothing to do with the prompt
Were mostly silent with two minutes left
Interviewers switch Better back and forth conversation- for the pair that were better friends
A little more awkward and silence for the pair that didnt really know each other
There were some tangents also- but not necessarily a bad thing because they are getting to
know more about each other
Were done with one minute left
Both pairs did not make much use of the notes section- maybe emphasize the importance of
the notes section from the beginning?
Also they did not write the notes until after the interview- I thought we were simultaneously
writing the notes section and interviewing
Writing Period- Undercover work (6 minutes)
One student was not motivated. Did not want to write until the last couple of minutes
Some stopped writing after two or three sentences
Finished writing with 2:10 left to spare
Lack of depth with writing
Finished with one minute to spare
My profile writing (6 minutes)
More engaged in writing about themselves
Lost one student to distractions
Another student finished in 1:30 minutes
All done writing in 3:30 minutes
Ended around 5 PM
Scoring portion
Took a while 10~15 minutes but for four different prompts
2,4,4,6 points
They answered different questions, not the ones on the pieces of paper
Feedback
kinda fun

Interview

section was the fun part

Get to know your partner more
You get more into it as the game progresses
Like the action restrictions
Thinks the word restrictions would be hard
Change about the game- make restrictions harder
o Ex. get up, rub your stomach
o Talk while doing push ups
Classroom environment- felt like it was outside of a classroom
Felt like it was doable for the class
Can add stuff that theyre learning in the class- ask their opinions for it
Good for middle schoolers
Creates empathy
Wanted more time for interview
Like the video/ dont know
Writing time was too much- just wrote down her partners answers, just stated again what
she said
One student took notes while interviewing- others forgot to utilize notes section
Thinks teachers would be interested- most teachers want to make sure you know stuff
One student would pay $5 for it
it was fun - at least one girl said that shed play it outside of class (the boy wouldnt, but
would be happy to play it in class)
Give a minute or two to come up questions before interview session
If they didnt like the student they were paired with- they would not want to do the interview
unless teacher made them
Let them go off topic, let them know more about their classrooms
Asking the questions part- good first day of school part
Writing about your own self threw them off

Mary Ellen: I overviewed the judging panel for scoring, and I definitely think this was the biggest
problem area. Heres what I observed:
-They did not want to read all of the scoring criteria descriptions
-we had 2 girls on the panel. 1 was pretty into it, 1 was totally off the wall (might have
been because we took a pizza break before scoring)
-They were very generous on scoring (ex: Well give her a kind of on vocab b/c she did say I,
the, and movie...note: these were literally the only 3 similar words in the piece)
-I still think it takes too long
-We didnt really have a way to announce criteria to everyone...we should either write how to do
this in the teacher script, or totally change scoring in a new way. But the people not on the panel
didnt really know what they were being judged on/why they got points
-We should probably interview a teacher about this...it might be worth it to ask the teacher if
shed be up for scoring on her own. Alternately, I actually think now that what one of the

playtesters from our in-class playtest suggested (that the kids pass their paper to someone else
who scores them) might work...might be more inclusive for everyone.
-The problem Id see with this is students who just cant focus. For example, the girl on
the judging panel who was off the wall wouldnt have given much thought to the scoring
-The girls only looked for evidence sometimes when comparing the 2 pieces of writing we should emphasize that for every decision you make, you should find evidence in the writing
piece to back it up
-We need to specify the writing skills criterion: they didnt realize that they were only judging
the undercover work piece on this, not the my profile piece on this
Celina, you have the scoring sheets, one of which has my notes for how scoring went on it.
Could you please add anything to this doc that Ive forgotten?
Mary Ellens Notes from document:
- Only one of them was into it, one couldnt focus
- Specify that theyre judging the undercover work for writing skills
- Tell them to talk about why theyre arguing for the point totals
- Vocab giving points if only one or two words match

Other:
When we meet, we can give yall an example of what it was like to accidentally broach a
sensitive topic with a student. I think that our game is structured well & it will work - I learned a
lot about the student, and felt like the student was accepted by the interviewer and by others in
the room who overheard. BUT I felt dumb because I forgot to announce that they dont need to
answer any question that they dont want to - WE NEED TO ADD THIS TO THE TEACHER
DOC ASAP
Another other: Can we write them a thank-you note? They rocked playtesting

User-Testing Refinement
To:
From:
Celina
Date:
Subject:

Jennie Chiu
Lee Williams, Mary Ellen Potter, Grace Hwang, Shelby Ware, Rachel Han,
Abboud
November 17, 2016
Undercover Detectives: User-Testing Refinement

The Undercover Detectives team playtested with middle school students at the Boys and Girls
Club of Charlottesville to gather feedback on the games design from its target population. In this
document, the feedback has been summarized and translated into tangible revisions to be
incorporated into the games final iteration. We noted six main problems which need to be
rectified: 1) how players are scaffolded with respect to the interview process, 2) the description
and utilization of the notes section, 3) the scoring structure and process, 4) the potential for
sensitive topics to arise out of interviews, 5) potential for even more engagement via expanded
restrictions, and 6) too narrow target population parameters. Notably, the issues observed at this
stage of refinement, although important, are of a more subtle nature than those previously faced.
No significant game mechanics must be changed at this point in our design process. Rather, most
of our revisions are focused upon clarification and existing concept expansion.
Interview Supports
Problems
During playtesting, the interviews were difficult for the players to pick up right away. They
experienced confusion over what questions to ask during interviewing, and then were unsure
how to answer the My Profile writing page. Due to this confusion, many players did not use
the full interview or writing time. They began asking random, irrelevant questions which
detracted from the targeted purpose of the mission cards and further complicated the writing and
assessment stages of the game. A couple times, the My Profile writing did not match up with
the Undercover Work writing, so scoring was not possible. Moreover, players did not utilize
the full interview time because they did not have a clear grasp of what questions they were able
to ask.
Solutions
Much of this confusion arose during the interviews and then propagated forward, muddling the
writing stages and then complicating the assessment process. The main issue was a disconnect
between when information was offered during the teachers description of the interview part of
the game - particularly regarding what questions players should ask - and when that information
was most needed. Using the learning principle of information just in time (Gee, 2005), we
reorganized our Teacher Guidebook to instruct teachers to describe the interview process (as
they did before), then model a successful interview (using a variety of questions, noting how
they were above and beyond just the two on the mission card), and then remind the students of
this information right before they began the interviews. We will also include more explicit
instructions regarding to which prompt players are to respond, and these will be placed in the
Teacher Guidebook right before students need that particular information.

In addition to clarifying, reorganizing and reiterating instruction, teachers will also allow
students to take two minutes before the interviews start to brainstorm possible questions to ask.
As a result, students will feel more agency and control in how they approach the interviews.
Notes Section
Problems
Many users did not take advantage of the notes section, partly because it posed a cognitive
overload to them and partly because students were unaware they were allowed to use the notes
section while interviewing. We received feedback to support both of these explanations.
Solutions
Considering the long-term nature of our game and its significant learning-curve on the front end,
we decided not to remove the notes section from the game. The two reasons for not using the
notes section seem to be easily remedied: a) players learn how to gather data in an interview
format more effectively as the game progresses and thus are able to avail themselves of the notes
section down the line; and b) we provide more just in time information for players to recognize
the added benefit of the notes. The notes exist not to add difficulty, but to help students do well.
As such, a full description of the notes section will be added to the Teacher Guidebook to fully
inform students about the notes: they may either use them or not, whichever is more helpful for
them.
Several concepts from the reading support this decision. James Paul Gees notions of information
just in time (previously discussed), agency and customization all provide rationale for allowing
players the option to take notes if they would like (2005). Having options which are
customizable at the individual level yet not mandatory will increase a players sense of agency,
in turn bolstering engagement and identity within the game context. Moreover, adding the notes
component adds a system element into our game. As noted in the Game Design Workshop,
complex behavior patterns can arise from several elements operating in a system. Having the
option to take notes poses a risk to some (they need to spend time listening to the interviewee)
yet also offers an added benefit to others (it can help them remember what the individual said).
As the weeks progress and gameplay deepens, we expect players to try out different strategies,
including note-taking, for potentially rich results which will develop out of the dynamic system
of interrelated game components.
Scoring System
Problems
Overall, the scoring process was too wordy and confusing. Players did not have the attention
span or motivation to read the detailed criteria and then use it to evaluate their peers writing.
Players on the panel did not use evidence from the writing when deciding point totals, which
made the assessment somewhat arbitrary and thus even more disengaging for the rest of the
players. Panelists at times were unsure if they should judge the Undercover Work or the My
Profile work (when they should be evaluating the Undercover Work in comparison to the My
Profile work).
Scoring also took too long due to a lack of attention from the panelists, extended deliberation by

the panelists and the long period of time it took the panelists to read the Undercover Work and
My Profile writing.
Solutions
In order to expedite and simplify the scoring process, the teacher facilitator will read the My
Profile and Undercover Work pieces for the individual being judged aloud to the class and
panel. She will then hand over the document for the five panelists to use as evidence. Panelists
will then be asked three questions which were carefully created to be distilled versions of the
three main criteria we developed for assessment of the writing in our last round of revisions. The
questions will be asked by the teacher, discussed by the panel, then quickly answered with a
yes (2 points), maybe (1 point), or no (0 points). The teacher will provide just in time
information by explicitly mentioning exactly which piece of writing is being judged and he/she
will emphasize using evidence from the readings to back up panel decisions. The panel gets 30
seconds per question to reply, further speeding up the process.
Tracy Fullerton notes that clear goals and feedback are key elements in achieving player flow in
a game (Game Design Workshop, 2008). Players must know what needs to be done in order to
reach the desirable level of challenge and engagement which constitutes flow. Initially, we
handed student panelists a rubric with criteria on it, and asked them to allocate points to the
player based on what they read. This functionally isolated the rest of the players: engagement
dropped and they were handed scores back which seemed arbitrary. Our solution of presenting
clear, easily answered questions to the panelists will involve the rest of the class in the scoring
process while providing them with valuable feedback regarding what they did well, or what they
could improve upon.
Sensitive Topics
Problems
During playtesting, one question came up where a student asked his partner what the best thing
to happen to him that week was. Although no sensitive information was mentioned, we
recognized the potential for uncomfortable topics to be raised. This possibility poses some
ethical issues and is inextricably linked to the potential for differences to be highlighted across
students, perhaps leading to more divide as opposed to one of the games goals of increasing
social cohesion.
Solutions
One key revision we plan to make is to instruct teachers on two accounts: 1) to explicitly notify
students (repeatedly) they do not have to answer any question they feel uncomfortable answering
and 2) to use their discretion when pairing players. In Designing Games for Learning,
educational game design is conceptualized as being inherently teacher-reliant. Classrooms are so
diverse that there is no true one size fits all game which will be successful across the board. As
a result, customization and discretion options must be available for all teachers to implement
based upon their unique knowledge of their classroom dynamics. Specifically, the design pack
mentions the importance of ethos, or the culture and quality of the classroom social environment.
As external game designers, we intend to build a game which can be fit into many different
environments with the teachers discretion providing the final essential component. As a result,

we will clearly convey to teachers the possibility of certain topics to come up, and we will advise
them to take the steps that only they know to be necessary in order to maintain a
well-functioning classroom. Furthermore, we will modify any topics we have now which could
potentially spur undesirable topics from being brought up among students.
Using our Guidebook, teachers will empower students to approach the game as something they
create and shape on their own with their unique personalities. Implicit in this approach is the
ability to deconstruct or reject questions/topics which may breach their sense of comfort.
Drawing on principles of agency and identity, this will not only lead to safer gameplay, but will
also increase engagement and enjoyment by the students (Gee, 2005).
Expanded Restrictions
Problems
As noted in previous rounds of feedback, the most enjoyable aspects of the game were the
hindrances imposed upon interviewers. Players from our target audience believed the game could
be more engaging if restrictions were built out even more than we already have expanded them.
Solutions
Much like last round, we will brainstorm more creative restrictions to include on the mission
cards. Some of these will be more challenging - to ensure flow is maintained - while others will
be more silly and expressive - to ensure we provide space for free play within our games more
rigid structure (Game Design Workshop, 2008). As noted in the Game Design Workshop, we
have found that when players are fully concentrated on following the restrictions of the game
while still pursuing the mission on their mission card, there is a loss of self-consciousness
(2008). This state of mind is desirable not only due to its association with flow and engagement,
but because it will reduce the likelihood of ethical issues arising from self-centered displays of
divisiveness.
Target Population
Problems
Although we came into this round of playtesting with a desired target population of just sixth
graders, we were only able to playtest with seventh and eighth graders. We observed that the
game still posed a significant amount of challenge to this population, which made us rethink the
appropriateness of our chosen target population.
Solutions
We decided to broaden our target population to include all middle school students, from sixth
grade through eighth grade. We decided to maintain our inclusion of sixth graders because our
writing prompts and learning objectives align closely with Virginia Standards of Learning for
sixth graders under the Communications and Writing Objectives subject areas (English
Standards of Learning Curriculum Framework, Board of Education, Commonwealth of Virginia,
2010). Thus, although the game may be more difficult for this population, it presents the greatest
opportunity for transferable learning into realistic applications (such as SOL tests). That being
said, seventh and eighth graders also enjoyed and felt challenged by the game, so we believe it

could pose a benefit to these groups as well. Given the level of customization we are offering
teachers, many diverse audiences could play this game with successful results.
Conclusion
Overall, playtesting with our target users was very encouraging. Despite some confusion around
the interview process, players generally enjoyed the game and declared that they would play it
again (if only in the school context). Moreover, several of the playtesters noted that as they
progressed through the game, they began to understand how it worked at a deeper level. We
believe that this is reflective of the fact that although our game poses a somewhat steep initial
learning curve, after a few rounds of playtesting the users will understand the core mechanics,
and from there will greatly advance in their learning of empathy, writing, and social cohesion.