feedback

Using Playtester Feedback, part 3

Playtesters are the sole authority of their own experience.  There is not a single person in existence better qualified to comment on the player’s experience than the player herself.  We designers need our playtesters to share their experiences with us.  It is critical to gather as much of these as possible.

It would be ideal if our playtesters could state exactly what issue they see in our design.  In that case, they could point us directly where we need to put our eyes.  Unfortunately, that will seldom be the case.  Be mindful that playtesters are not generally experienced designers.  They know what they like (or don’t) but they may not be able to fully articulate why they like it.  This is not to say in any way that their feedback isn’t useful but instead to emphasize the role of playtest data as an indicator.  This is why it’s so very important to ask follow-up questions.

I recommend that your follow-up questions should be structured along the lines of “What is the issue you’re seeing” or “how would your change address that issue” or “let me say this back to see if I’m fully understanding you”  and “What if the game (had/didn’t have) _____. Do you think that would address the issue?”

 

Sarah Aronson Uses Her Feedback
Writer Sarah Aronson related her relationship with feedback in a 2012 post on Through the Tollbooth.  She says that the feedback she receives points her to what is working, to what is not working, to the seeds of good ideas which need watering and a bit of TLC.  Aronson used her next post to share some of her key questions: where are the hooks that pull you in?  Is the plot balanced? What were the most memorable moments?  How was the pacing?  How was the length?

 

Daniel Solis Listened Beyond the Comments

Designer buddy Gil Hova is one of the folks I consulted while working out my thoughts for this series.  He referenced a blog post by Daniel Solis in which Daniel laid out his process for evaluating playtester feedback.  Daniel says near the end of the article “I didn’t end up using any of these new suggestions directly, but instead tried to suss out what those suggestions were trying to fix. They were trying to make early choices matter.” Again, playtester feedback may not be the solution but it is a valuable indicator.

 

Gil Hova Listened Beyond the Comments

designer_headshotGil also shared his experience applying Daniel’s approach.  Gil brought his new game Primetime (which is excellent by the way and you should plan to buy a copy as soon as it drops) to a gamer con and showed it around.  The first two groups each told him that it was boring to have so much money but little to do with it.  “I should be able to spend money to X,” they said where X was any number of player options.

Gil stepped back and asked what underlying problem might lead to these comments.  His conclusion: it was not a problem of too few options for the money, it was a problem of too much money!  He tightened the game economy up drastically and the game began to sing.

 

An Enhancement For Rolling Freight

Rolling Freight came with us to BGG.Con a few years ago, shortly before it went to the printer.  This would be our last chance to get outside feedback, to look for loopholes in the rules and the like.  One of the players mentioned that improvements lost value over the course of the game and that they should therefore get cheaper as the game proceeds. Through three years of development, this idea had never occurred to any of us–what an awesome idea!

That idea had to be left out of the core set.  Rolling Freight was already a bit rules-heavy and I was concerned that the extra rule would have been the straw that broke the camel’s proverbial.  But I held on to it.  And I put it in Expansion #1: Great Britain/India.  And it will appear in Expansion #2: Austro-Hungarian Empire/Mexico as well.  A huge breakthrough. All for the low low price of listening to a playtester.

 

The last three columns have been devoted to my thoughts on the relationship between designers, playtesters, and their feedback.  This topic–like core engagement–is an essential tool in the designer’s toolbox.  It is one to which we will often return.  The phenomenal Julia Cameron who we can imagine knows a thing or two about the creation as she has been an author, artist, poet, playwright, novelist, filmmaker, composer, and journalist remarked in her book The Artist’s Way that

“All too often too often we try to push, pull, outline and control our ideas instead of letting them grow organically. The creative process is a process of surrender, not control. Mystery is at the heart of creativity. That, and surprise.”

Researching this series led me to evolve my position as each new source made it progressively clearer that this is a broad concept with no simple answers.  Reading your comments led me to adjust my position as each new response made it progressively clearer that we all will have our own take on the issue.  Writing this series has matured me as a designer.  I hope that in reading it, you have too.

 

If you’re near the west coast this weekend, I have the honor of attending Protospiel San Jose.  Grab your prototype and drop by. I’d love to see you there! 🙂

 

How do you playtest?  What sorts of feedback are you seeking?  Do you give directed instructions or do you encourage testers to explore your game?  How do you record your data?  Share with your fellow readers in the comments below.  And if you’re enjoying what you’re reading, create and account with WordPress and follow this blog.  If you keep reading, I’ll keep writing.

 

Advertisement

A Response to “Using Playtester Feedback”

Reader Yves Tourigny Posted a response to part 2 of this series yesterday.  It is so well argued that I asked for his blessing to quote it here.

Yves position and mine don’t differ by much but they do differ in some interesting ways.  Whether you are still seeking your own voice as a designer or if you have your voice and wish to strengthen it, there are some solid ideas to contemplate here.

 

“I remain unconvinced. Your reply to argument 4 leaves me scratching my head. Who are these analytic geniuses with whom you playtest your games? A single play of any game may leave you with the impression that you have a deep understanding of it, but if that’s the case, I’m afraid that I’ve already filed your feedback under “crank”. The inner workings of a game may in theory be laid bare, but the complexity that emerges when even simple systems start to interact should give pause to anyone who thinks they have an easy fix.

“This is especially true if the playtesters have not read the rules, but have had the game explained to them. Or when the playtest is a partial one, where the entire game may not have been experienced.

“Repeated plays, or a very simple game, may very well give playtesters a complete, holistic picture of the game. This would simply make their cogent analysis of any problems far MORE useful than any solutions they might have for those problems.

“There are times, I believe, when playtester feedback is almost completely irrelevant. Case in point: I playtest many of my games with my two sons. One is 17, and has Aspergers, and the other is 8, and has a very strong desire to express his opinion about everything under the sun. I’ll ask for their feedback, specifically for things they liked or didn’t like. If the 8-year old starts offering solutions, or suggestions (e.g. “Maybe you could add (something that would add nothing)?”), I politely steer the conversation back to useful territory.

“This is in fact what I do with any playtester. I’m not concerned about hurting their feelings if I don’t use their suggestions, because I question them to get to the root concern. Why do they feel this suggestion is required? What problem is it that they are trying to solve?

“Just to be clear, I’m not saying I never want to hear solutions or suggestions, only that without a clear statement of the problem they are next to useless.”

 

Using Playtester Feedback, Part 2

“Learning and innovation go hand in hand. The arrogance of success is to think that what you did yesterday will be sufficient for tomorrow.”         –William Pollard

 

Part one of this series generated intense response from both camps. Some lauded my comments. Others decried them. So long as we’re debating, we’re thinking. As long as we’re thinking, we’re growing.  So long as we’re growing, we can avoid the arrogance of success.  Keep those comments coming folks!

A designer I know asks of his playtesters, “give me your problems, not your solutions.”  In support of this position, he offered five arguments.  My collaborative philosophy finds these arguments fundamentally flawed.

 

Argument 3. Playtesters tend to get attached to their suggestion and resentful if that suggestion isn’t used in the final design.

This is a valid concern.  It is also a concern that can be alleviated with appropriate courtesy.  Thank each person for their input.  Thank them for helping you make the best game possible.  State that you’ll be looking at all of their suggestions, working to integrate them and refine them in the best way possible.

This response also keeps the peace when two playtesters come up with conflicting suggestions during the same test.  Acknowledge that you won’t be able to integrate both suggestions and that everyone is working for the same goal–to make the game as good as possible.

communication_feedbackA similar excellent solution to this issue comes from Imaginatik in their article Making Feedback More Positive.  “Instead of sending individual [acceptance or rejection] messages, send a group response to all interested parties, praising them for their contributions as a collective. Indicate that each individual response was crucial in helping the review team reach a consensus. Also, mention that all ideas will be moved to the Idea Warehouse for possible future consideration. This way, each person feels good about his or her contribution as opposed to getting negative feedback.”

In other words, thank all playtesters for their contributions.  Then express that every perspective adds to your picture of the game as a whole and helps you to make the best game possible.  In this way, unused ideas have been acknowledged and the tester knows THAT YOU KNOW that she made a meaningful contribution.

This is all most playtesters need to hear.  Playtesters want to know that you appreciate them.  They want your acknowledgement and respect.  Give it to them.

 

Argument 4. Playtesters cannot see the inner workings of the design.  They are therefore ill-equipped to make informed suggestions.

This argument might be valid in the world of computer game design but it is not a reasonable statement about tabletop games.  The opposite is true.  One of the principal reasons many CG design schools begin by making students create tabletop board games is precisely because tabletop games DO reveal their inner workings.

Imagine you’re playing a dungeon crawl-type game like Orc Vengeance on your smart device.  you know that your hero’s sword has power 38 and your hero has strength 53 but the damage it deals seems to vary from about 60 to 130.  Do you know why that variation exists?  Is the enemy’s armor a factor?  Its agility?  Terrain?  Is your target resistant to your attack?  Or vulnerable to it?  Does the game use a randomizer?  If so, how?  Through trial and error you will likely figure out many of these details.  This will take time and even then you will probably not figure out all of them.  It is tricky to study CGs from the outside because the game engine is obscured by the interface.

Now imagine you’re playing a dungeon crawl board game like Descent.  The rules are there to be seen, to be analyzed, to be assessed.  You can look at the die faces and estimate your odds.  You know what modifiers exist and when they matter.  The inner workings of traditional tabletop games are quite close to the surface.
Argument 5.   Many players don’t want to raise a problem without having a solution as well. Asking players to express only the issues frees them to raise issues to which they see no solution.

This is certainly the most well-meaning among the arguments–well-meaning but implausible.

We need playtesters to be candid. We need playtesters to express every opinion that occurs to them in whatever form that opinion should take.  Once you’ve got that feedback, then you can decide what best to do with it.  Fail to get the feedback and you’ve got nothing to work with.

Many people have been trained by their experiences in the corporate world not to raise issues without having a solution to offer as well.  We definitely need playtesters to disregard self censorship.  However, instructing them to ‘offer issues, not solutions’ constitutes a direct order them to self-censor. It is not plausible that such directed censorship will free them from internalized censorship.

Instead, encourage openness. Practice saying phrases like “I’ll look at that,” and “how do you feel that would improve the game,” and “how do the rest of you feel about that.” Expressions like these make it safe for them to offer every concern and suggestion.  You will quickly see a change in their candor and their expressiveness.  I guarantee it.

 

Argument 6.   If I incorporate too much playtester feedback, it won’t be my game design anymore.

This was not one of the arguments he put forward but in developing this article, a few other folks did raise the question.

Fashion designer Tom Ford was posed with exactly this quandary in his 2014 interview with Kinvara Balfour.  He has a number of designers working under him and their creations bear his name.  His reply mirrors that which creative leaders have said across the centuries, that ‘while you do listen to everyone’s feedback, it is you who makes the final decision and this is what entitles you to put your name on it.’

While others may have given you a huge number of ideas, it is you who decided what to incorporate and how to incorporate it.  It is this which makes you the craftsman, not the originality of your ideas.  Ask any designer, artist or craftsman and she will tell you that ideas are cheap and plentiful.  Achievement is derived from refining and editing those ideas.

There may be several arguments for “give me your problems, not your solutions”  but I found them to be fundamentally flawed.  Today, we debunked the remaining arguments against soliciting playtester feedback.  That attitude is dismissive, arrogant, unfortunate and ultimately toxic.

On Friday, we return to look at positive ways to playtest and ways to best use player feedback.  See you in three days!

And if you’re near the west coast this weekend, I’ll have the honor of attending Protospiel San Jose.  Grab your prototype and drop by. I’d love to see you there! 🙂

How do you use playtester feedback?  What role would you give your playtesters?  Do you give them any directed instructions or do you encourage them to explore your game?  Or are you a laissez faire tester? Share with your fellow readers in the comments below.  And if you’re enjoying what you’re reading, create and account with WordPress and follow this blog.  If you keep reading, I’ll keep writing.

Using Playtester Feedback, Part 1

A designer buddy quoted another designer to me a few weeks ago.  “Give me your problems, not your solutions.”

This confused me.

So I consulted the designer in question.  Yup.  “Give me problems, not solutions.” That’s a direct quote.

I asked for further clarification.  Wouldn’t you?

He obliged my curiosity and related the reasons which led him to this conclusion.  First, he expressed that the suggestions playtesters give are usually bad.  Second, that while the playtester is relating the suggestion, he was generally running other–presumably better–variants in his head.  Third, that playtesters tended to get attached to their suggestion and resentful if that suggestion isn’t used in the final design.  Fourth, he stated that the playtesters could not see the inner workings of the design and were therefore ill-equipped to make informed suggestions.  Fifth, that many players don’t want to raise a problem without having a solution to offer as well so asking players to only express issues freed them to raise issues to which they see no solution.

A friend of his–another designer–stepped up and agreed with these assertions before they left for dinner, leaving me to reflect on their point of view.

feedbackI went on to discuss this position with other designers.  I have sought the opinion of creative people whom I respect.  I consulted folks that can be trusted to give you an honest opinion, who are candid to the point of brutality.  These conversations led to a number of useful perspectives.  Considering each enabled me to codify my own position.

I am only one person. I won’t pretend to be an expert on any designer’s method but my own.  If an adage like “give me problems, not solutions” fits your design philosophy and it works for you then by all means continue.  But I cannot imagine working successfully with such an attitude because it seems to me that the arguments underlying this approach are fundamentally flawed.

Argument 1.  Playtester suggestions are usually bad.

A friend in college was terrified of asking women out because they might say no.  My response was and is that “you’re right.  That interesting woman over there might say no.  But if you don’t ask her, you’re saying no for her.  You’re denying her the chance to say yes.”

Do you feel that the suggestions playtesters give are usually bad?  Let’s assume you’re right.  The idea that tester wants to give might be bad.  But if you tell him not to express it, you’re denying him the chance to offer a great suggestion.”

“Usually useless” = “sometimes useful.”  So listen to every idea that comes your way and keep looking for the good ones.

Who cares if 99% of the suggestions playtesters offer are bad?  Be assured that 99% of my ideas are bad.  Einstein said the same thing about his ideas.  That’s exactly why we need as many ideas as we can get.  With a 1% success rate, we had better be looking everyplace we can for those diamond ideas.

Argument 2. While the playtester is relating the suggestion, the designer could be contemplating other–presumably better–variants in his head.

Playtesters are the sole authority of their own experience.  There is not a single person in existence better qualified to comment on the player’s experience than the player herself.  We designers seek–and sometimes bribe or beg–her to share her experiences with us.  It is critical to gather as much of these as possible.

You certainly need to hang on to the ideas in your head.  It is equally important to hang on to the ideas coming from your testers.  You are going to have a hard time processing playtester feedback while your brain is off creating other variants.  And people can tell when you’ve checked out.  If you stop listening while they’re offering feedback, your testers will rightly choose not to return to your table.  Playtesters choose how to expend their energy.  They’ll choose the designer with the courtesy to respect their feedback.

It is for this reason that I always have my notebook open and ready to record.  I’m set to jot down my thoughts and feelings while still recording those of the testers.  “It felt like _____,” and “why would I ever_____,” and “this would be better if ______” are all data you need to capture.  Sort through them later. Remember that 99% of them are probably bad–both you ideas and theirs.  That’s why you need them all. Every single one of them.

 

Today, we debunked the first two arguments against soliciting playtester feedback.  On Tuesday, we return to examine the three which remain along with a sixth one which arose during my research.

How do you use playtester feedback?  What role would you give your playtesters?  Do you give them any directed instructions or do you encourage them to explore your game?  Or are you a laissez faire tester? Share with your fellow readers in the comments below.  And if you’re enjoying what you’re reading, create and account with WordPress and follow this blog.  If you keep reading, I’ll keep writing.