Using Playtester Feedback, Part 2

“Learning and innovation go hand in hand. The arrogance of success is to think that what you did yesterday will be sufficient for tomorrow.”         –William Pollard

 

Part one of this series generated intense response from both camps. Some lauded my comments. Others decried them. So long as we’re debating, we’re thinking. As long as we’re thinking, we’re growing.  So long as we’re growing, we can avoid the arrogance of success.  Keep those comments coming folks!

A designer I know asks of his playtesters, “give me your problems, not your solutions.”  In support of this position, he offered five arguments.  My collaborative philosophy finds these arguments fundamentally flawed.

 

Argument 3. Playtesters tend to get attached to their suggestion and resentful if that suggestion isn’t used in the final design.

This is a valid concern.  It is also a concern that can be alleviated with appropriate courtesy.  Thank each person for their input.  Thank them for helping you make the best game possible.  State that you’ll be looking at all of their suggestions, working to integrate them and refine them in the best way possible.

This response also keeps the peace when two playtesters come up with conflicting suggestions during the same test.  Acknowledge that you won’t be able to integrate both suggestions and that everyone is working for the same goal–to make the game as good as possible.

communication_feedbackA similar excellent solution to this issue comes from Imaginatik in their article Making Feedback More Positive.  “Instead of sending individual [acceptance or rejection] messages, send a group response to all interested parties, praising them for their contributions as a collective. Indicate that each individual response was crucial in helping the review team reach a consensus. Also, mention that all ideas will be moved to the Idea Warehouse for possible future consideration. This way, each person feels good about his or her contribution as opposed to getting negative feedback.”

In other words, thank all playtesters for their contributions.  Then express that every perspective adds to your picture of the game as a whole and helps you to make the best game possible.  In this way, unused ideas have been acknowledged and the tester knows THAT YOU KNOW that she made a meaningful contribution.

This is all most playtesters need to hear.  Playtesters want to know that you appreciate them.  They want your acknowledgement and respect.  Give it to them.

 

Argument 4. Playtesters cannot see the inner workings of the design.  They are therefore ill-equipped to make informed suggestions.

This argument might be valid in the world of computer game design but it is not a reasonable statement about tabletop games.  The opposite is true.  One of the principal reasons many CG design schools begin by making students create tabletop board games is precisely because tabletop games DO reveal their inner workings.

Imagine you’re playing a dungeon crawl-type game like Orc Vengeance on your smart device.  you know that your hero’s sword has power 38 and your hero has strength 53 but the damage it deals seems to vary from about 60 to 130.  Do you know why that variation exists?  Is the enemy’s armor a factor?  Its agility?  Terrain?  Is your target resistant to your attack?  Or vulnerable to it?  Does the game use a randomizer?  If so, how?  Through trial and error you will likely figure out many of these details.  This will take time and even then you will probably not figure out all of them.  It is tricky to study CGs from the outside because the game engine is obscured by the interface.

Now imagine you’re playing a dungeon crawl board game like Descent.  The rules are there to be seen, to be analyzed, to be assessed.  You can look at the die faces and estimate your odds.  You know what modifiers exist and when they matter.  The inner workings of traditional tabletop games are quite close to the surface.
Argument 5.   Many players don’t want to raise a problem without having a solution as well. Asking players to express only the issues frees them to raise issues to which they see no solution.

This is certainly the most well-meaning among the arguments–well-meaning but implausible.

We need playtesters to be candid. We need playtesters to express every opinion that occurs to them in whatever form that opinion should take.  Once you’ve got that feedback, then you can decide what best to do with it.  Fail to get the feedback and you’ve got nothing to work with.

Many people have been trained by their experiences in the corporate world not to raise issues without having a solution to offer as well.  We definitely need playtesters to disregard self censorship.  However, instructing them to ‘offer issues, not solutions’ constitutes a direct order them to self-censor. It is not plausible that such directed censorship will free them from internalized censorship.

Instead, encourage openness. Practice saying phrases like “I’ll look at that,” and “how do you feel that would improve the game,” and “how do the rest of you feel about that.” Expressions like these make it safe for them to offer every concern and suggestion.  You will quickly see a change in their candor and their expressiveness.  I guarantee it.

 

Argument 6.   If I incorporate too much playtester feedback, it won’t be my game design anymore.

This was not one of the arguments he put forward but in developing this article, a few other folks did raise the question.

Fashion designer Tom Ford was posed with exactly this quandary in his 2014 interview with Kinvara Balfour.  He has a number of designers working under him and their creations bear his name.  His reply mirrors that which creative leaders have said across the centuries, that ‘while you do listen to everyone’s feedback, it is you who makes the final decision and this is what entitles you to put your name on it.’

While others may have given you a huge number of ideas, it is you who decided what to incorporate and how to incorporate it.  It is this which makes you the craftsman, not the originality of your ideas.  Ask any designer, artist or craftsman and she will tell you that ideas are cheap and plentiful.  Achievement is derived from refining and editing those ideas.

There may be several arguments for “give me your problems, not your solutions”  but I found them to be fundamentally flawed.  Today, we debunked the remaining arguments against soliciting playtester feedback.  That attitude is dismissive, arrogant, unfortunate and ultimately toxic.

On Friday, we return to look at positive ways to playtest and ways to best use player feedback.  See you in three days!

And if you’re near the west coast this weekend, I’ll have the honor of attending Protospiel San Jose.  Grab your prototype and drop by. I’d love to see you there! 🙂

How do you use playtester feedback?  What role would you give your playtesters?  Do you give them any directed instructions or do you encourage them to explore your game?  Or are you a laissez faire tester? Share with your fellow readers in the comments below.  And if you’re enjoying what you’re reading, create and account with WordPress and follow this blog.  If you keep reading, I’ll keep writing.

Advertisements

18 thoughts on “Using Playtester Feedback, Part 2

  1. yvestourigny says:

    I remain unconvinced. Your reply to argument 4 leaves me scratching my head. Who are these analytic geniuses with whom you playtest your games? A single play of any game may leave you with the impression that you have a deep understanding of it, but if that’s the case, I’m afraid that I’ve already filed your feedback under “crank”. The inner workings of a game may in theory be laid bare, but the complexity that emerges when even simple systems start to interact should give pause to anyone who thinks they have an easy fix.

    This is especially true if the playtesters have not read the rules, but have had the game explained to them. Or when the playtest is a partial one, where the entire game may not have been experienced.

    Repeated plays, or a very simple game, may very well give playtesters a complete, holistic picture of the game. This would simply make their cogent analysis of any problems far MORE useful than any solutions they might have for those problems.

    There are times, I believe, when playtester feedback is almost completely irrelevant. Case in point: I playtest many of my games with my two sons. One is 17, and has Aspergers, and the other is 8, and has a very strong desire to express his opinion about everything under the sun. I’ll ask for their feedback, specifically for things they liked or didn’t like. If the 8-year old starts offering solutions, or suggestions (e.g. “Maybe you could add (something that would add nothing)?”), I politely steer the conversation back to useful territory.

    This is in fact what I do with any playtester. I’m not concerned about hurting their feelings if I don’t use their suggestions, because I question them to get to the root concern. Why do they feel this suggestion is required? What problem is it that they are trying to solve?

    Just to be clear, I’m not saying I never want to hear solutions or suggestions, only that without a clear statement of the problem they are next to useless.

  2. yvestourigny says:

    Here’s a question. Which of these three positions are you arguing for?

    A) A solution/suggestion is useful feedback even if there is no clear problem that it is meant to address.
    B) A solution/suggestion is useful feedback if it is accompanied by the clear statement of a problem.
    C) Another position, which I have clearly misunderstood.

    I think that A) can almost by definition never be true because you (the designer) will either:

    (i) press for more details, hoping to get to B)
    (ii) identify the problem yourself, and so will have done the second part of B) yourself
    (iii) ignore the feedback, or
    (iv) make what is to you is an arbitrary change (otherwise it would be case ii) on the chance it might improve your game

    As an arbitrary example, let’s say someone left this comment:
    “Your article was okay, but you should use more words.”

    You might:

    (i) Ask clarifying questions. “Do you mean I should explain things in more detail? That the article needs more content? That I should use a bigger vocabulary?”
    (ii) Already believe that the article glosses over some areas that could be further expanded upon.
    (iii) ignore the feedback
    (iv) Add a few paragraphs, despite not feeling that this is necessary.

    If what you are arguing for is B), then I think that could be made clearer. 🙂
    If it is C), then I humbly await further clarification.

    • I would say (C). I am looking for a variety of data types which will (hopefully) point to the deeper issue that should be addressed.

      I wouldn’t expect most playtesters to be able to articulate every element of the game but I do believe that their comments are useful indicators.

      Designer Gil Hova related a perfect example of this recently–in which playtester suggestions were not the solution at all but pointed him at the real issue.

      Gil’s story will be appearing in its full glory in Friday’s column.

  3. yvestourigny says:

    Quote away!

    The brief description of the example cited supports my point (in a way that the full anecdote may not…). The suggestion (let’s call it “Add Strup”) was not useful – what was useful is that the suggestion led to the underlying issue (let’s call it “Bork”). Imagine how much more useful the feedback could be if instead of saying “Add Strup”, the playtester said “Add Strup because it addresses Bork”, or better yet, since the suggestion wasn’t retained, “I think Bork is an issue.”

    I could be wrong, but I feel that even though the arguments quoted in your article in support of “give me your problems, not your solutions” are weak, that advice is extremely sound, for reasons that I hope I have made clear. Your designer friend may simply have been right for the wrong reasons.

    • Yves, the only issue I might have with your argument is that it supposes a playtester capable of realizing that it is Bork which is the problem. Absolutely, if the tester is acute enough to spot the Bork problem, then saying that would be most helpful but what if the tester says “I see a Bork issue. Adding Strup will deal with this issue” when the real problem is that there are too many Zemos in the game. As designer, I think you must treat every comment testers make as a symptom but not necessarily the cure.

      • I wonder if you’re proving the counter argument here with this example, Kevin. I’ll wager your play tester’s reaction, “I see a Bork issue. Adding Strup will deal with this issue,” is likely to send a designer down the wrong path, if the real problem is that there are “too many Zemos.” Had the play tester’s feedback been simply, “I see a Bork issue,” the designer would necessarily look for the root cause of the Bork issue, and hopefully suss out the Zemo trouble himself, knowing the Strup is just a bandaid.

        This isn’t to say solutions are never useful, but if a dozen play testers offer a dozen solutions to a single problem, the useful data is the problem, not the solutions.

      • You may be right Brett. I’ll have to think on that.

        It may be that I’ll end this series in the other camp. Or somewhere in the middle, perhaps?

      • yvestourigny says:

        Yes, exactly.

        Either the player can identify an issue (or can be helped to do so by asking him questions) or he cannot.

        If he cannot, then whatever solution he proposes will in all likelihood be useless or, charitably, accidentally useful.

        If he can, then it is far more useful for him to do so than to ONLY provide a solution. If he ALSO has a solution, great. However, a solution cannot be assessed if the problem isn’t clear, and what the playtester perceived as an issue may not be one for the designer.

        In any and all cases, in my opinion, “playtester problems” > “playtester solutions”. If I have to work backwards from their perceived solution to get to the root of the perceived issue, so be it.

  4. I think our mutual friends must place a greater value on efficiency of play test feedback, working on a tighter schedule than we part-time designers who have the luxury of time.

  5. “While others may have given you a huge number of ideas, it is you who decided what to incorporate and how to incorporate it. It is this which makes you the craftsman, not the originality of your ideas. Ask any designer, artist or craftsman and she will tell you that ideas are cheap and plentiful. Achievement is derived from refining and editing those ideas.

    There may be several arguments for “give me your problems, not your solutions” but I found them to be fundamentally flawed. Today, we debunked the remaining arguments against soliciting playtester feedback. That attitude is dismissive, arrogant, unfortunate and ultimately toxic.”

    Couldn’t agree more with both those paragraphs. I know as a designer and creator, there’s a natural instinct to raise my hackles at suggested solutions, and this is something that I actively try to suppress (until I’m in the very final stage of development). It’s the process that matters and more ideas, more iterations, does lengthen development but I honestly believe it makes a batter. I take everything sensible on board, put it on the table and run it through the hypothetical game design evaluation engine, and then evaluate what’s junk. That process can lead to a better solution or an insight into what is actually working within the game and why.

    I think a “problems not solutions” attitude might work for a very dry, unthematic game with zero variability, but honestly, that’s not a game I’d to test, design or play. Thanks for the insight!

  6. alanleduc says:

    “However, a solution cannot be assessed if the problem isn’t clear, and what the playtester perceived as an issue may not be one for the designer.” Yves, briefly, makes another good point here.
    All to often, “problems” are really just personal taste. Without knowing those preferences of the playtesters, it’s harder to use their feedback. If playtesters aren’t aware of a designers goals for his game, they also find it harder to provide useful feedback.
    For example, one might hear “I need more things to spend my money on”, which could mean “there aren’t enough choices”, which could lead to “there are only false choices”, which might be because your two currencies are two equally valued at the point in the game which the playtester is thinking of. Or maybe the playtester just loves take-that elements, and there weren’t any in you game.
    Other designers, in particular, are prone to giving feedback that would direct the game along the lines that they would take if they were designing it. They might just see different problems then you do, for better or worse.

  7. Thanks for great articles. Just one thought. You might have reached the same conclusions, but anyways, here goes..

    Anyone who has worked in repairs will tell you that it is important to identify if there is a problem in the first place before going on a wild goose chase trying to fix things in accordance to problem descriptions. I think we all agree on that. When repairing things you should always reproduce the faulty behavior if it’s not immediately obvious what the problem is (like a burned component). A lot of times the problem is in the user, a simple misunderstanding. The user did not read the manual or misunderstood something. But clarifying such issues is no less important however. Why did the user misunderstand? It doesn’t have to be the case always that the user is dumb (or for the sake of this discussion: opinionated), it might be that the product is not as intuitive as it could be or is simply badly designed in some minor way (as opposed to faulty). In board game design I imagine the same principle applies. You should try to find out why the user was confused even if there is no problem to be found. Cause confused customers are a problem in itself, a problem that you will profit from solving. Perhaps the rules were not clear enough? Perhaps some element in the game could be streamlined to avoid confusion? Perhaps the connection between mechanics and theme were broken at some point for that player? Surely if one of your play testers has this problem in his own mind, other users will have the same thoughts when the game is played by hundreds. And they might not be so courteous to tell you, they’ll just stop doing business with you, playing or buying your game. For every 1 complainer, there are at least 10 that simply just don’t come back. This is why I think all feedback is important. True, you as the designer is the expert, and the consumer is likely to give bad advice. But as a designer you should try to make the experience of playing the best it can be. So imaginary problems become real problems.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s