A designer buddy quoted another designer to me a few weeks ago. “Give me your problems, not your solutions.”
This confused me.
So I consulted the designer in question. Yup. “Give me problems, not solutions.” That’s a direct quote.
I asked for further clarification. Wouldn’t you?
He obliged my curiosity and related the reasons which led him to this conclusion. First, he expressed that the suggestions playtesters give are usually bad. Second, that while the playtester is relating the suggestion, he was generally running other–presumably better–variants in his head. Third, that playtesters tended to get attached to their suggestion and resentful if that suggestion isn’t used in the final design. Fourth, he stated that the playtesters could not see the inner workings of the design and were therefore ill-equipped to make informed suggestions. Fifth, that many players don’t want to raise a problem without having a solution to offer as well so asking players to only express issues freed them to raise issues to which they see no solution.
A friend of his–another designer–stepped up and agreed with these assertions before they left for dinner, leaving me to reflect on their point of view.
I went on to discuss this position with other designers. I have sought the opinion of creative people whom I respect. I consulted folks that can be trusted to give you an honest opinion, who are candid to the point of brutality. These conversations led to a number of useful perspectives. Considering each enabled me to codify my own position.
I am only one person. I won’t pretend to be an expert on any designer’s method but my own. If an adage like “give me problems, not solutions” fits your design philosophy and it works for you then by all means continue. But I cannot imagine working successfully with such an attitude because it seems to me that the arguments underlying this approach are fundamentally flawed.
Argument 1. Playtester suggestions are usually bad.
A friend in college was terrified of asking women out because they might say no. My response was and is that “you’re right. That interesting woman over there might say no. But if you don’t ask her, you’re saying no for her. You’re denying her the chance to say yes.”
Do you feel that the suggestions playtesters give are usually bad? Let’s assume you’re right. The idea that tester wants to give might be bad. But if you tell him not to express it, you’re denying him the chance to offer a great suggestion.”
“Usually useless” = “sometimes useful.” So listen to every idea that comes your way and keep looking for the good ones.
Who cares if 99% of the suggestions playtesters offer are bad? Be assured that 99% of my ideas are bad. Einstein said the same thing about his ideas. That’s exactly why we need as many ideas as we can get. With a 1% success rate, we had better be looking everyplace we can for those diamond ideas.
Argument 2. While the playtester is relating the suggestion, the designer could be contemplating other–presumably better–variants in his head.
Playtesters are the sole authority of their own experience. There is not a single person in existence better qualified to comment on the player’s experience than the player herself. We designers seek–and sometimes bribe or beg–her to share her experiences with us. It is critical to gather as much of these as possible.
You certainly need to hang on to the ideas in your head. It is equally important to hang on to the ideas coming from your testers. You are going to have a hard time processing playtester feedback while your brain is off creating other variants. And people can tell when you’ve checked out. If you stop listening while they’re offering feedback, your testers will rightly choose not to return to your table. Playtesters choose how to expend their energy. They’ll choose the designer with the courtesy to respect their feedback.
It is for this reason that I always have my notebook open and ready to record. I’m set to jot down my thoughts and feelings while still recording those of the testers. “It felt like _____,” and “why would I ever_____,” and “this would be better if ______” are all data you need to capture. Sort through them later. Remember that 99% of them are probably bad–both you ideas and theirs. That’s why you need them all. Every single one of them.
Today, we debunked the first two arguments against soliciting playtester feedback. On Tuesday, we return to examine the three which remain along with a sixth one which arose during my research.
How do you use playtester feedback? What role would you give your playtesters? Do you give them any directed instructions or do you encourage them to explore your game? Or are you a laissez faire tester? Share with your fellow readers in the comments below. And if you’re enjoying what you’re reading, create and account with WordPress and follow this blog. If you keep reading, I’ll keep writing.
Kevin G Nunn 
Seeing as you’ve actually used some of my improvement suggestions in some way/shape/form, I think I agree with your approach 🙂 But, I think there is a middle ground – let them know that it’s ok to bring up a problem they can’t solve, and proactively warn them that although you’ll consider their suggested solutions you don’t guarantee you’ll use them. You’re assertive enough to let people know that you don’t mind harsh criticisms; there may be other designers with softer personalities that people might be more reluctant to do this with. If the designer tells them “Be harsh, I can take it – and I’d rather you be harsh with me now than read hundreds of bad reviews after publication” then maybe that would encourage candor, if a designer is concerned they aren’t getting it.
I agree completely, Anye. The next part of this article includes similar tips under ‘playtester resentment if her suggestion isn’t used.’
A particularly useful article on this point comes from Imaginatik, (http://www.imaginatik.com/blog/making-feedback-more-positive) offering a blanket statement which offers affirmation and appreciation even while acknowledging that not every suggestion can be used.
great thoughts! I love to hear problems and I love to solve them. For me, that’s the fun of design and I’m very selfish when it comes to sharing that fun. However, you’ve opened my eyes to the benefits of others suggestions… especially if I only have to share 1% of the fun!
An excellent analysis, as we might expect.
If I heard ‘Give me your problems, not your solutions’, I’d take it as a request to tell the designer what I thought wasn’t working with the game. When one of my games is being played, I watch the players intently. What is confusing them, what is counter-intuitive, what are they trying to do but can’t – those are the problems. If they can express them to me, so much the better – that’s the best interpretation of this demand.
Once a problem is identified, wise designers will listen to potential solutions, no matter how improbable; even the largest pile of manure may support a flower! Or it could be burned. Could we build a house, if we had enough manure? Is the manure such that it can be tossed, and a different game created? Has it attractive patterns, can it be sold to aliens as art?
All ideas should be welcome, though most must be discarded. Why would one shut that door? The possible benefits far outweigh the minor amount of time that might be spent. Your playtesters’ spend their time on your game; is your time so important you can’t listen to ANY comments your game inspires? Surely not.
Trying playtester suggestions, time permitting, gratifies playtesters and demonstrates better than any argument, why a thing will or won’t work. We recently transformed a fine 3-4 player game into a 3-6 player game by coaxing the designer to make a fairly simple change. Receptive designers make better games.
But the best question I am asked is WHY does this work this way? I should know why I have any constraint or mechanic in the game – is it for balance, for smoothness, to make other actions intuitive? If I can’t explain why something happens the way it does, I should be willing to consider changing it, and if a random player suggestion inspires a change that improves the game. I’d better have my notebook out and open.
Playtester feedback should come in the form that is easiest for the playtester to express, not the convenience of the designer.
Feedback is the beginning of a discussion.
Though you may not agree with or use the suggestion, it indicates the problem.
This is a great piece on play tester feedback. For me personally, I go back and forth with how to deal with feedback. It’s almost like therapy. You have to kind of listen for the underlying unspoken truths behind the words. I like how in your piece you present different angles not just one perspective. I look forward to the next installment. I’ve linked this blog to my latest piece on the League of Gamemakers: Refining Your Game. http://www.leagueofgamemakers.com/refining-your-game/
-Luke
And that’s why the most annoying feedback comment to hear is, “I don’t like this, but I don’t know why.” If someone can tell me what they’d rather see happen in my game, then that at least tells me what they’re thinking about and what they didn’t like to begin with.
It’s not the playtester’s job to fix my game. I hate the notion that if they don’t have a suggestion, they shouldn’t mention the problem: I’ve never understood that idea in any context. But I welcome ALL suggestions, and I think my designs have benefited greatly for it.
I’m in the camp who doesn’t want to hear the suggestion, at least not until the problem has been sufficiently well stated, and maybe not even then.
Most of the playtesting that I do is with other (opinionated) designers. We are all brimming with ideas and solutions, not all of them bad. However, I’ve taken to heart psychologist Norman Maier’s lesson (from his book ‘Problem solving and creativity in individuals and groups’):
“Do not propose solutions until the problem has been discussed as thoroughly as possible without suggesting any.”
He noted that the natural tendency of any group is to propose possible solutions as they start discussing the problem. The discussion then centers around the merits of the proposed solutions, people become emotionally attached to their solutions (which speaks to the topic of your next post), and superior solutions may not be sought.
If the problem isn’t clearly identified, then there is no way to evaluate the suggestion, other than by instinct. In the event that it does not work to solve the ptoblem(s), I would be no further ahead.
If the problem HAS been clearly identified and described, then I see no harm in listening to suggestions. On the other hand, I value my time, and I’m not willing to sit and listen to just anyone’s suggestions if I expect that only 1% are “good”. I like to think that my hit rate is at least 3%, so I’d be better off retreating to my office. 😉
This discussion reminds me of a fundamental premise of requirements analysis: you must define the problem before you attempt to solve it. And the definition of a problem is “The difference between a situation as perceived and the situation as desired.” That definition means that any problem can be solved in three ways:
1. Ensure that the perception is accurate. The problem may merely be a misunderstanding.
2. Validate the desire. What may seem to be a problem may merely be a preference, or even a whim, that doesn’t provide enough return on investment to be worth resolving.
3. Change the situation. This is the most effort, and can have unintended consequences, so it’s the option that should be chosen last. But too much of the time we pick this option first.
I’m kind of on the fence too. Any creative professional can tell you, it’s not about the ideas, it’s about the development. While playtesters will be better at looking things from a new perspective and identifying problems and designers will be better suited at coming up with solutions that best fit the design, it doesn’t follow that any sort of role overlap should be mutually exclusive. If only 1% of changes are good, then you’d better have 100 changes (although I wouldn’t be quite so pessimistic about the %, bearing in mind the obviously bad ones can be quickly eliminated with a quick exploration). If someone has taken the time to sit down and play your game, I think the least you can do is hear them out, although depending on the personality of the playtester, you may be dealing with some hurt feelings about not taking a particular suggestion into the game.
I think it’s also a case of stages of development. Suggestions are more welcome in early tests when you really want to try and get as many ideas as possible so you can refine them down to a workable game, but less helpful in balance testing when playtesters don’t fully understand the underlying economy and interactions with the game.
Reblogged this on play without fear and commented:
My designer buddy, Kevin Nunn, is exploring the different ways we game designers utilize play tester feedback. This series has sparked an interesting discussion in the comments and is worth checking out.
I haven’t committed to either camp in the debate, but I can see the merit in our mutual friend’s play test mantra: “Give me problems, not solutions.”
Considering the argument from a data analysis perspective, if a dozen play testers offer a dozen solutions to a single problem, the useful data is the problem, not the solutions.
Like most game design questions, the answer is clearly “it depends”. If you’re playtesting with a lot of game designers, you’d be nuts not to solicit and listen to their suggested solutions. If you’re playtesting with people who are just playing what may turn out to be fun games, even though they’re prototypes, people who don’t really think of themselves as playtesters, then you’re unlikely to solicit much comment, and often you won’t get many comments. The PTers are helping me identify problems, whether explicitly, or by my observations of their behavior. Though I cannot imagine not listening to a suggested change, the time it takes is minimal and “two heads are better than one”, though I also agree that the problems are mine to solve.
In other words, it depends on the mindset of the testers, and on how much they think like designers. There is no single answer.