Using Playtester Feedback, Part 1

A designer buddy quoted another designer to me a few weeks ago.  “Give me your problems, not your solutions.”

This confused me.

So I consulted the designer in question.  Yup.  “Give me problems, not solutions.” That’s a direct quote.

I asked for further clarification.  Wouldn’t you?

He obliged my curiosity and related the reasons which led him to this conclusion.  First, he expressed that the suggestions playtesters give are usually bad.  Second, that while the playtester is relating the suggestion, he was generally running other–presumably better–variants in his head.  Third, that playtesters tended to get attached to their suggestion and resentful if that suggestion isn’t used in the final design.  Fourth, he stated that the playtesters could not see the inner workings of the design and were therefore ill-equipped to make informed suggestions.  Fifth, that many players don’t want to raise a problem without having a solution to offer as well so asking players to only express issues freed them to raise issues to which they see no solution.

A friend of his–another designer–stepped up and agreed with these assertions before they left for dinner, leaving me to reflect on their point of view.

feedbackI went on to discuss this position with other designers.  I have sought the opinion of creative people whom I respect.  I consulted folks that can be trusted to give you an honest opinion, who are candid to the point of brutality.  These conversations led to a number of useful perspectives.  Considering each enabled me to codify my own position.

I am only one person. I won’t pretend to be an expert on any designer’s method but my own.  If an adage like “give me problems, not solutions” fits your design philosophy and it works for you then by all means continue.  But I cannot imagine working successfully with such an attitude because it seems to me that the arguments underlying this approach are fundamentally flawed.

Argument 1.  Playtester suggestions are usually bad.

A friend in college was terrified of asking women out because they might say no.  My response was and is that “you’re right.  That interesting woman over there might say no.  But if you don’t ask her, you’re saying no for her.  You’re denying her the chance to say yes.”

Do you feel that the suggestions playtesters give are usually bad?  Let’s assume you’re right.  The idea that tester wants to give might be bad.  But if you tell him not to express it, you’re denying him the chance to offer a great suggestion.”

“Usually useless” = “sometimes useful.”  So listen to every idea that comes your way and keep looking for the good ones.

Who cares if 99% of the suggestions playtesters offer are bad?  Be assured that 99% of my ideas are bad.  Einstein said the same thing about his ideas.  That’s exactly why we need as many ideas as we can get.  With a 1% success rate, we had better be looking everyplace we can for those diamond ideas.

Argument 2. While the playtester is relating the suggestion, the designer could be contemplating other–presumably better–variants in his head.

Playtesters are the sole authority of their own experience.  There is not a single person in existence better qualified to comment on the player’s experience than the player herself.  We designers seek–and sometimes bribe or beg–her to share her experiences with us.  It is critical to gather as much of these as possible.

You certainly need to hang on to the ideas in your head.  It is equally important to hang on to the ideas coming from your testers.  You are going to have a hard time processing playtester feedback while your brain is off creating other variants.  And people can tell when you’ve checked out.  If you stop listening while they’re offering feedback, your testers will rightly choose not to return to your table.  Playtesters choose how to expend their energy.  They’ll choose the designer with the courtesy to respect their feedback.

It is for this reason that I always have my notebook open and ready to record.  I’m set to jot down my thoughts and feelings while still recording those of the testers.  “It felt like _____,” and “why would I ever_____,” and “this would be better if ______” are all data you need to capture.  Sort through them later. Remember that 99% of them are probably bad–both you ideas and theirs.  That’s why you need them all. Every single one of them.

 

Today, we debunked the first two arguments against soliciting playtester feedback.  On Tuesday, we return to examine the three which remain along with a sixth one which arose during my research.

How do you use playtester feedback?  What role would you give your playtesters?  Do you give them any directed instructions or do you encourage them to explore your game?  Or are you a laissez faire tester? Share with your fellow readers in the comments below.  And if you’re enjoying what you’re reading, create and account with WordPress and follow this blog.  If you keep reading, I’ll keep writing.

Advertisements

Playtest Feedback Form: Doing It The Looney Labs Way

While researching my current series on playtesting, I discovered an excellent example in the Looney Labs playtesting workshop page here.

Awesome and wonderful folks that they are, I secured their blessing to share it with all of you.  Question Answerer 1st Class Alison Looney tells me that it has been working well for them.

This form was obviously designed to collect playtest data on several games at once.  You will have to make a some adjustments If you’re only working on one or two games.  Setting that particular detail aside, I think there are several things any of us us can learn from the Looney Labs approach. 

Image

Notice that it asks few specific questions but instead has been designed to be open ended.  Notice that a huge amount of space has been dedicated to “comments,” including the back of the page.  Keeping their form open ended will encourage Looney’s testers to be sincere.  Some will share many impressions and comments while others may offer few.  This is a good thing.  In both cases, the feedback will be sincere.

I have seen several feedback forms that attempted to cover every conceivable question.  I created a several of these myself.  The problem with this approach is that it presupposes the creation of such an exhaustive list to be possible.  But no matter how hard we try, there will always be questions we missed.

At the other extreme, a blank piece of paper gives your testers limitless freedom but no guidance.  The tabula rasa approach to data gathering is more likely to lead to the creation of paper airplanes.

Looney Labs has found an excellent balance. They make sure to ask the critical questions–what did you play?  What did you like best?  Second best?–and left the rest to the tester’s creative mind to offer.

We would all be wise to follow a similar model.