An Experimental Psychologist's Take on Beta Reading Part III: Data Collection

So far in this series we've talked about selecting beta readers, recruiting them, and getting them to read. Now is the part we’ve all be waiting for: getting feedback.

There are two challenges to soliciting feedback (especially if the beta readers are not writers themselves). First, readers might not have much to say beyond “It was okay.” Second, even if they do say something,  they might just be telling you what you want to hear*. I tried to minimize these problems by doing the following:


1) Asking open-ended non-leading questions about my manuscript, and


2) Providing objective anchor points for subjective evaluations.

When a reader finished my book, I first asked them for general reactions. At this stage, writers tend to give detailed responses, while non-writers vary in how much they have to say. After I got a beta reader’s unprompted response, I sent a list of questions.

The following questions are tailored specifically for my book, but they might give you some ideas.

1. On a scale of 1 to 7, please rate your average level of engagement throughout the book. (1 = you have to pay me to keep reading, 4 = I'll read it if it's lying around and I have free time, 7 = I am neglecting important life tasks to keep reading)

This was the closest I got to asking if they liked the book. To offset the ackwardness of putting betas on the spot, I tried to get as close as I could to an objective measure and provided concrete examples for different values of the scale. While people were still too nice to say that my book sucked, the concrete examples often prompted them to tell me how they arrived at their answer (i.e, “Six, because I found myself reading instead of going out to see a movie,” or “Four and a half, because I stopped reading after chapter 10, but got more into it when I picked it up a week later”). This gave me a clearer picture of what people actually thought.

2. If your interest level varied throughout the book, can you tell me where it fluctuated and why?

This question was useful for picking up inconsistencies in pacing.

3. How did you feel about the interaction between [young impressionable protagonist] and [dangerous yet strangely intriguing antagonist] throughout the course of the book?

4. How did you feel about the development of [female protagonist] and [male protagonist]’s relationship throughout the course of the book?

5. How did you feel about [ZOMG major plot twist]? Did you buy it? Did you see it coming?

6. How did you feel about the climax and resolution? Did it feel believable? Was it a satisfying ending?

7. What would you say is the novel's biggest strength and biggest weakness?

My betas found this question obnoxious, like those job interview questions where they ask you about your biggest weakness (The correct answer is “I work too hard.”), but it was good for balancing out beta reader personalities. For the overwhelmingly positive people, it was useful to get something that they didn't like.For the “praise is for pansies”camp, it was nice (and soothing to my ego :-P) to see if they liked anything at all.

8. Did you get a good sense of the characters? Were there any characters that were particularly well fleshed out or needed more fleshing out?

Note that different readers will have different criteria for strong characterization. For example, some readers thought my villain was my strongest character, while others thought he needed the most fleshing out. It turned out that the readers who thought he was strong were drawn to his dialogue and mannerisms (on-screen charisma, if you will), while the readers who thought he needed work placed more value on backstory and motivation. It's helpful to ask for clarification if you're not  sure what people mean.

9. What type of books do you usually read for pleasure, and what percentage of those books are fantasy? What about YA fantasy? What are some books that you think were done well?

The most important question in the survey. You must know where your readers are coming from, and what they are using as the gold standard. A comment on pacing from a classics lover is completely different from the same comment made by a thriller reader. The same goes for character development, plot structure, voice, and pretty much everything else. In general, people who regularly read your subgenre will be the most helpful, but I ended up getting useful tidbits from every beta reader regardless of their reading habits.

A few other notes:

1. You will be getting a lot of comments. Which ones should you actually address?

One rule of thumb is to address 1) feedback that immediately resonates with you and 2) feedback that doesn't resonate, but comes up too many times to ignore. Sometimes it's a tough call, which leads me to the second point.

2.It helps to have an objective and trusted sounding board to help you with borderline cases

When I wasn't sure about a certain piece of advice, I was lucky enough to have my critique group to bounce e-mails and ideas off of.

3.Don't argue with feedback.

Remember, you're a psychologist, and the last thing you want to do is contaminate your data by arguing with your test subject. Instead, think to yourself “my, what an interesting specimen of reader,” and ask him to elaborate while taking copious notes. Then you can go home and write a five page psychoanalysis about how that reader’s early relationship with his mother ended up giving him tragically bad taste in literature. (I'm just kidding about that last part. I swear.)

4.  Be prepared to give feedback on the feedback
I found that non-writers were more likely than writers to ask about the quality of their feedback. This is probably the because the process is new to them. So be prepared for the question, and use your judgment. Sometimes beta readers were curious about whether they agreed with others, so I went through their comments and summarized what other people thought. Other times, I just picked a few points and commented on those. I avoided bringing up issues I disagreed strongly with because I'm the type of person who once embroiled in a debate, MUST WIN.

5. Some other helpful tips from blog readers:
Friends of the blog have been sharing their own ways of getting feedback. Margo provided her beta readers with a PDF with the link at the end of each chapter to a Google survey with questions about what worked, what didn't, and what was boring. Jack does his writing on Google Docs and has beta readers interact with him and each other using the chat function while he's actually writing.

Next week, we'll talk about what I learned from the experiment.  Now readers, it's your turn. Any suggestions for good questions to ask your beta readers?

*In experimental psychology, this is referred to as demand characteristics, when a participant develops a theory of what the experiment is testing and therefore doesn't act naturally. The most common instance would be the "good participant," the beta readers that tells you your book was awesome because they think that’s what you want to hear. But there are also “bad participants .” One particularly mischievous beta reader (who may or may not be married to me) made a point of reading to the middle of the most exciting parts and then making an elaborate display of yawning and announcing that he was going to do something else. *sigh*


To get regular updates from the blog, use the subscription options in the left sidebar.