An Experimental Psychologist's Take on Beta Reading Part III: Data Collection

So far in this series we've talked about selecting beta readers, recruiting them, and getting them to read. Now is the part we’ve all be waiting for: getting feedback.

There are two challenges to soliciting feedback (especially if the beta readers are not writers themselves). First, readers might not have much to say beyond “It was okay.” Second, even if they do say something,  they might just be telling you what you want to hear*. I tried to minimize these problems by doing the following:


1) Asking open-ended non-leading questions about my manuscript, and


2) Providing objective anchor points for subjective evaluations.

When a reader finished my book, I first asked them for general reactions. At this stage, writers tend to give detailed responses, while non-writers vary in how much they have to say. After I got a beta reader’s unprompted response, I sent a list of questions.

The following questions are tailored specifically for my book, but they might give you some ideas.

1. On a scale of 1 to 7, please rate your average level of engagement throughout the book. (1 = you have to pay me to keep reading, 4 = I'll read it if it's lying around and I have free time, 7 = I am neglecting important life tasks to keep reading)

This was the closest I got to asking if they liked the book. To offset the ackwardness of putting betas on the spot, I tried to get as close as I could to an objective measure and provided concrete examples for different values of the scale. While people were still too nice to say that my book sucked, the concrete examples often prompted them to tell me how they arrived at their answer (i.e, “Six, because I found myself reading instead of going out to see a movie,” or “Four and a half, because I stopped reading after chapter 10, but got more into it when I picked it up a week later”). This gave me a clearer picture of what people actually thought.

2. If your interest level varied throughout the book, can you tell me where it fluctuated and why?

This question was useful for picking up inconsistencies in pacing.

3. How did you feel about the interaction between [young impressionable protagonist] and [dangerous yet strangely intriguing antagonist] throughout the course of the book?

4. How did you feel about the development of [female protagonist] and [male protagonist]’s relationship throughout the course of the book?

5. How did you feel about [ZOMG major plot twist]? Did you buy it? Did you see it coming?

6. How did you feel about the climax and resolution? Did it feel believable? Was it a satisfying ending?

7. What would you say is the novel's biggest strength and biggest weakness?

My betas found this question obnoxious, like those job interview questions where they ask you about your biggest weakness (The correct answer is “I work too hard.”), but it was good for balancing out beta reader personalities. For the overwhelmingly positive people, it was useful to get something that they didn't like.For the “praise is for pansies”camp, it was nice (and soothing to my ego :-P) to see if they liked anything at all.

8. Did you get a good sense of the characters? Were there any characters that were particularly well fleshed out or needed more fleshing out?

Note that different readers will have different criteria for strong characterization. For example, some readers thought my villain was my strongest character, while others thought he needed the most fleshing out. It turned out that the readers who thought he was strong were drawn to his dialogue and mannerisms (on-screen charisma, if you will), while the readers who thought he needed work placed more value on backstory and motivation. It's helpful to ask for clarification if you're not  sure what people mean.

9. What type of books do you usually read for pleasure, and what percentage of those books are fantasy? What about YA fantasy? What are some books that you think were done well?

The most important question in the survey. You must know where your readers are coming from, and what they are using as the gold standard. A comment on pacing from a classics lover is completely different from the same comment made by a thriller reader. The same goes for character development, plot structure, voice, and pretty much everything else. In general, people who regularly read your subgenre will be the most helpful, but I ended up getting useful tidbits from every beta reader regardless of their reading habits.

A few other notes:

1. You will be getting a lot of comments. Which ones should you actually address?

One rule of thumb is to address 1) feedback that immediately resonates with you and 2) feedback that doesn't resonate, but comes up too many times to ignore. Sometimes it's a tough call, which leads me to the second point.

2.It helps to have an objective and trusted sounding board to help you with borderline cases

When I wasn't sure about a certain piece of advice, I was lucky enough to have my critique group to bounce e-mails and ideas off of.

3.Don't argue with feedback.

Remember, you're a psychologist, and the last thing you want to do is contaminate your data by arguing with your test subject. Instead, think to yourself “my, what an interesting specimen of reader,” and ask him to elaborate while taking copious notes. Then you can go home and write a five page psychoanalysis about how that reader’s early relationship with his mother ended up giving him tragically bad taste in literature. (I'm just kidding about that last part. I swear.)

4.  Be prepared to give feedback on the feedback
I found that non-writers were more likely than writers to ask about the quality of their feedback. This is probably the because the process is new to them. So be prepared for the question, and use your judgment. Sometimes beta readers were curious about whether they agreed with others, so I went through their comments and summarized what other people thought. Other times, I just picked a few points and commented on those. I avoided bringing up issues I disagreed strongly with because I'm the type of person who once embroiled in a debate, MUST WIN.

5. Some other helpful tips from blog readers:
Friends of the blog have been sharing their own ways of getting feedback. Margo provided her beta readers with a PDF with the link at the end of each chapter to a Google survey with questions about what worked, what didn't, and what was boring. Jack does his writing on Google Docs and has beta readers interact with him and each other using the chat function while he's actually writing.

Next week, we'll talk about what I learned from the experiment.  Now readers, it's your turn. Any suggestions for good questions to ask your beta readers?

*In experimental psychology, this is referred to as demand characteristics, when a participant develops a theory of what the experiment is testing and therefore doesn't act naturally. The most common instance would be the "good participant," the beta readers that tells you your book was awesome because they think that’s what you want to hear. But there are also “bad participants .” One particularly mischievous beta reader (who may or may not be married to me) made a point of reading to the middle of the most exciting parts and then making an elaborate display of yawning and announcing that he was going to do something else. *sigh*


To get regular updates from the blog, use the subscription options in the left sidebar.

8 comments:

  1. I especially like "a few other notes: 1-3." If asking others for honest opinions, you better not cry or get mad when you get them. An editorial question is, "Were there any parts that were not clearly understandable or that you felt didn't belong." It would be useful to give these questions before they read so they can be thinking about them.

    ReplyDelete
  2. Haha, I often argue with beta readers (as noted as one of the things I shouldn't do). I should probably stop.

    As for question, there should probably be something about the beginning. As in, how engaging was the beginning? And maybe some number ratings (because number ratings are awesome).

    I am so going to copy this list of questions and use them next time I need beta reading.

    (I am also someone who HAS to win any debate/argument. I just keep going until the other party gives up.)

    ReplyDelete
  3. Thanks for sharing this. I'll definitely be referring back to this when I'm at the beta reader stage. :-)

    ReplyDelete
  4. Considering the recent feedback I got from friends and family on a short story (all praise, only two with helpful comments), I can certainly appreciate what this kind of questionnaire would be able to accomplish. I may have to have to steal this and adapt it for my own nefarious purposes. Mwahahahaha!

    Although, my purposes aren't as nefarious as your beta tester who may or may not be your husband... luckily, my wife hasn't yet reached the elaborate faking of boredom stage of our relationship.

    The only question I might think to add is similar to #8, but for situations or concepts rather than characters. "Were there any spots where more information would have helped your understanding (i.e. questions remain unanswered)? Were there any spots where I provided too much explanation (where you'd have preferred to infer)?" The wording would need to be fine-tuned, of course, but you get the idea.

    Fantastic work as usual, Livia. Thanks for sharing!

    ReplyDelete
  5. Another awesome post, Livia.
    Thanks for the shout-out in #5! Though, for my next round of Google forms, :) I may take a cue from some of the questions you've listed here. Such good points. Thanks again for offering up such fantastic insights!

    ReplyDelete
  6. I've seen work (fiction) that was clearly bad writing posted on the internet, yet praised effusively by many people in the comments. Always makes me wonder if I can trust compliments. As always, thanks for the post. I may actually decide to be more mindful with my book, prior to publishing. You're a good influence.

    ReplyDelete
  7. As someone who literally just sent a novel draft to the first round of beta readers, I am planning to steal shamelessly from your list.

    Orson Scott Card wrote about having a "Wise Reader" in his book on how to write spec fic. I can't seem to find the whole thing on a quick search of his website, but it seems to have become a pretty widespread idea. I have used his questions in the past to ask things like, "Were there any times you said OH COME ON!" or "Were there any times you found yourself getting bored?" and the like.

    *cough* I like your questions better. They're more specific and the scale of 1-7 will probably be appealing to betas.

    ReplyDelete
  8. Linda – yeah, it's important to be able to keep some emotional distance from your work

    Jake – Lol. I find that it's easier not to argue with beta readers when there are a lot of them, because you know that their opinion will either be confirmed or not by the rest of the readers.

    Cally - glad you found it useful

    Nate - when I hear the phrase "elaborate faking of boredom" in a marital context, all of a sudden, I'm not thinking about beta reading anymore

    Margo - I love Google docs. We used them to get our wedding guests' mailing addresses.

    Janice – yeah, the Internet is not very reliable for feedback. In some circles, you just get praise. In others, you get a lot of trolls.

    Stephanie – I'm familiar with the wise reader concept, but have never heard any of this specific questions. Thanks for bringing them up.

    ReplyDelete