How to Write Questions

Jerry Vinokurov's guide to writing questions, originally posted here.

Full text

Now that the discussion of the various tournaments has finally petered out, I thought I'd promulgate some dogma on question writing, seeing as how I'm always yelling really loudly about question quality and such. I encourage other players to contribute, and I'll incorporate good suggestions into the original post. Before I get going, I'd like to point out that many of the things I'll be writing about are settled issues with respect to question writing, but many others reflect my own stylistic preferences, and I'll be sure to try and point out which are my own biases. To begin:

Before Writing Questions

Look around you. Look around you. Just look around you. Do you see what we're looking for yet? That's right, the subject of today's lesson is sources.

I'd like to start this guide with a general discussion of what makes a good question-writing source. There are many things that one can use, but obviously not all sources are equally reliable. Notably unreliable sources include Wikipedia and random geocities web pages. Reliable sources include scholarly journals, textbooks, published books, and encyclopedias of various sorts. If you can get your hands on it, the electronic version of the Encyclopedia Britannica is superb for many topics, particularly history. Masterplots are a good resource for literature plot summaries, and there are many topic-specific encyclopedias that are surely available at your local university (or public) library (I'm assuming that virtually all of you have some sort of university affiliation).

Wikipedia may be worthy of a special topic of its own; I mention it here to point out that while it may be a nice source for brainstorming before you actually start writing, you should be thrice-wary of anything you read on Wikipedia. I don't want to get into a discussion of the problems that inhere in the Wikipedia model. Suffice to say that it is not a trustworthy source, though it does in fact contain much that is true. Anything read on Wikipedia should be cross-checked with a more reliable scholarly source. For that matter, pretty much anything you read on the internets that doesn't come from a journal should be double-checked. There are some sites which are pretty reliable and contain some useful information; those are usually associated with universities in some way. The worst part of using Wikipedia as a source is the noted problem of Wikiplagiarism, which is what occurs when people just copy sentences out of Wikipedia. Wikiplagiarism can be easily spotted by competent editors because it contains weird phrasings that usually shouldn't appear in questions, as well as editorializing. I'll have more on style topics below.

Another useful source is online lecture notes (thanks Dwight), whether they come from a class you're taking or just from some stuff professors at other institutions put up. These are generally reliable (they have been in my experience) and ordinarily are pretty easy to get, since many professors just make their notes available to whoever. A notably good source for such notes is MIT's Open CourseWare site, which contains a lot of excellent information on various scientific topics (and some others too).

All this makes it sound like sources are hard to get to and require a lot of work. For the most part, you should be able to get a lot of mileage out of a few books available from whatever library is easiest for you to get to.

Deciding What to Write About

This is arguably the hardest phase of the whole process; I know for me it's usually the stumbling block, whereas once I have my topics picked out things generally go pretty smoothly. How can you make a good decision regarding what to write? Read below to find out!

First, you have to know your audience. Are you writing for a novice tournament, a standard invitational, or ACF Nationals? This will determine, to some degree, what answer selections are appropriate in your packet, and later on, what clues you use to construct this question. Second, you should ask yourself whether the answer you are thinking about will make for a good question; although most answers are fine, all other things being equal, some answers don't lend themselves well to good questions and should be avoided if possible. I'll try to outline some of that below. Finally, you should ask yourself whether writing on this particular answer choice is going to require a lot of research. This has less to do with writing a good question and more with budgeting time for doing so. If you find yourself spending an hour per tossup, you're either doing something wrong or picking topics that take far more time to research than they should.

With regard to difficulty, context matters a lot. If you're writing for a tournament like ACF Fall, which is going to include a lot of new players who are, at best, familiar with the high school canon, the goal should be for every tossup to be answered by every team if read to that team to completion against empty chairs. Obviously, this is an ideal; sometimes, people blank on the answer, and sometimes they just don't know it, but at tournaments which are supposed to be introductions to collegiate quizbowl for most participants, the great majority (90%, say) should at least have heard of the answer. So, a tossup on Herbet Spencer would be fine, whereas a tossup on Social Statics is almost certainly too hard.

For harder tournaments, the field of allowable answers is expanded, but it can still be hard to know whether what you're writing is appropriate. I think the following method provides a decent rule of thumb that one can follow: consult past instances of this tournament. If a clue appears multiple times in a tossup, but doesn't appear as a bonus part, it may be all right to make it the hard part of a bonus. If something appears as the hard part multiple times, you could probably promote it to one of the easier parts. If something appears multiple times as a middle or easy part of a bonus, it's probably good tossup material. At first, this might seem like a daunting task, but I think you'll find that if you read packets in practice and go to tournaments, you'll quickly get a handle on what's hard and what's not. At the very worst, by following this method, you will likely err on the side of an easier topic, which is fine. It also makes a difference whether you're writing tossups or bonuses. For example, a European history bonus that included parts on the War of the Spanish Succession, the Treaty of Utrecht, and Philip V would be perfectly reasonable for ACF Fall, but a tossup on Philip V himself would probably be excessive. On the other hand, Philip V might be acceptable as a tossup at ACF Nationals.

I'd also like to address a particular misconception that has formed around some tournaments like ACF Nationals and Regionals. It appears that these tournaments are erroneously viewed by some teams as the perfect opportunities to drag out that tossup on minor works of Selma Lagerloff. Please resist the temptation to do this; before writing, you should really do what I suggest above and check to make sure that whatever it is you're going to write about has a decent chance of being answered. Depending on the results of the above procedure, you might do better by making this subject a bonus part or a clue to something more gettable. Many times, an inexperienced player will hear something that he thinks is tossup-worthy in class and rush to write about it, only to stump an entire tournament. Also, it's generally a good idea to avoid things that are too highly specialized. Some specialist material, especially in science questions, is inevitable, but even within the category, no one should have to have a Ph.D. in the subject to answer the question.

Mostly, I've been writing about tossups. As far as bonuses are concerned, a good way of constructing a bonus is the "easy-middle-hard" model which is now pretty much predominant in quality tournaments. The easy part of the bonus should be answerable by pretty much any team at the tournament. The middle part might be answerable by about half the teams, whereas one might need to have really good knowledge to get the third part; probably no more than 20% of teams should reasonably be able to 30 a bonus. Of course, bonuses vary in difficulty for different teams, but in general, a tournament with well-distributed bonus difficulty will have bottom teams converting not much less than (preferrably around) 10 PPB, whereas the best teams could conceivably top 20 PPB, and bonus conversion should be well-correlated with overall place in the tournament.

This brings me to the somewhat peripheral but still important issue of consistency. This is less important for tossups, which contain internal difficulty gradations, than it is for bonuses. If all the science bonuses at a tournament are super hard, you're going to screw some teams really badly. For this reason, following the above bonus writing model is a good way to ensure that you aren't making any one category too hard. Some teams are going to just not know some areas, and that's fine, but if some bonuses are consistently harder than others, that has the potential to really damage teams. In accordance with this principle, the practice of the impossible third bonus part should also be avoided. The possibility of getting 30 points on a bonus should always be there for the talented team; overly difficult bonus parts effectively turn 30 potential points into only 20, which tends to really narrow the gap between teams.

Ok, this is the part where I address some objections to my own advice. There are some people who adhere to the view that one should write about things people are likely to know about instead of things that have come up before. Matt Weiner is one notable proponent of this view. I'd like to note that I don't completely disagree with this, but I am of the opinion that things that have come up in packets before are things that people are likely to know. One might make the argument that this rewards people who read or hear lots of packets, and this is of course true. But I contend that these people will be rewarded anyway, because if you write about what people know, and people play on those packets, the people who played on them will have the same advantage when the same topic comes up, regardless of the reason for why it comes up in quizbowl.

For more information on canon expansion, please refer to Andrew Hart's excellent guidelines on the subject.

Actually Writing the Questions

Ok, with that long preamble out of the way, we can get down to the business of actually writing some questions. I'll try to provide some guidelines for general question writing and also dissect some questions to demonstrate what makes for good and bad tossups and bonuses. Here we go!

Tossups

The first principle of good tossup writing is pyramidality. What is pyramidality, you ask? Pyramidality is simply the principle of putting harder clues earlier in the questions, with the idea being that the more knowledgable team will answer the tossup first because they know more about the subject. I like to think of the pyramid as representing the proportion of teams that would know the answer to the tossup at some particular spot in the question. So the apex of the pyramid represents a specialist in the subject, and the base represents most people who play quizbowl and have heard of the answer. Ideally, there should be a relatively smooth gradation between the apex and the base, although of course real tossups resemble ziggurats (or perhaps mastabas) more than actual pyramids. Nevertheless, the idea is clear: there are gradations of clue difficulty in tossups which is intended to distinguish between more knowledgable and less knowledgable players.

There's no magic formula for writing tossups, unfortunately. If there were, we could have computer write tossups for us and the problem of generating questions would be solved once and for all. While Ray Luo labors to come up with such a machine, I'll just go ahead and describe, in general terms, what the various levels of the tossup pyramid might look like and what kind of clues they might contain. Keep in mind that this is not meant to be an exhaustive list, but I hope it will prove instructive.

The first clue of a question should contain information that is unlikely to be known by anyone who has not spent at least some time studying whatever the answer might be. When I say "studying" I don't necessarily mean studying in school, but rather this word should be read as meaning "absorbing information about," and I write "studying" for short. So, reading a book by or about, say, Herman Melville, qualifies as studying in this sense. If I were determined to learn a lot about Moby Dick I would certainly read the work itself, but I might also read various critical interpretations of it. That kind of clue would be pretty useful to me then; it would give me an advantage over someone who has just read the book but not the criticism. In science, such a clue might reference a little-known application of some effect, for instance. In history, it could be a clue about an academic historian's interpretation of some event. And so on.

I think this is a good point to insert a brief caveat about such first clues. It can be hard sometimes, especially when employing clues of the type "Joe Blow said this-and-that about this work," to decide if a clue will actually be useful to anyone. I address this below in the section entitled "The Usefulness of Clues." Moving on:

The next couple of clues (what I'm going to refer to as "the middle") should successively narrow the answer space down to the actual correct answer of the tossup. This is the part that is usually the hardest to execute, since clues that are really easy or really hard might be obviously so, but the arrangement of middle clues is tough because you're trying to create a relatively fine gradation from the start to the end of the tossup. It's hard to say exactly what those clues should be, but maybe some examples would be instructive. If the question is on a work of literature, (say, the very same Moby Dick), you might describe a somewhat-little-known episode in the book, followed by the actions of some better-known characters, followed by some even better-known names or events. In a science tossup, you might talk about a lesser-known consequence of some effect, then maybe the equation that describes this effect, followed by a better-known consequence. Or something like that.

The key component of clue ordering is to ask yourself, what would a person who knows a lot/a decent amount/not so much/very little (pick one) be likely to know about whatever it is that you're writing about. This requires some intuition in some cases, because unless you've been around the circuit for a while, you probably don't know that certain clues for certain things are considered "stock" and thus shouldn't be anywhere but in the end of a tossup. However, for many situations, you can figure this out; in literature, for example, it should be straightforward to see which characters are minor and which are major, which events are central to the work and which are just vignettes. If you're writing on some idea or theory, the name of its formulator probably belongs in the end, unless it's obscure. Likewise for common definitions (like in a math question), or major scenes in novels, or most famous lines of poetry, or whatever. You should quickly get some sense for what the clue ordering might be like by reading a bunch of tossups; that's not to say that you have to slavishly emulate what came before you (maybe you found a neat new clue or something) but it does mean that if you follow the conventional clue order you will probably write a decent tossup.

The giveaway is self-explanatory. It should contain the most famous bit of information about whatever the answer is. It should usually not, in my opinion, contain tangential clues (the James Webb telescope comes to mind) about what the answer sounds like or anything that requires lateral thinking. Note the emphasis on "usually." Sometimes an oblique reference to another thing that may help someone get the question is fine (mostly I'm thinking of common-link tossups), but most of the time it's not.

Bonuses

There's a lot less to say about writing bonuses than about writing tossups, mostly because the confusion associated with pyramidality is to some extent absent in bonuses. Bonuses should generally follow the "easy-medium-hard" model of bonus parts and the majority of the bonuses you write should be of the 10-10-10 variety. We can loosely define "easy" to mean that we would like 90% of the field to convert this part, "medium" to mean that maybe 40-50% of the field will convert it, and "hard" to mean that perhaps 10-20% of the field should convert it. These are not hard-and-fast numbers; they are just vague signposts to explain what is meant by the difficulty levels. Obviously, these may be adjusted downward or upward for various tournaments, but I think they are generally a good indication of how to break down bonus parts by difficulty.

In writing bonuses, avoid the "impossible third part." A sufficiently comptetent team may not get 30 points on each bonus, but the potential for 30 points should be there. A bonus part on something that no one who is not an expert on some particular subcategory wouldn't even have heard of is a bad idea. It is also bad to have the bonus difficulty differ substantially across categories within a tournament (and especially within a single packet). Such bonuses tend to screw an unfortunate team and can make the difference in close games. Of course, there will be variability between teams, and some categories will be known better than others, but the potential for the structure of the packet (rather than the clue content) to influence the outcome of the game should be minimized.

Style

This may be the most subjective section of all, but I think it's necessary to say something about the stylistic issues inherent in question writing. It should go without saying that packets should be grammatically correct, but they're often not, so please read your questions out loud to yourself to make sure the sentence structure makes sense. We all make mistakes of course, but some tournaments I've been to have featured missing words or spelling errors or grammatical mistakes in almost every question. This is the minimum that can be expected of any writer. I'll say it again: PROOFREAD!

The first clue of a tossup should always be uniquely identifying (to the best of the writer's abilities; sometimes information may match two answers despite the writer's best attempts). Also, the first pronoun should refer unambiguously to the answer and should appear IN THE FIRST SENTENCE. I can't stress this enough! If the first sentence of a tossup finishes and I don't know what category the answer goes in (person, place, or thing) then this question is bad because it doesn't make clear what it's looking for.

More generally, there are different ways to word questions, and different approaches to writing. On one level, you may choose between simple, declaratory sentences (e.g. "This guy wrote about Joe Blow in Work X. Then he wrote about Jane Doe in Work Y.") or compound sentences with clauses and subclauses (e.g. "This guy wrote about Joe Blow in Work X, after which he used his experience in the Pastry War to write about Jane Doe in Work Y."). You get the idea, I'm sure. I personally like the latter style, but some people like the former. I don't think there's anything wrong with either one, provided you are economical about your word choice. Since questions are typically limited in length by editors, you should make sure that all the words you use are meaningful. If a clue can be rewritten with shorter words in place of longer ones, you should probably do that. This saves space and helps you pack more information into a tossup.

Within clues, there are different ways of ordering information. For example, notice how above I used the form "wrote about Joe Blow in Work X," rather than, "His Work X concerns the adventures of Joe Blow." This is deliberate; any random quizbowl player is less likely to know the main character of any given novel than the name of the novel itself. And so on. I recommend using this construction to create pyramidality within clues themselves.

Finally, a word on gender. For a long time, there has been what I think is a nasty trope in quizbowl, which involves writing things like "this writer," or "this scientist," or whatever in an attempt to disguise the fact that the person in question is female. This is annoying, because instead of saying "she," which is shorter, you're now using up more space and you're not even hiding the fact that it's a woman, because everyone now knows to watch for this. So my suggestion is that people use "this [blank]" in the first couple of clues, but then just transition to the male or female pronouns. In most cases, the field of women whatevers is not nearly so narrow as to shrink the potential answer space down to something obvious.

The Usefulness of Clues

This is another problem that I see come up over and over again. Clues get used in tossups that are just not useful to anyone. Egregious examples of these kinds of clues are "Joe Blow notably studied this thing." These clues are unhelpful because unless you know exactly who Joe Blow is (and even then, he might have notably studied several things), you can't possibly get anything out of this. Such a clue could be converted into a useful one by writing, "Joe Blow notably addressed this issue in his tract 'On Stuff'," provided that "On Stuff" is actually about only that one thing. Clues containing numbers in science questions (e.g. "Its albedo is a million," "This quantity is 234.9 for water," etc.) are also useless, because no one memorizes these numbers. Vague clues like "In this novel one character goes to the store for some lettuce," are also not useful since they are probably not uniquely identifying. Clues like "This river is 543 miles long" are also dumb because no one, not even geography wizard Jeff Hoppes, knows these things. In short, if you are tempted to pad a question with the kind of information found in an almanac, don't. Make sure that all your clues convey useful information that someone could actually get the question from.

All this leads me to the culmination of this lengthy opus, which is perhaps the most important part.

What to Avoid, and Why

I've done my best to cover how questions ought to be written; now I'll point to some mistakes people make in writing questions and explain why you shouldn't do these things.

Tossups

Giveaways in the first line are bad. Since pyramidality is good and this makes the question not pyramidal, it is bad. Ok, that was the obvious one.

Stock clues. These are clues that were abundant in old, pre-2002 or so quizbowl, like Michael Faraday being apprenticed to a bookbinder or Saussure writing about Indo-European languages (must be the most reused lead-in ever). These clues are bad because instead of learning anything about the subject at hand, people just memorize clues which end up being recycled by lazy writers. Sometimes, the recycling of these clues by beginning writers is inevitable, but editors should know better. Keep in mind that not ever lead-in is a "stock" clue, since there are plenty of different ways to write a tossup on, say, Herman Melville, and some of them may recycle clues about his lesser-known works. However, clues that come up again and again (the Saussure clue is bad because it's his only other published work, so it gets referenced all the time, becoming "stock"). My suggestion for avoiding this is to search the Stanford Archive for the topic and then see if the lead-in you want to use appears many times in older packets. If it does, don't use it.

Question Transparency. This is a huge one, and I didn't really address it above. I'll do so now. The idea of question transparency is that if you write the question poorly (or if you choose your answer poorly), it will become very obvious what the answer is without the player actually having to know anything. Case in point, picked at random from whatever Moon Pie packet I happened to be looking at: a tossup on Grimm's Law, which mentions something about exceptions to it being described by the spirant law and later in the first sentence mentions voiceless dentals. Now, I know nothing, but nothing about voiceless dentals. All I have is a little radar that beeps "linguistics" when I hear that line. Also, I speak English well so I know that laws have exceptions. What is a law of linguistics? Why, Grimm's law is! This is the classic example of a question that rewards lateral thinking and knowing that an answer vaguely matching the characteristics being described exists. Basically, if someone can "figure out" the answer based on the fact that it becomes obvious (for reasons having to do with poorly placed clues, linguistic hints, whatever) within the first two clues or so (as opposed to because they actually know it from information provided), then the question is bad.

Editorializing and needless verbiage. Everything you write in a tossup should contain information that helps players answer the question. I've seen at least one packet from Moon Pie that contains all sorts of needless editorial content and mountains of words that don't help at all when playing. Any words that don't contain useful clues are just hindrances for the players; moreover, since space is limited, these questions tend to suffer from sharp difficulty drops and create buzzer races. Speaking of which... Buzzer races. If you write a question that contains clues on nothing but Melville's poetry and then a giveaway on Moby Dick you have engineered a stupid and pointless buzzer race, because one either has to be a Melville expert to get this question or wait until the end and hope one is fast enough to beat everyone else who will buzz at the same time. This goes contrary to the idea of pyramidality and having clues which get progressively easier. It also fails to discriminate between two teams with potentially different Melville knowledge, neither of which contains Melville experts.

List tossups. These kinds of questions violate several rules of good question writing, including the one that mandates that the subject of the answer be revealed in the first sentence. This has the added effect that potentially any commonality between the listed things should logically be an acceptable answer. For example, a question that begins "Julius Caesar, John F. Kennedy, William McKinley," with the intended answer being "they were all assasinated" would be terrible anyway, but logically would have the answer "they were all men" be acceptable. This is stupid and wrong, so don't write these questions, as nothing good can come of it. There does not exist a list tossup that cannot be remade into a better question with the same answer just by changing the wording of the question.

Bonuses

List bonuses. These are dumb and uninteresting. These bonuses amount to almanac memorization and tell you nothing interesting about the subject at hand. Authors from works, VSEPR shapes, rulers from dates of rule, whatever. All these are equally dull and should be avoided. The presence of such questions is one of the best indicators of lazy question writing.

Almanac bonuses. These include memorizing Nobel Prize lists, physical constants, and so on. A relative of the list bonus, and bad for all the same reasons.

Wacky bonus forms. The 5-10-15 is deprecated; so is the 30-20-10, which is better as a tossup anyway. The 5-10-15 is bad because it penalizes the team twice: once for not knowing the hard part, and once again for making the hard part worth more. The 10-10-10 is king; you should follow this bonus convention unless you have a good reason to change the value of some of the parts (such as if you have two answers required for one part and you'd like to make them 5 each).

Varying bonus difficulty. Already discussed above, but all your bonuses should aim to be roughly the same in difficulty. They won't be of course; some teams will know some categories and not others, and that's ok. But if all teams are doing poorly on a certain category, then your bonus difficulty is unbalanced. Also, randomly difficult bonus parts in supposedly easy tournaments are also bad. Calculation bonuses. These are bad for the same reason that the 5-10-15 is bad, only more so. These bonuses tend to doubly punish teams that lack a specialist since they require you not to just be somewhat familiar with a topic but to know the actual formula. They also punish specialists who may just not be very fast when it comes to calculating something, or who can't remember the exact formula.

Marzipan, a kick in the balls, both or neither. This is a lousy format because it too punishes a team multiple times, especially if it's used in a science context. These bonuses are annoying because they tend to contain trick questions and encourage guessing. While it's logically possible for such a bonus to not be awful, chances are that if you write one, it will be awful. So don't.

Conclusion

Ok, this has been long for me to write and for you to read, so I'll wrap it up right here. Some closing comments: contrary to what Charlie Steinhice will tell you, standards are not "optional" and while there remain debates about specific things in question writing, most of the core issues really are settled in good tournaments. The outstanding debates are like the outstanding debates about evolution: no one with any expertise in the subject disagrees on the core, though they may debate the specific mechanisms. I hate to make it seem like I'm some sort of quizbowl pope speaking ex cathedra, but it's pretty much true. These issues have been hashed out over the years by experienced players and a lot of thought has gone into making quizbowl questions good. If you are a new player, do not ignore these guidelines and attempt to reinvent the wheel. Rather, take note of these rules, read them once, maybe twice, and then read them again when you feel compelled to write a crappy question.

I welcome everyone's comments on this document, including any suggestions for things I might have missed. I will incorporate good ideas into this original post, which will hopefully be stickied for future reference. If it occurs to me to add something, I'll do so (I plan a little addendum on time management in editing in the future). I also plan to do a dissection of good questions and bad questions somewhere down the line in this thread to give people an idea of what distinguishes one from the other. If other experienced players would care to pitch in with their own ideas, I'm sure that working together we can make this quizbowling world a better place.