Letter to Questions Galore

The following letter was sent by Greg Gauthier to Questions Galore on Wednesday, May 27, 2009. This letter could be considered a criticism of Questions Galore, and the principles the letter posits can be used to argue against other bad quizbowl question writers.

Apart from the addition of wikilinks, the letter is reprinted here using the exact words of the original document.

Introduction

I am Gregory Gauthier, a former Wheaton North Scholastic Bowl player and current player at Vanderbilt in the collegiate circuit. I am also a member of the High School Quizbowl Forum,¹ posting under the screen name rjaguar3. Since I began playing in 2004, I have been exposed to tens of thousands of questions. Some of them were good academically-based questions that rewarded knowledge and learning both in school and outside of school; others were not.

When I got word that the 2008 Wheaton North Frosh/Soph tournament had received a very negative reception, related to the choice of question provider, I was worried for my high school alma mater. After hearing at least one coach promise never to return with his team again if the question provider were unchanged,² I tried to ensure that the tournament would better suit the needs of the teams there. Ultimately, I offered to edit a set of Questions Galore questions. After receiving the set of questions (JV sets 11-19 for the 2008-09 season), partially edited by Coach Kidd, one week before the tournament began, I was utterly shocked at how far below the standards of quizbowl the questions I received were. I did end up replacing about twenty questions with questions I wrote or had previously written but never used, and rewrote about fifty more questions to meet the community’s standards. Half of the questions, however, remained unchanged. Upon completing the editing, I was disappointed that I could not have done a better job, and disappointed that Questions Galore produced a set where more than 80% of the toss-ups failed to meet the quizbowl community standards. This was completely unacceptable; as such, Coach Kidd has agreed to let me and two of my former Wheaton North teammates write next year’s frosh/soph tournament.

The purpose of this letter is to summarize what good questions are and how to write them, and then to analyze questions that appeared in the unedited set and see what problems they had and how I tried to fix those problems.

What Is a Clue?

Quite simply defined, a clue is a fact that points directly to the answer of the toss-up and explains why it is academically significant or worth knowing. A good clue must also meet the following standards:(3)

1. Factual correctness—the clue must actually be a fact, not a riddle or a puzzle based on hypothetical circumstances. Additionally, clues and questions should be sourced so that the answer on the paper is the correct answer. Several of the questions I edited had the wrong answer, including stating that the Colossus of Rhodes depicts Apollo when it actually depicts Helios (2T13), giving the wrong number of rings for Uranus (1B11c), claiming that the Titanic was designated a steamship, rather than a Royal Mail Ship (3T2), and describing the French horn as "the bass instrument [with] the greatest range" (1B7a). There is no reason why questions and clues should not be meticulously fact-checked to ensure accuracy and reduce the number of angry players and protests.
2. Specificity/uniqueness—a clue, especially a lead-in, must point to one and only one answer. Vague clues, like "The title place never has a specific name, throughout the main portion of the novel only has one inhabitant, and at the end of the novel has many inhabitants but no longer has its original inhabitant,"(4) are not helpful to players. Consider using the mantra "names rule": a name is inherently more interesting, more specific, and almost certainly more-uniquely identifying than a slew of true statements that do not point players to the right answer. Hard clues should be hard because the names are less recognizable, not because the names are completely removed.
3. Ability to stand alone—each clue should point to one and only one answer, or otherwise mention any alternative answers before it is read (for example, "Like gold, this metal dissolves in aqua regia").
4. Relevance—this ties in directly to what a clue is. A clue should not consist of biographical platitudes such as a person’s birthdate, birthplace, educational institutions attended, people whom that person studied under, parents’ occupations, or any biographical details that are boring and not helpful. Instead, write clues about what the person did and why he or she is important. Also, avoid quotations (except for short quotations from literature), as there is no reason that a quotation cannot be rewritten better and more densely (with more names and pieces of information) as a more refined clue.
5. Interestingness—clues should be interesting to play on. See #4 for more details.
6. Non-transparency—a clue must not be transparent. A transparent clue is one that allows players to buzz based on logic, reasoning, and common sense, rather than on a concrete fact. An example would be the first clue of my edited Frederic Remington question: "One of his works appeared on the cover of Harpers Weekly, and was titled The Apache War—Indian Scouts on Geronimo’s Trail." After this first sentence, a player may not be familiar with the painting, but that player can make the logical connection that this painting must be a painting of the West (from the words "Apache" and "Geronimo"), and the only well-known American painter of the West is Frederic Remington. Another example comes from Missouri’s Lexington Tournament: "This Mexican painter of the mid-20th century painted her own reality,"(5) which lets out of the bag that the answer is the only female Mexican painter likely to come up in high school quizbowl, Frida Kahlo.

Avoid CLOOS

CLOOS "are factual statements that do not help teams confidently buzz without having to guess.(6) CLOOS are named for the fact that they are superficially similar to clues, but fail to help teams buzz. In the article, I describe three varieties of CLOOS:(7)

1. Style CLOOS "that are broad, general, nonspecific, and vague," such as art style ("This painter was known for depicting beautiful landscapes with large brushstrokes"), literary criticism ("This play is about how love can be deadly"), reception/fame clues ("This Nickelodeon show was popular in the 1990s"), and chemical element non-clues ("It is a silvery metal that is essential for humans but deadly in large quantities"). Note that those clues are not necessarily bad provided that they are paired with specific names that make the clue point to one answer.
2. Almanac CLOOS based on figures that an expert would find not to be worth knowing, like country statistics ("This country's GDP in 2008 was $64 billion"), scientific constants ("It melts at 1153 degrees Celsius"), birthdates ("Born July 22, 1940"), and numbers related to geographic features ("This lake has a surface area of almost 193 square miles")
3. Lecture CLOOS, clues that merely talk or ramble about something without giving any indication of what type of answer is sought. The best example comes from the Lexington tournament, where the first toss-up of the tournament read as follows:

   Alexander the Great is considered one of the greatest military geniuses of all times. Although he inherited an empire that included most of the Greek city-states, he and his men went on to conquer most of the known world of their day. He was the first king to be named "the Great". In which conquered country did Alexander order a city to be designed and founded in his name at the mouth of a river?

   ANSWER: Egypt

Here, the bolded text gives no indication of what type of answer is being sought. Instead it merely lists true statements about Alexander the Great.(8) Someone who knows a lot about Alexander the Great’s career has no better chance of answering the question than someone who happens to know where the most famous Alexandria is located.

One way to tell if you have written a tangent is to read the question aloud to someone else. If the listener can reasonably buzz in after the first sentence and say "true," then you’ve probably written a tangent. A good question states what type of answer it is after, and avoids changing directions or adding additional specifications later in the question.

Weiner’s Law #1

Matt Weiner, a noted player on the quizbowl circuit, has written an accepted law stating that question writers are "neither cute nor clever."(9) Statements like "Let’s see how much you know about circles" (8T4), "What a resume!" (5T1), and "Don’t be a couch potato!" (1BR1) are not cute, not funny, and not helpful to players. Similarly, topics like spelling words with chemical element symbols (9B12) and solving number riddles (4B7) are also neither cute nor clever. Please avoid writing on such subjects again.

Numberwang!

Quizbowl is an exercise designed to reward academic pursuits and knowledge of material found in Advanced Placement and early collegiate classes, not as a "remedial elementary-school exercise."(10) Yet Questions Galore has written several computational math questions that require no more mathematical knowledge than a fourth grader would know, or that primarily tests fourth-grade arithmetic skills rather than advanced mathematical knowledge. I have called such questions Numberwang!(11) Examples include "What is ¼ of ½ of 122?" (3TR1), "Evaluate the expression 3x (y + 10) if x = 2y – 4, and y = 9" (2T2, which rewards players who can quickly multiply 42 by 19 rather than players with actual algebra knowledge) and "What is the surface area of an orange that has a radius of four centimeters?" (7T14, where the answer given is 200.96, primarily testing how quickly someone can multiply by 3.14 rather than actual geometric knowledge). While I (and many others in the quizbowl community) believe that math computation questions are bad, if you are going to write them, please try to avoid turning the question into a number-crunching race, as that does not demonstrate real mathematical knowledge.

A Note on Pyramidality

In recent years, there has been a lot of discussion about pyramidality and what it is. Pyramidality is a concept that toss-ups should reward players who have more knowledge of the answer. To that end, a pyramidal question consists of clues, factual statements pertaining to the importance or significance of the answer, arranged in descending difficulty order. As such, the structure of a toss-up is as follows:

1. Lead-in, the least well-known and most difficult clue about the subject. Despite being not very well-known, an expert on the subject should be able to buzz in with the right answer from this clue. It should not be so obscure that not even the best player has a chance to buzz after the lead-in with the correct answer. Additionally, the lead-in must be uniquely identifying: it must explicitly state what type of answer is sought, and after the lead-in is read, there should be only one possible answer.
2. Middle clues, more well-known clues about the subject, arranged in descending difficulty level, that sort teams based on their knowledge of the answer.
3. Giveaway, the most well-known clue pertaining to the subject. In particular, "sounds like," pop culture, and other word based giveaways should be avoided in favor of the most well known clue about the significance of the toss-up subject. When asking about a creative work, it is acceptable and expected to put the creator’s name at the end of the toss-up, even though it is not uniquely identifying because it is the most-well known clue and the rest of the toss-up talks about one and only one of that creator’s works.

Questions Galore Critique

Questions Galore 7T9:

In 1924, this musician composed "Rhapsody in Blue," an early concerto for piano and the orchestra that contained elements of jazz. What was the full name of this American composer?

ANSWER: George Gershwin

My edits:

This composer wrote the music for songs titled "It Ain’t Necessarily So" and Summertime." He won a Pulitzer Prize for Drama for scoring Of Thee I Sing. In addition to Porgy and Bess, he composed "An American in Paris" and a 1924 piece that begins with a clarinet glissando. Name this composer of "Rhapsody in Blue."

ANSWER: George Gershwin (prompt Gershwin)

Problems with original question:

• "Rhapsody in Blue" is the most well-known clue about G. Gershwin. There is no reason why it should be the first buzzable clue in the question. Putting "Rhapsody" in the lead-in does nothing to distinguish teams that have a player familiar with the works of G. Gershwin from teams that memorize titles like "Rhapsody in Blue."
• As written, the question could constitute a hose with the constraint placed at the end of the question: "What was the full name of this American composer?" Had the full name been underlined, teams that buzzed in on the giveaway clue in the first line would have been punished because they cannot read the mind of the question writer. Hence, such a hose penalizes teams that have more knowledge. Teams should be able to buzz with confidence once they hear a clue they know, certain that the question will not disqualify their answer for a reason yet to be read. Fortunately, Ms. Kidd did her job and only underlined Gershwin.
• "Rhapsody in Blue" is the only clue in the question. The other stuff is vague filler. This is no better than a 1-line question.

What I tried to fix:

• Added more clues to the question. According to my count, there are now 7 clues (“It Ain’t Necessarily So,” "Summertime," Of Thee I Sing, Porgy and Bess, "An American in Paris," piece with clarinet glissando, "Rhapsody in Blue"). This allows for teams that know more about G. Gershwin to buzz before teams that know less, provided that…
• Clues are arranged in descending difficulty order. Note that G. Gershwin’s most famous work is at the end of the question. Less well-known works appear in the middle of the question, while the lead-in contains obscure titles that an expert on G. Gershwin’s music or music in general would be able to get.
• The underlining of the question requires the first initial (as Ira Gershwin was also a famous American composer) but it has a prompt, allowing for teams that buzz in early and say Gershwin to get an opportunity to demonstrate exact knowledge. Note that I did not include "First and last name needed at the beginning," as that makes the question transparent (it broadcasts that the answer will have a shared last name), but that is also an acceptable practice, just as long as it precedes the question so that teams know well what they are expected to give.

Questions Galore 7T15:

This novel has been called "The Great American Novel." It explores the efforts of one man to accomplish the American Dream—success, money, social standing—in attempt to win a woman’s heart. What is this F. Scott Fitzgerald novel that takes in place in New York during the Jazz Age?

ANSWER: The Great Gatsby

My edits:

A minor character in this work include a man who is surprised that one person's books are real, Owl Eyes. The death of the title character results from George Wilson’s anger that Myrtle was run over. The title character is known for his lavish parties in West Egg, although he was born in North Dakota under a different name. Name this novel narrated by Nick Carraway about James Gatz’s desire for Daisy Buchannan, a work by F. Scott Fitzgerald.

ANSWER: The Great Gatsby

Problems with original question:

• What shocked me when I read this question for the first time was that this question on one of the most famous works of American literature did not mention any characters or plot events in that work. Thus players who have actually read the book have no advantage over list-memorizers who may not know as much literature.
• The first clue is not uniquely identifying. Ask ten literary professors what the "Great American Novel" is, and you’ll get ten different answers.(12)
• The other clues are also vague and probably not unique. Show, don’t tell: use actual plot events and character names; those are uniquely identifying and helpful to players, unlike filler text such as "It explores the efforts of one man to accomplish the American Dream."
• Although it is good not to put "F. Scott Fitzgerald" at the beginning of the question, the author’s name does not belong anywhere before the very end of the question. Ninety-nine percent of the time, the author is the most famous clue for a work, and therefore it should be at the end of a pyramidal question.

What I tried to fix:

• I added more clues using characters and plot events (“surprise that one person’s books are real,” Owl Eyes, George Wilson, Myrtle, character dies by being run over, West Egg, protagonist born in N.Dak., Nick Carraway, James Gatz, Daisy Buchannan). People who’ve read the book (like me) would rightly be able to answer this question before people who’ve read plot summaries, who would answer before someone who only knows that F. Scott Fitzgerald wrote The Great Gatsby. Knowledge and scholastic excellence is rewarded with a clue-filled question like this one.
• Clues are arranged in descending difficulty order. In particular, the author’s name is at the end of the question.
• There are no vague or opinionated clues. All the clues are hard facts from the novel that lead to The Great Gatsby and only The Great Gatsby''.

Questions Galore 6T10:

This American artist made solitude and introspection a vital theme in his paintings of American landscapes. Loneliness, emptiness, and stagnation of town life are seen in this realist’s work. He was an intensely private individual and once declared, "I don't think I ever tried to paint the American scene; I'm trying to paint myself." Name the artist who painted New York Movie, Office in a Small City, People in the Sun and Nighthawks.

ANSWER: Edward Hopper

My edits:

In one of this artist’s works, he painted a row of buildings on Seventh Avenue. In another of his paintings, three people sit silently in a diner in Greenwich Village. Name the artist who painted New York Movie, Office in a Small City, People in the Sun, Early Sunday Morning and Nighthawks.

ANSWER: Edward Hopper

Problems with original question:

• This question contains phrasing and structure very similar to a source on the Internet without citing that source.(13) This constitutes plagiarism, and this is absolutely unacceptable, especially for a competition that rewards academic achievement.
  • There was also another question that was a clear-cut case of plagiarism. A question on Nathan Hale reads

    Connecticut's State Hero was part of a literary fraternity at Yale that debated topics in astronomy, mathematics, literature, and the ethics, which may account for his eloquent speech before being executed in which he stated, "I only regret that I have but one life to give my country." Name him.

    ANSWER: Nathan Hale

  while the Wikipedia article on Nathan Hale states that

  The Hale brothers belonged to the Yale literary fraternity, Linonia, which debated topics in astronomy, mathematics, literature, and the ethics of slavery.

  Again, plagiarism of any form is absolutely unacceptable.

• Aside from the titles at the end of the question, there are no useful clues. Everything before the giveaway consists of style CLOOS that do not help even an art expert.
• I fully expect that if the original question were read to teams, the question would turn into a buzzer race in at least two thirds of the rooms. There is a real difficulty cliff(14) with Nighthawks; a lot of players will buzz there. To alleviate that problem, middle clues that describe the work are helpful.

What I tried to fix:

• I removed the vague clues, as they simply waste space and time.
• I added clues about the actual paintings, so that someone familiar with Hopper’s work will get a question before someone who memorizes titles. Also, note that the descriptions precede titles, as descriptions are less well-known than titles. Finally, I added Early Sunday Morning to the list of titles, as a player who knows more than Nighthawks is probably going to know that title as well, alleviating the difficulty cliff.

Questions Galore 4T3:

Gas cools and becomes a liquid. During this physical change, the energy given up by the gas is transferred to its surroundings as heat. What kind of physical change is being described here?

ANSWER: exothermic reaction

My edits:

This word describes any reaction that results in a system’s enthalpy decreasing. Examples of this type of reaction include the mixture of an acid and a base and combustion. What term describes reactions that heat their surroundings?

ANSWER: exothermic

Problems with original question:

• This question is a hose. Not until the last sentence does the question state what type of answer it seeks. Additionally, the first sentence strongly implies that the answer will be "condensation," but players who buzz in and give the term for the defined reaction will find themselves unpleasantly surprised that the question is turning a completely different direction. This is bad quizbowl.
• Once the type of answer sought is known, the question boils down to (no pun intended) a one-liner. A long question does no good if players cannot confidently buzz upon hearing a useful clue.
• There are multiple answers to the question. As I read it, "condensation" would be a perfectly correct answer, as would "exergonic reaction" or "spontaneous reaction." The question should include all possible alternative answers or be rewritten so that there is only one correct answer.

Notes about Answer Selection

Some of the answer selection in the tournament set was wacky. I can classify this into two groups:

1. Illiterature

   This refers to any literature question that asks about a work that is marginally or not at all considered academically important literature. Here are the examples I found:

   • The Bomb by Theodore Taylor (2T3)
   • Stephen King and John Grisham (6B1ac)
   • Know Your Power: A Message to America’s Daughters by Nancy Pelosi (8T8)
   • Harry Potter transportation (9B5)
     • This is even more annoying because it appeared in the championship round.

   Note that these questions are acceptable if they are classified as pop culture. Asking questions about non-literature works in the literature distribution jobs the players out of a question on an academically important work of literature.

2. Obscure

   We all have our pet subjects. However, over the years, quizbowl has developed a canon of subjects that players are expected to know. While the canon does expand over time, trying to force the issue by putting impossible answers in tossups effectively shortens the game (because the toss-up will predictably go dead in every room), meaning that a perfectly-deserving bonus will go unread and that there will be less opportunity for a better team to demonstrate superiority over a less knowledgeable team.

   The following list is compiled from my five years of quizbowl experience.

   • Madam C. J. Walker (2T15)—no results found in any high school or collegiate packets
   • Richard Blackwell (4T5)—appears once in a bonus stem in Chicago A’s packet for Penn Bowl 1996, but never as an answer

Notes about Question Recycling

Through my personal experience, I have discovered that questions written in Questions Galore packets dating back to the 2006-07 season were reused on WNAB Nashville’s quizbowl show, Westfield Insurance Quizbusters.(15) Question recycling is only acceptable under the limited provision that it is stated outright and that no one who could have heard recycled questions would be able to attend a tournament where recycled questions are being used without knowingly committing a violation. As of now, Questions Galore makes no attempt to my knowledge to ensure this; hence, the question recycling is unacceptable and a betrayal of the company’s promise that "all sets are changed on a yearly basis."(16)

Footnotes

1. http://www.hsquizbowl.org/forums/
2. Matt Laird of Loyola Academy
3. The list is adapted from "Clue." Quizbowl Wiki. 28 Dec 2007. 26 May 2009 [1]
4. Michael Sacks. Tossup 5 from Round 2 of the 2009 Homewood-Flossmoor Varsity Tournament.
5. Round 6, Tossup 31. Hereafter, the notation 6T31 will indicate this. Bonuses will be denoted rBn, with letters referring to individual bonus parts where necessary.
6. Gregory Gauthier. "CLOOS." Quizbowl Wiki. 8 Apr 2009. 26 May 2009 [2].
7. The remaining text in numbered sections 1 and 2 is taken from the previously cited article.
8. Actually, the statement that Alexander was the first king to be named “the Great” is false, as Sargon the Great lived almost 2000 years earlier, and Persian king Cyrus the Great used the title two centuries before.
9. "Weiner’s Laws" Quizbowl Wiki. 21 Mar 2009. 26 May 2009 [3].
10. Matt Weiner. "2009 MSHSAA Sectional/State Predictions." 10 May 2009. 26 May 2009 [4].
11. This is a reference to a sketch on the British show That Mitchell and Webb Look, where it is the title of a parody game show where contestants yell out random numbers and the host occasionally jumps in, saying "That’s Numberwang!"
12. See http://www.amazon.com/Contenders-for-Great-American-Novel/lm/33H3DJF07GCDY, http://www.esquire.com/fiction/book-review/great-american-novel-0309 and http://www.randomhouse.com/modernlibrary/100bestnovels.html for examples of the diversity of opinion about what is considered the "Great American Novel."
13. "Hopper, Edward." WebMuseum Paris. 14 July 2002. Retrieved 25 May 2009 [5]. The site declares that Hopper "gained widespread recognition as a central exponent of American Scene painting, expressing the loneliness, vacuity, and stagnation of town life. Yet Hopper remained always an individualist: 'I don't think I ever tried to paint the American scene; I'm trying to paint myself.' " (emphasis added to illustrate blatant similarities between text and question.
14. A difficulty cliff is a place where a toss-up suddenly becomes much easier. In general, if most of players will have heard of the clue after the difficulty cliff but none of the preceding clues, then there is probably a difficulty cliff, which will likely lead to a load of buzzer races if not corrected.
15. Two examples of recycling from the 2006-07 set include a question on the Rosetta Stone that began "What tablet written in three languages" and a toss-up about the moon being the body illuminated by earth shine.
16. "Home." Questions Galore. 2009. 25 May 2009 [6].

External links

Original thread on the hsquizbowl forums