Re: Winning versus scoring

• From: tak_seorangpun
• Date: Sun, 18 Feb 2001 02:18:28 -0800
• Message-ID: <000T8V8BK.000V474K9.12170551-quizbowl_at_...>

In message 4085, Eric H., NAQT ICT invitations
coordinator, writes:
> I will grant you that for a team
that was 6-7 to qualify ahead of a
> team that
was 9-4 in the same field is an alarmingly
anomalous
> situation. If it should ever be allowed to
happen, it should happen
> only where the 6-7
team's stats are emphatically better. The
results
> this year in the Mid-Atlantic may be a big red
warning sign that we
> need to alter the weights of
the winning percentage factor to make
> such a
result less likely in future, or otherwise adjust
procedures to
> decrease the likelihood that teams
more than a game apart in results
> *from the
same field* can be selected in reverse order unless
the
> statistical difference is simply
overwhelming.

The core problem is that every SCT provides a ranking
of its participating teams, but this local ranking
is *not* always consistent with the global ranking
computed (later) by NAQT.

NAQT *could* make ICT
invitations in such a way as to *guarantee* that a
lower-ranked team would not be invited before a higher-ranked
team from the same SCT. From the comments above
(proposing merely to "adjust procedures to decrease the
likelihood ... unless the statistical difference is simply
overwhelming"), it appears that NAQT has already decided
explicitly to *reject* this idea out of hand. Why?

A
method that respects local rankings would not have to be
overly complicated. One simple way of picking N teams
for the ICT based on SCT performance is as
follows:
1. Rank all the SCT teams using the current secret
formula.
2. From the top N teams, count how many played in
each SCT (say M1, M2, ...).
3. Invite the top M1
teams from SCT 1, the top M2 teams from SCT 2, etc.,
according to the local rankings from those
SCTs.

> For this year, however, the bottom line is that
we had a system, we
> applied it without bias
to anyone, and this is how it came out. We
can
> look at the results and say "you know, we really
need to change
> something here for the future."
We could not look at the results the
> system
gave us and say, "you know, we don't like how this
came out
> this year, so we're going to change
this now retroactively to get a
> different
result."

I agree, but this is yet another reason why the
secret formula should be made public. One of the great
advantages of open source software is that bugs can be found
and corrected before they cause damage.

Also,
if NAQT had given out the secret formula, SCT hosts
could have used it to compute rankings of the
participating teams. Players on globally lower ranked teams
would not get such a nasty shock to see locally lower
ranked teams invited while they were
not.

Thirdly, if players know the secret formula, then they can
use it to form strategies to optimize their chances
of getting an ICT invitation. This is not a bad
thing.

I can understand why NAQT chooses not to reveal
some things about its operations -- such as how its
questions are written and edited, or even its subject
distribution (anyone can figure this out easily) -- but I do
not understand the rationale for keeping the
statistical formula secret.