A Level Playing Field

Five Reviews

Once you have submitted your idea, five members of the Evaluation Panel will be randomly assigned to score your submission. They will enter a single score and comment for each of four distinct traits. If you are chosen as one of the finalists, those scores and comments will be provided to the Selection Committee, who will decide which idea to develop further (the author of the top idea will receive $10,000). However, in order to rank all of the ideas, scores from the Evaluation Panel members are calculated in the most fair and transparent way. You are welcome to “test drive” the actual scoring tool used by the Evaluation Panel members.

Here is an explanation for how those scores are calculated and the way in which we seek to level the playing field for every participant:

Scoring Details

Each of the four traits will be scored on a 0-5 scale (in increments of 0.1). Examples of possible scores for a trait are: 0.5, 3.5, 5.0.

The most straightforward way to ensure that everyone is treated fairly would be to have the same Evaluation Panel members score every idea; unfortunately, due to the number of ideas, that is not possible—it would take too long and require too much input from each judge.

Since we are not having the same Evaluation Panel members score every idea, the question of fairness needs to be carefully explained. It makes sense that one Evaluation Panel member scoring an idea may be very harsh, giving everyone a 1.0, while another Evaluation Panel member may be very easy, giving everyone a 5.0.

Let’s look at the scores from two hypothetical Evaluation Panel members:

The first Evaluation Panel member is a lot more lenient than the second, who gives much lower scores. If your idea was scored by the first Evaluation Panel member, it would have a much higher total score than if it was assigned to the second.

To fix this issue, we utilize a mathematical technique relying on two measures of distribution, the mean and the standard deviation.

The mean takes all the scores assigned by an Evaluation Panel member, adds them and divides them by the number of scores assigned, giving us an average score. So, if an Evaluation Panel member is easy, he will have a much higher average score than a harsh one.

Formally, we denote the mean like this:

The standard deviation measures the “spread” of an Evaluation Panel member’s scores. So, maybe two of them both give the same mean (average) score, but one assigns a lot of zeros and fives, while the other gives a lot of ones and fours. We can see how it wouldn’t be fair to you if we didn’t consider this difference.

Formally, we denote standard deviation like this:

To ensure that the scoring process is fair, we rescale all the scores to match the population of Evaluation Panel members. In order to do this, we measure the mean and the standard deviation of all scores across all of the Evaluation Panel members. Then, we change the mean score and the standard deviation of each Evaluation Panel member to match.

We rescale standard deviation like this:

Then, we rescale mean like this:

Basically, we are finding the difference between both the distributions for an Evaluation Panel member and those for all of the Evaluation Panel members combined, then adjusting each score so that no one is treated unfairly according to which Evaluation Panel members they are assigned.

If we apply this rescaling process to the two Evaluation Panel members in the example above, we can see the outcome of the final resolved scores; they appear more similar, because they are now aligned with typical distributions across the total population of Evaluation Panel members.