Revisiting The ISL Scoring System: An Analysis of Jackpot Points (Part 3)

Courtesy: Steve Gambino.

This is the third of a four-part series taking a look at the current ISL scoring system and the impact made by jackpot points.

I think most people who’ve swum a 200 would agree, the third fifty is usually the toughest. In part one, we got off the blocks with some easy speed, commenting on the mechanistic issues with the jackpot system: complex, gimmicky, misnamed, and out of sync with what ought to be valued. In part two, we picked up our tempo a bit, getting into the data and exploring how the distribution of talent in a given event determines the jackpot margin’s impact in that event, and the potential discrepancies in value that this can lead to between events and between genders. As we flip turned to finish out part two, we pondered whether finding more optimal values for the jackpot margins could salvage the system. Now let’s dig deep, go to our legs, and build to a strong finish.

Exploring the Jackpot Margins – Is there an ideal margin?

As far as I know, the method for computing the jackpot margin has not been announced publicly, but I’ve observed that most jackpot margins are about 4% (rounded)* of the Short Course Meters (SCM) World Record for each event (or possibly the league records since they’re often the same or similar). And the specific margin chosen is surprisingly significant, particularly if we are to levy criticism toward the league about an unimpactful scoring rule. In fact, the criticism of its lack of impact may not be quite as justified as it previously seemed. I’ll explain in more detail, but first let’s get our bearings.

*Note: They seem to round the margin to a precision of 0.05 seconds. Considering 4%, sometimes this would need to be rounded up to match their official margins and sometimes it would be rounded down. For all of my computations going forward, I used the ceiling function for simplicity. This is to say 1.11 seconds rounds up to 1.15. Additionally, I’ll compute all margins as the given percentage of the SCM WR.  

First, we need to recognize that there exists a jackpot margin sufficiently large where no jackpot points are to be earned, effectively equivalent to not having a margin at all. For the 2020 season, the event with the largest margin of victory (relative to the SCM WR) is the Women’s 200 BR from Match 6, where Kelsey Wog beat Haley Black by 25.35 seconds. This is about 18.8% of the SCM WR of 2:14.57.

Table 5: 2020 Match 6 Women’s 200 BR Results
Rank Name Time
1 WOG Kelsey 2:17.13
2 ESCOBEDO Emily 2:17.77
3 RENSHAW Molly 2:17.81
4 ULYETT Jocelyn 2:18.96
5 CARRARO Martina 2:21.84
6 GUNES Viktoriya 2:22.79
7 LARSON Breeja 2:23.46
8 BLACK Haley 2:42.48

So, any jackpot margin greater than 18.8% wouldn’t yield any stolen jackpot points for the entire season (except for DQ’s or DNF’s, which are independent of time anyway). In other words, if you make the window large enough, every swim will make it through, and no jackpot bonus will be awarded.

Additionally, there exists a jackpot margin sufficiently small such that the winner steals all of the points from every other competitor (except in the instance of first place ties. Note that ties for 1st actually occur more than one might expect – a total of 18 times across the first three seasons). A margin of less than one hundredth of a second is guaranteed to suffice here, so let’s say 0.009 seconds.

You might expect that as you shift the jackpot margins from sufficiently large enough (for no jackpot bonus) to sufficiently small enough (for all points to be awarded as a jackpot bonus for every event), the amount of impact on the league standings would progressively increase. More specifically, you might guess that it would increase monotonically (never decreasing). This seems to make intuitive sense – the more jackpot points stolen, the more impact, right? However, this is not the case, and for me, this is where the story gets really curious.

Let’s explore the effects of changing the jackpot margin for the 2020 season. Starting at the sufficiently large end, 18.8%, and progressively adjusting the jackpot bonus by -0.1% at a time, we can see how and when the match results become affected. Table 6 summarizes all changes from 18.7% to 12.6% omitting all steps where nothing new occurred and gives a sense of when some of the events with the largest relative margins of victory began to be impacted by jackpot bonuses.

Table 6: Summary of Changes to Match Result by Jackpot Margin 18.7%-12.6% (as percentage of SCM WR)
% of SCM WR Match Results Changed Notes
18.7 0 Match 6: W 200 BR has first jackpot bonus
17.1 0 Match 10: W 100 IM has jackpot bonus
14.2 0 Match 12: W 100 BK, Match 10 W 200 IM
14 0 Match 3: W 100 BR has jackpot bonus
12.6 0 Match 8: M 50 BK has jackpot bonus

You can see that, not surprisingly, there isn’t much interesting happening when the margins are very large. While margins this large may impact some individual events, they do not impact the overall match results. Continuing to reduce the jackpot margin, it isn’t until we drop to 5% that we start to see some effect on the actual outcomes of a match – detailed in Table 7. At this point, the result of Match 9 would change – Iron would have overtaken Tokyo.

Table 7: Changes to Match Result by Jackpot Margin 5.0%-4.6% (as percentage of SCM WR)
% of SCM WR Results Changed Notes
5 2 Match 9: IRO and TOK swap 2nd/3rd
4.9 2 Match 9: IRO and TOK swap 2nd/3rd
4.8 0 All events have jackpot bonuses
4.7 1 Match 2: AQC moves up from 4th to tie for 3rd w/ DCT
4.6 1 Match 2: AQC moves up from 4th to tie for 3rd w/ DCT
4.5 0
4.4 1 Match 2: AQC moves up from 4th to tie for 3rd w/ DCT
4.3 4 Match 2: AQC and DCT swap 3rd/4th

Match 9: IRO and TOK swap 2nd/3rd

4.2 2 Match 2: AQC and DCT swap 3rd/4th
4.1 2 Match 2: AQC and DCT swap 3rd/4th
4 (approx. official) 0

This remains true at a margin of 4.9%, but the results revert back to the original standings at 4.8%. We then see a small change to Match 2, where Aqua bumps up to tie for third with DC if the margin is set to 4.7%.

These are pretty small overall changes, but it is interesting to note that these changes occur when the hypothetical margin is greater than the official values (~4%), where we see no impact. So, the jackpot system can have an impact, it just isn’t at the specific margins that have been selected. Further, it isn’t just that the margins aren’t sensitive enough to be relevant either, since these listed above are all greater. It’s also not true that the margins are too sensitive, because if we continue to explore how decreasing them affects the results, we’ll see impact return and stay after 3.5%. Naturally, this leads to the question of what is actually occurring to cause the jackpot bonus to impact the match results. I will address this in the next section. For now, the full list of changes is outlined in the Table 8:

Table 8: Changes to Match Result by Jackpot Margin 4.5%-0.1% (as percentage of SCM WR)
% of SCM WR Results Changed Notes
3.5 2 Match 9: IRO and TOK swap 2nd/3rd
3.4 7 Match 2: AQC moves up from 4th to tie for 3rd w/ DCT

Match 6: LON and LAC swap 1st/2nd

Match 8: LON and CAC swap 1st/2nd

Match 9: IRO and TOK swap 2nd/3rd

3.3 4 Match 2: AQC and DCT swap 3rd/4th

Match 8: LON and CAC swap 1st/2nd

3.2 2 Match 2: AQC and DCT swap 3rd/4th
3.1 4 Match 2: AQC and DCT swap 3rd/4th

Match 8: LON and CAC swap 1st/2nd

3 2 Match 2: AQC and DCT swap 3rd/4th
2.9 2 Match 2: AQC and DCT swap 3rd/4th
2.8 2 Match 2: AQC and DCT swap 3rd/4th
2.7 2 Match 2: AQC and DCT swap 3rd/4th
2.6 2 Match 2: AQC and DCT swap 3rd/4th
2.5 2 Match 2: AQC and DCT swap 3rd/4th
2.4 2 Match 2: AQC and DCT swap 3rd/4th
2.3 2 Match 2: AQC and DCT swap 3rd/4th
2.2 2 Match 2: AQC and DCT swap 3rd/4th
2.1 3 Match 2: AQC and DCT swap 3rd/4th

Match 6: AQC moves up from 4th to tie for 3rd w/ NYB

2 4 Match 2: AQC and DCT swap 3rd/4th

Match 6: AQC and NYB swap 3rd/4th

1.9 2 Match 2: AQC and DCT swap 3rd/4th
1.8 2 Match 2: AQC and DCT swap 3rd/4th
1.7 2 Match 2: AQC and DCT swap 3rd/4th
1.6 4 Match 2: AQC and DCT swap 3rd/4th

Match 6: AQC and NYB swap 3rd/4th

1.5 4 Match 2: AQC and DCT swap 3rd/4th

Match 6: AQC and NYB swap 3rd/4th

1.4 2 Match 2: AQC and DCT swap 3rd/4th
1.3 4 Match 2: AQC and DCT swap 3rd/4th

Match 6: AQC and NYB swap 3rd/4th

1.2 2 Match 2: AQC and DCT swap 3rd/4th
1.1 4 Match 2: AQC and DCT swap 3rd/4th

Match 6: AQC and NYB swap 3rd/4th

1 6 Match 2: AQC and DCT swap 3rd/4th

Match 5: LAC and LON swap 1st/2nd

Match 6: AQC and NYB swap 3rd/4th

0.9 6 Match 2: AQC and DCT swap 3rd/4th

Match 5: LAC and LON swap 1st/2nd

Match 6: AQC and NYB swap 3rd/4th

0.8 6 Match 2: AQC and DCT swap 3rd/4th

Match 5: LAC and LON swap 1st/2nd

Match 6: AQC and NYB swap 3rd/4th

0.7 4 Match 2: AQC and DCT swap 3rd/4th

Match 5: LAC and LON swap 1st/2nd

0.6 4 Match 2: AQC and DCT swap 3rd/4th

Match 5: LAC and LON swap 1st/2nd

0.5 6 Match 2: AQC and DCT swap 3rd/4th

Match 5: LAC and LON swap 1st/2nd

Match 6: AQC and NYB swap 3rd/4th

0.4 6 Match 2: AQC and DCT swap 3rd/4th

Match 5: LAC and LON swap 1st/2nd

Match 6: AQC and NYB swap 3rd/4th

0.3 6 Match 2: AQC and DCT swap 3rd/4th

Match 5: LAC and LON swap 1st/2nd

Match 6: AQC and NYB swap 3rd/4th

0.2 4 Match 5: LAC and LON swap 1st/2nd

Match 6: AQC and NYB swap 3rd/4th

0.1 4 Match 5: LAC and LON swap 1st/2nd

Match 6: AQC and NYB swap 3rd/4th

To summarize the main conclusions so far:

  • The problem isn’t just that the jackpot margin isn’t sensitive enough – we see this because margins greater than the official, like 5%, cause an impact on the match results.
  • The jackpot margin isn’t too sensitive either – we see this because margins less than the official, like 3%, cause an impact on the match results.
  • The impact on match results does not increase monotonically as the jackpot margin decreases – if it did, we would see the number of results changed only increase or remain the same, never decrease. Instead, there are points where it jumps up and points where it drops down without a clear pattern.

Overall, the current jackpot margins have no impact on the results, but that’s not to say a jackpot system can’t or shouldn’t in general. Nudge those values a little higher or a little lower, and match outcomes may just change a bit accordingly.

So, what is so special about a 5% margin in Match 9, and Iron and Tokyo, that causes this to be the earliest (highest) place where jackpots are meaningful?

Match 9, Iron and Tokyo, and Understanding the Change at a 5% Margin

With a jackpot margin at 5.1%, the standings (and points earned) for Match 9 would be as follows:

  1. Energy Standard (569.5)
  2. Tokyo Frog Kings (423)
  3. Iron (414.5)
  4. Toronto Titans (299)

However, adjusting the jackpot margin to 5% – just by shortening the cutoff by a few hundredths of a second – the results change and the standings would be:

  1. Energy Standard (559.5)
  2. Iron (425.5)
  3. Tokyo Frog Kings (423)
  4. Toronto Titans (298)

The complete list of differences in the results of Match 9 are outlined in the Table 9.

Table 9: What Changes from 5.1% to 5%?
Event Winner Jackpotted (Highlighted = new to 5%) Point Shift from 5.1% to 5%
W 200 BR Kelsey Wog (TOR) 2:18.47 Viktoria Gunes (ENS) 2:25.25 ENS-1, TOR+1
M 200 FL Chad le Clos (ENS) 1:50.24 Leonardo Santos (IRO) 1:57.6 ENS+2, TOR-2
Machkenzie Darragh (TOR) 1:55.71
M 50 FL Nicholas Santos (IRO) 21.78 Cole Pratt (TOR) 24.04 IRO+11, ENS-11
Yuki Kobori (TOK) 23.97
Finlay Knox (TOR) 23.44
Marco Orsi (IRO) 22.95
Andrey Zhilkin (ENS) 22.93
Chad le Clos (ENS) 22.92

Essentially, Nicholas Santos gains an additional 11 points for Iron by shortening the cutoff from 1.15 seconds (5.1% rounded) to 1.10 seconds (5.0% rounded). Specifically, with the margin set for 5.1%, the resulting cutoff time of 22.93 allows Chad and Andrey to sneak in without losing their points. When the margin is set to 5.0%, the cutoff time of 22.88 just barely cuts them off. This difference of a few hundredths yields an 11-point swing from Energy to Iron, allowing Iron to surpass Tokyo for second place.

Interestingly, this showcases our previous discussion around the Men’s 50 FL as a leading jackpot-heavy event. Hopefully this also begins to paint the picture of just how precarious the selection of a jackpot margin can be.

The difficulty with this selection is that the specific situations that yield an impact from a change in jackpot margin are incredibly circumstantial. This can be demonstrated even better if we compare the 5% margin to the 4% margin. Table 10 summarizes all of the changes between Tokyo and Iron:

Table 10: Summary of Changes Between IRO and TOK for Jackpot Shift from 4% to 5%
Event Point Shift
W 100 FL TOK +2
M 100 FL IRO +3
W 200 BK IRO +3
M 200 BK IRO +1
W 200 IM IRO +1, TOK -3
M 50 BR IRO -5
W 50 BK TOK -1
W 400 FR TOK -3
W 100 FR IRO +1
M 100 FR TOK -1
M 200 FL IRO +3
W 200 BK IRO +2
M 100 IM IRO -1
W 50 FL IRO -4
M 100 BR IRO -4
W 400 IM TOK -3, IRO +3
M 400 IM TOK -3
W SKN IRO -1, IRO + 5

The point swings are haphazard. A few point-boosts here, but a few point-drops there, end up yielding a negligible net effect in terms of the final match outcome. To understand the essence of this, I think the following analogy is appropriate:

Imagine you were standing in front of a crowd of a few hundred people. If each of them projected music from their phone speakers, you’ll hear the music play at a certain total loudness. Imagine then having a handful of people adjust the volume of their phone, some up by a few steps, some down by a few steps. While there may be a lot of individual changes occurring, you’d likely not notice this as it would have a negligible net effect on the total volume you’d perceive from the whole crowd. This is what is occurring with the jackpot points.

Looking at any one event in isolation, there appears to be a significant swing of points – just like if you only heard a single phone speaker, you’d notice the change in volume. As these aggregate over the entire set of events, the changes all meld together to the point where they’re no longer noticeable. While the internals of the system are dynamic, from an external perspective, it’s largely static.

Even then, this only tells the story of how a few of the matches could have been impacted. And those impacts may seem underwhelming. Yet, the story doesn’t end here; we’ve still got another fifty to swim. There is one big factor we’ve still got left in the tank, one more piece of the puzzle to consider. Let’s bring it home strong in part four.

See Also

ABOUT STEVE GAMBINO

Steve grew up swimming in Middletown, CT.  Before recently moving to Worcester, MA in September, he spent the past five years in Rhode Island, teaching math at CCRI and coaching age group swimming with Crimson Aquatics.  Steve has an M.S. in Mathematics from University of Rhode Island, has previously served as a consultant for the ISL for the development of their rating system, and currently works as an Assistant Professor of Mathematics at Quinsigamond Community College in Worcester.

In This Story

2
Leave a Reply

Subscribe
Notify of
2 Comments
oldest
newest most voted
Inline Feedbacks
View all comments
QuestionableSwammer
3 days ago

So is the conclusion basically “if you change the jackpot margin, things change?”

Stewart 100 back gold in Fukuoka
3 days ago

Speaking of ISL, Tina Andrew is posting a lot of unrelated (and controversial) contents with the official account of New York Breakers. 🙄Good for marketing and publicity of ISL, I guess.