The Marin Post

The Voice of the Community

Blog Post < Previous | Next >

Guy

11th Graders' Performance in the Time of COVID

Posted by: Gaetan Lion - February 4, 2025 - 3:06 PM PST

The allusion to Gabriel Garcia Marquez's book "Love in the Time of Cholera" is purposeful. By the end of this article, we will uncover that the reference is most relevant.

Introduction

This essay focuses on analyzing the academic performance of the Tamalpais Union High School District (Tam) in Marin County. I will benchmark Tam's performance by doing the following:

The 5 school districts used as benchmarks include the following:

  1. Palo Alto Unified High School District (Palo Alto) in Santa Clara County.
  2. Los Gatos-Saratoga Union High School District (Los Gatos Saratoga) in Santa Clara County.
  3. Mountain View-Los Altos Union High School District (Mountain View) in Santa Clara County.
  4. Fremont Unified High School District (Fremont) in Alameda County.
  5. Acalanes Union High School District (Acalanes) in Contra Costa County.

The 5 school districts are more of a gold standard than a representative control group. By definition, we know these districts perform better than Tam given their respective rankings. But, it will be informative to compare their respective trends pre and post COVID.

Hypotheses

I have read numerous times that students' academic performance has suffered because of lock-downs during COVID. And, that their current performance may not have fully recovered to their pre COVID level.

I have also read in our local media that Tam's performance has deteriorated and is falling further behind the mentioned superior districts.

Are both statements accurate? We will soon find out.

The impact of COVID at a high level

At a high level there is not much evidence that COVID has hurt the performance of 11th graders in the mentioned districts.

The table below compares the average % that meets or exceeds the CASPP standards in English and Math. And, it compares the average during the pre COVID years vs the post COVID years.


As shown above, at the State level English performance during the post COVID period declined just a bit. In Math the decline was more pronounced. However, when focusing on the districts, 5 out of 6 of the districts did better in English, and 3 out of 6 did better in Math during the post COVID period.

This high level analysis just focuses on averages and medians. It does not capture pre and post COVID yearly trends. We will soon see that when focusing on trends, we can observe that COVID appears to have a lagging effect several years out.

English performance trend

The table below does not show any data in the 2019 - 2020 school year because the CASSP testing was not conducted during the onset of COVID (March 2020). It shows a clean break between the pre and post COVID years.


We can visualize the above data below. The blue line at the bottom represents the State. The deep orange line, second from the bottom, represents Tam. As expected, Tam performs a lot better than the State. But, it does not quite match the gold standard group.


Notice on the graph above, that the 2020 - 2021 school year (shown as 2021) performance, most affected by the lock-downs, was very strong. Beyond this school year, the performance declines.

Math performance trend

Exact same comments as for the English performance trend section.



Performance of the 2023 - 2024 school year (shown as 2024)

The local press mentioned much disappointment in Tam's performance during this most recent school year, including concerns that Tam is falling further behind the mentioned districts.

The table below shows the difference in performance during this most recent school year vs the average during the pre COVID period.


The table above shows that Tam's lackluster performance in 2024 was common among districts.

In English it fared better than the State, and much better than Palo Alto, And, its performance was similar to Mountain View, Fremont, and Acalanes.

In Math, Tam fared much better than the State, Palo Alto, and Acalanes.

When focusing on relative performance, Tam's 2024 results (a decline) fall in the ball park of the other districts, and is much better than the State.

Measuring the Effect Size or Impact of COVID

Here we are measuring what is the effect size of COVID on the post COVID years. We do that by calculating the number of standard deviations above or below the average during the pre COVID period. Let me illustrate this calculation with an example below.

Let's say a district performance is 56% in 2021. During the pre COVID years, its average was 54%, and its standard deviation was 2%. This district standardized Z value would equal:

(56% - 54%)/2% = 1.0

This would reflect a pretty strong performance that is 1 standard deviation above its average during the pre COVID years (equivalent to 83d percentile).

The table below discloses the effect size (as Z values or number of standard deviations away from the pre COVID average). Again, using this other framework of relative performance, Tam is not that distinguishable from the other districts. And, in both English and Math it performs much better than the State.


The table above also shows the impact or effect size that COVID had on various student grades. As shown above, the 10th graders who experienced the COVID lock-downs in the 2019 - 2020 performed surprisingly well in the 2020 - 2021 school year. But, the younger grades typically experienced a downturn in performance.

This mentioned pattern is much pronounced at the State level for English and even more so for Math as shown on the graph below.


Explaining why the 10th graders performed so well and the younger grades so badly by the time they got tested as 11th grader is challenging. The abrupt drop in performance between the 10th and the 9th graders is arresting.

Although the districts follow a somewhat similar pattern, at the district level there is much idiosyncrasy or randomness. This is the case for the Los Gatos Saratoga District, one of the strongest performers in the Bay Area.


While at the State level, the effect size (positive or negative) was much more pronounced in Math, for Los Gatos Saratoga, it is a lot more pronounced in English.

At the State level, the 9th graders performed really poorly. Meanwhile, in Los Gatos Saratoga, the 9th graders performed very well in English (1.4 standard deviation above the pre COVID average). And, their Math performance was not far off their pre COVID average. The Los Gatos Saratoga 8th graders performed really poorly in English. Meanwhile, their 7th graders performed quite well (when tested as 11th graders, they exceeded the district's average performance pre COVID).

Tam's pattern is also interesting. The graph below shows that Tam's performance holds up well going as far back as 8th graders (when tested as 11th graders). But, the performance drops precipitously for the 7th graders (when tested as 11th graders).

The lagging Effect of COVID

Reviewing the above COVID effect size analysis is perplexing.

At the State level, the lagging effect impacted the 9th graders, and that negative effect remained constant for the lower grades (when tested in 11th grade).

For Los Gatos Saratoga, this lagging effect primarily hits the 8th grader in English only.

For Tam, this lagging effect primarily hits the 7th grader in Math only.

One may argue that the State level figures are more representative because they capture the whole population. And, that the districts results are more noisy because they are based on small samples.

On the other hand, one may advance that there are genuine differences between districts. And, that the State level figures entirely mask these in-between districts differences.

From a math standpoint, both statements are correct.

If you thought that Gabriel Garcia Marquez's "Love in the Time of Cholera" was confusing, so is "11th Graders' Performance in the Time of COVID."


THE END











Gaetan Lion is an independent researcher with expertise in economics, econometrics, investments, risk management, statistics, and quantitative analysis. His areas of interest include demographics, economics, investments, Climate Change, health care, sports, and politics.