Will Your School District’s Strategy Substantially Improve Student Achievement? A Case Study
Jefferson County Colorado lies to the west of Denver and has a population of almost 600,000. It is quite affluent — 42% of adult residents are college graduates, and median household income was $70,164 in 2015.
The Jefferson County School District is the 36th largest in the United States. It serves 86,000 students, with 160 schools and total revenues of about a billion dollars per year. Of Jeffco’s 5,000 teachers, 98% are rated by their school principal as effective or highly effective. Thirty two percent of Jeffco students are eligible for free and reduced lunch (which includes students who “choice-in” to Jeffco from surrounding districts), and eleven percent attend charter schools.
In November 2018, the Jeffco Board of Education will likely ask local voters to approve a significant tax increase to pay for the implementation of superintendent Jason Glass’ new strategy for Jeffco schools. This potential ballot initiative raises three critical questions I will address in this paper: Does Jason’s strategy make sense? Is it likely to succeed? And does it require a tax increase?
In previous articles, I’ve noted that while “strategy” is a frequently used term, when you ask people to define it you get a very wide range of answers. So it is important to begin by reiterating the definition I’ve used for forty years:
It is also important to begin by putting Jeffco’s latest strategy into historical context, as the problems facing the district today have remained unresolved for a very long time.
The History of Student Achievement Problems in Jeffco
For over thirty years, it has been clear that Jeffco has a student achievement problem. Between 1971 and 1992, Jeffco published an annual report on its achievement results, titled “Are We On Target?” Starting in 1985/86, these reports were based on the results from the Iowa Tests of Basic Skills for grades 3 through 8, and the Iowa Tests of Achievement and Proficiency for grades 9 through 12. These were taken by all students every year. Also included were the results for Jeffco students who took the ACT and SAT college entrance tests during this period.
By the mid-1980s, it had become apparent that Jeffco’s math scores on the ACT and SAT were slipping. Between 1985/86 and 1991/92, Jeffco’s performance on most of the Iowa tests also declined compared to the national norm, and there was a significant drop in the number of Jeffco students scoring in the top quartile. The district itself noted that “results suggest that Jefferson County has not kept pace with the improvement in basic skills which has occurred nationally.” In 1992, Jeffco announced it would no longer publish the “Are We On Target?” report.
According to Marilyn Saltzman (F.M. Duffy Reports, January 2006), “in April 1997 Jeffco hired a research firm to conduct a survey of Jefferson County voters. Results showed that 48 percent believed the district was on the wrong track, while only 37 percent said it was moving in the right direction.”
In July 1997, Jane Hammond was hired as Jeffco’s new Superintendent, in order, as she noted, “to bring 144 relatively autonomous schools into one high-performing school district. The goal was to increase student achievement in a community with high standards and expectations” (Jane Hammond 2009 interview on AASA.org).
Hammond undertook an extensive process to develop a new strategy for the district, while the Board of Education established a new Strategic Planning and Budgeting Council “to oversee the strategic planning process as well as the implementation and evaluation of its results” (Saltzman article).
A key highlight of the new strategy was the successful 1999 passage of a unique and nationally recognized “incentive” mill levy tax increase whose cost was tied to improved achievement results (the target was based on the percent of students scoring proficient or advanced in grade 3 reading, grade 4 reading, grade 4 writing, grade 5 math, grade 7 reading, grade 7 writing, grade 8 math, and grade 8 science).
Here is what Hammond said about this “performance mill” in an interview with the American Association of School Administrators: “On the November 1999 ballot, we asked voters to approve a mill levy override that would provide $25 million annually in additional funds. Another $20 million in increased annual funding was contingent upon improved student performance on the Colorado Student Assessment Program, or CSAP. To earn the entire $45 million, we had to improve the percentage of students scoring advanced or proficient on the CSAP by 25 percent over three years. This unique approach worked, and we passed the first mill levy override since 1983.”
“The next steps were critical: To meet our promise, we had to wisely invest the $25 million in upfront money to improve student achievement. We developed a comprehensive program based on best practices research. Our plan included hiring instructional coaches in elementary and middle schools and significantly reducing class size in 3rd and 4th grades, which are the first grades tested. We also implemented increased teacher training, extended learning programs for students in need of extra help and special programs to serve our at-risk populations.”
“The baseline test score, calculated from 1999–2000 test results, was 53.2 percent of students scoring advanced/proficient. This meant the three-year goal would be 66.5 percent at the advanced/proficient level, a 25 percent gain. Just one year after implementing our Performance Promise Program, we increased to 60.4 percent of students scoring at that level — 13,459 more students or the equivalent of 138 classrooms! The improvements made by Jefferson County students exceeded those of most other students in the state. These results meant that in a single year, we met more than half of our three-year goal. As a result, the district will receive approximately $10.6 million in additional revenue to support educational opportunities for students in the 2002–2003 school year.”
In July 2002, the Jeffco Board of Education suddenly replaced Jane Hammond with Cindy Stevenson, a career Jeffco employee. Over the next three years, Jeffco failed to meet any more achievement improvement targets in the Performance Progress Program, and as a result received no further increases in mill levy funding. Since then, Jeffco’s student achievement results have stagnated, despite changes in control of the Board of Education and the arrival of new superintendents.
In 2016, the last year all Colorado 11th graders took the ACT assessment, 55% of Jeffco students did not meet the benchmark for college and career readiness in reading, 56% did not meet it in math, and 60% did not meet it in science (note that the latter two are for the general benchmarks , not the higher ones for students planning to pursue STEM majors in college). These painful results cannot all be blamed on poverty, as only 31% of the district’s students were eligible for free and reduced lunch in 2016.
Also in 2016, on Colorado’s state assessment tests (CMAS/PARCC), only 42% of Jeffco third graders met or exceeded the state standard for English Language Arts. For sixth graders, it was 49%, and for eighth graders, 46%.
Only 45% of third graders met or exceeded the state math standard in 2016, along with 41% of sixth graders. In eighth grade about one third of Jeffco students take an advanced math assessment. Of the two-thirds who take the test for grade-level math, just 19% met the state standard.
In a world of rapidly intensifying global competition, the multi-decade failure to successfully address Jeffco’s student achievement problems has likely had negative consequences for many of the county’s children.
In “The Impacts of Neighborhoods on Intergenerational Mobility: Childhood Exposure Effects and County-Level Estimates”, Chetty and Hendren analyze the impact on income at age 26 of county of residence from birth to age 20 on children born between 1980 and 1991 (i.e., who graduated from high school between 1998 and 2009). They further segment their findings into the impact on children whose parents were at the 25th percentile of family income (i.e., relatively poor) and the 75th percentile (i.e., relatively affluent). As about 40 percent of Jeffco families are at or above the 75th family income percentile (based on 2013 Census data), I will focus on those results here.
Chetty and Hendren’s analysis finds that living in Jeffco from birth to age 18 results in a likely negative impact on children’s future income (at age 26) that is so large that it places Jeffco in the bottom 5% of all counties nationally for children from relatively affluent families (the authors extended analysis reaches the same conclusion for children in the top 1% of family incomes). For example, compared to a child who grows up in an affluent family in Fairfax County, Virginia, a child raised in Jeffco is likely to have an income at age 26 that is 25% lower (because the Jeffco child is likely to be 10% under the mean for 26 year olds who grew up in affluent families, while the Fairfax child is likely to be 15% above it).
You can find similarly negative results by comparing Jeffco to large affluent suburban counties in other major cities, including, for example, Dupage County, IL (Chicago), Allegheny County, PA (Pittsburgh), or Norfolk County, MA (Boston). Here is what the New York Times special section on the Chetty and Hendron research had to say about these findings: “Jefferson County is extremely bad for children in rich families. It is among the worst counties in the U.S.”
In their analysis of possible causes of these results, the authors find that two have had the strongest impact on outcomes for children from families at the 75th income percentile: their county’s level of social capital (e.g., as described in Robert Putnam’s book, “Bowling Alone”), and the quality of their county’s schools. The data show that Jeffco’s level of social capital has remained relatively unchanged between 1990 and 2005, ranking in the middle of U.S. counties (data can be found here). That leaves Jeffco’s schools as the most likely driver of its deeply painful results.
Before discussing the latest Jeffco strategy, let us be crystal clear about the situation we face. For over thirty years, Jeffco has failed to successfully address its student achievement problems. That failure has likely had a significant adverse effect on the economic prospects of at least forty percent of the county’s children — and continues to do so today.
Yet with depressing frequency, the latest announcement of poor Jeffco student achievement results is usually met with the same response by the district’s defenders: “But Jeffco outperformed Colorado.” There are two replies to this observation. The first is that Jeffco should outperform, because it has a lower percentage of at risk students than the rest of the state. The second is to note that what matters for Jeffco’s children when they apply to college, the military, or a job is their performance versus absolute standards, not their performance relative to Colorado. If your child can’t do algebra, it doesn’t matter that Jeffco outperformed Colorado. He or she is not going to get the job.
Another frequently heard excuse for Jeffco’s low achievement scores is that, “The tests don’t cover what we teach.” Jane Hammond objected to this excuse: “If standardized tests are aligned to high quality, rigorous, standards-based curriculum, you are not teaching to the test, but teaching to the standards” (Education Week, 9/21/05). If state achievement tests don’t cover what Jeffco teaches, it is because Jeffco has not aligned its curriculum and instruction with state standards; it is simply incorrect to blame the tests for the district’s poor results.
The Root Causes of Jeffco’s Student Achievement Problems
It is also important to review what previous Jeffco leaders and Boards of Education identified as the root causes of the district’s repeated failure to improve student achievement results.
Each year Colorado law requires school district leaders to prepare (and the Board of Education to approve) a Uniform Improvement Plan (UIP) that includes at least three of the most important root causes of the district’s achievement shortfalls. Here are the major root causes that were identified in past Jeffco UIPs:
2010/11 UIP
· “Schools lack district-wide formative and benchmark progress monitoring tools in writing to calibrate the fidelity of implementation of the district writing curriculum.”
· “Students needing to catch-up do not receive quality and/or adequate time for universal/core instruction in addition to effective, aligned targeted/intensive interventions.”
· “Many secondary schools do not have a systematic, tiered system of support to respond to the needs of catch-up students.”
· “The district has not strategically resource mapped and allocated resources differently among schools to better meet the needs of catch-up students.”
2011/12 UIP
· “Schools lack district-wide formative and benchmark progress monitoring tools in writing to calibrate the fidelity of implementation of the district writing curriculum.”
· “Teaching and learning have not consistently demanded high expectations in every classroom due to superficial coverage of a large number of standards, lack of understanding of grade level mastery, lack of relevance for students, and lack of systemic progress monitoring.”
· “Educators lack consistently rigorous, growth-producing feedback through the evaluation process.”
· “The district is not meeting the state expectation of 100% highly qualified teachers…because an effective monitoring system is not in place for existing staff especially for secondary schools.”
2012/13 UIP
· “The district lacks systemic practices in writing instruction and effective feedback in every classroom.”
· “The district lacks structures to ensure consistent quality of professional development is provided to all instructional staff.”
· “District and school leadership lack systemic understanding around multiple pathways for students to stay on track to graduate.”
2013/14 UIP
· “Systemic implementation of intentional lesson design to engage students’ conceptual understanding of their learning and increase student cognitive load is not evident in all classrooms.”
· “Educators have received inconsistent training on effective literacy instruction and the use of research-based resources.”
· “Math instruction tends not to be differentiated, lacking a variety of structures to meet students’ needs.”
2014/2015 UIP
· No UIP because of transition from TCAP to CMAS assessment
2015/2016 UIP
· “There is a need for implementation across the district of systemic evidence-based instructional practices that promote learning of rigorous literacy skills and competencies to ensure every student can “read to learn” by the end of third grade.”
· “There is a need for implementation across the district of vertical alignment (PK through 12th) and interdependence of math concept development that leads to successful learning in algebraic thinking…There is a need across the district for systemic classroom-based instruction, assessment and grading practices throughout the elementary and middle years that focus on higher level math concepts and procedures.”
· “There is a need for consistent differentiated teaching and learning practices matched to student needs so that every student will be successful in learning rigorous math concepts.”
2016/2017 UIP
· “In many schools, there is a lack of systemic evidence-based instructional practices that promote learning of rigorous literacy skills and competencies to ensure every student can “read to learn” by the end of third grade…For many students, the various literacy interventions are not specifically matched to student learning needs and may create additional barriers to learning rather than supporting literacy growth…Evidence indicates that current professional development in standards/competency-based core instructional strategies and learning supports has had limited impact on the effectiveness of high level literacy practices and matching interventions to student needs…Evidence indicates that current professional development and resource allocation for literacy instruction has had limited impact on desired increases in student performance.”
· “In many schools, there is a lack of systemic classroom-based instruction, assessment, and grading practices throughout the elementary and middle years that focus on higher level math concepts and procedures leading to algebraic thinking…In many schools, there is a lack of systemic classroom-based practices that require application and transfer of higher order algebraic thinking to meaningful/relevant real world problems and contexts…There is a lack of understanding across the system of vertical alignment (PK through 12th) and interdependence of math concept development that leads to successful learning and algebraic thinking…There is a lack of commitment across the system to ensure consistent differentiated teaching and learning practices matched to student needs.”
· “There is a need across the district for a system-wide commitment to ensure classroom practices and programming choices that provide every student with the opportunity to successfully complete a Jeffco educational pathway…There is a need across the district for a clear understanding among students and staff of the most critical performance expectations to achieve in order to be prepared for each student’s learning pathway…The allocation (or reallocation) of resources and supports to meet students’ social, emotional, engagement, and advancement needs are not addressing barriers to learning for all students (including students with educational plans).”
2017/2018 UIP
· “All schools currently do not have the full capacity to meet all students’ early literacy learning needs due to inequitable resources and support, for a variety of reasons, including inconsistent structures and routines across schools for foundational classroom (core) reading instruction, variable evidenced-based instructional practices to differentiate teaching for all learners’ needs, and lack of literacy transfer and application to other content areas.”
· “The district and schools have not provided equitable resources and support for math students to have a well-established foundation in: mathematical reasoning to explain, justify, and evaluate their thinking used to solve problems and applications of reasoning across content areas.”
· “Due to inequitable resources and support, schools have limited capacity to differentiate instruction and engage students in relevant, authentic tasks, specifically in the core areas of mathematical reasoning and conceptual understanding in addition to memorization of facts.”
Apart from Jason Glass’ new claims of resource insufficiency and inequity in the 2017/2018 UIP, all the previous Jeffco UIPs — prepared by both former superintendents Cindy Stevenson and Dan McMinimee — tell a very consistent story. They also raise a critical question as to why the root causes they identified — which are behavioral, not financial in nature — have never been successfully addressed.
A Closer Look at What Jeffco’s Student Achievement Goals Really Mean
Jason Glass became Jeffco’s latest superintendent in July 2017. His new strategy for the district is described in two documents he has produced. “Jeffco Generations” describes Glass’ vision, while the district’s website identifies twenty- seven initiatives (which Glass calls “tactics”) to implement it. These range from “embracing the full range of the human experience” to “student directed learning”, “schools as community hubs”, and “building the Jeffco brand.”
In an earlier paper, I criticized Jeffco Generations for being a confusing mix of lofty aspirations and questionable, unprioritized goals, whose results will generally be evaluated using soft metrics that are easily gamed (e.g., employee survey results). I concluded that Jeffco Generations is not a real strategy for the district — that is, “a causal theory of how to achieve critical goals with limited resources in the face of uncertainty.”
I will now expand my critique, and focus on the probability that Jason’s twenty-seven initiatives will actually produce a substantial gain in Jeffco’s student achievement results. In particular, I will focus on Jason’s stated goal of increasing Jeffco’s average 11th grade SAT math score from its current level of 525 to 560.
To be sure, one can argue that other goals are also important. However, I have found few people and employers who don’t believe that college and career readiness in reading, writing, math, and science aren’t necessary, even if they aren’t sufficient for our children to survive and thrive in the harshly competitive and rapidly changing world that awaits them after high school.
Colorado requires all 11th grade students to take the same standardized assessment, both to promote college attendance and to provide educators, politicians, and the public with a final measure of the quality of the education the state’s children have received, for which Coloradans pay nine billion dollars each year (about one billion a year in Jeffco alone).
In 2017, Colorado switched from using the ACT test to using the SAT for this assessment. This decision was controversial, for many reasons. For the purposes of this paper, I’ll focus on just one: Some analysts have claimed that the SAT’s math test is significantly easier than the ACT’s (e.g., see “New SAT Easier than ACT?” by Marshall Findlay). The evidence from Jeffco seems to support this claim. For example, in 2017, 49% of Jeffco 11th graders failed to meet the minimum college and career ready benchmark score on the SAT math test. In 2016, 55% of Jeffco 11th graders failed to meet it on the ACT math test, as did 56% in 2015.
Before examining Jeffco’s math achievement goal and strategic initiatives in more detail, it is helpful to begin with some national baseline/reference data about the probability that students who are not on track will eventually catch up and meet the college and career ready benchmarks.
Let’s start with 2017 CMAS math test results, and the percentage of Jeffco seventh graders who were more than one and less than one standard deviation below the cut score for meeting the state math standard (I use seventh instead of eighth because seventh is last grade in which all students take the same CMAS math assessment):
The ACT organization estimates that eighth graders who are less than one standard deviation below the cut score for proficiency in math have a 19% chance of catching up to meet the college and career ready standard on the ACT assessment.
For 8th graders who are more than one SD below the cut score, the estimated chance of catching up is only 3%. Even if these percentages are slightly higher for seventh graders, they are still dauntingly low.
Unfortunately, one year into Jason’s tenure as superintendent of Jeffco Schools, there is no evidence that students performing below grade level in math are catching up. Jeffco students in grades three through ten take NWEA reading and math assessments in the spring, winter, and fall of every year. One of the outputs from these tests is the percentage of students who are predicted to not meet the college and career ready benchmarks on the ACT tests.
On the Spring 2018 NWEA math assessment, 55% of fifth graders, 58% of sixth graders, 56% of seventh graders, and 56% of eighth graders were not on track to meet the ACT benchmark. One year before, on the Spring 2017 NWEA math assessment, the comparable figures were 54%, 56%, 55%, and 56%. Nothing has changed.
Jason has set a goal for Jeffco of increasing its average Grade 11 SAT math score from the current 525 to 560. But what does this really mean?
Because Jeffco’s SAT math scores are approximately normally distributed (i.e., when plotted, they closely resemble the familiar “Bell Curve”), and because we know their standard deviation (we use the publicly disclosed figure of 107 for the state of Colorado), we can use statistics to identify the percent of students who would meet or exceed the SAT college and career ready benchmark if Jeffco’s average math score reached 560.
It is only 68%, compared to 51% today. This raises serious questions about whether Jason’s goal is sufficiently ambitious, given how fast many technologies are improving and the level of skill our children will need to survive and thrive in the future. Put another way, are Jeffco parents, employers, and taxpayers all in agreement that the district should accept that about one third of its student will fall short of college and career readiness in math? When Jeffco taxpayers are spending a billion dollars each year on their schools?
In contrast to Jason’s 68% goal, if Jeffco aimed for 75% of its students meeting the college and career ready standard, its average SAT math score would have to increase to 582.
Expressing the needed improvement in average SAT math score in terms of standard deviations instead of score points allows us to compare the size of the shortfall to make up to the reported results for a wide range of possible interventions that have been undertaken elsewhere to improve achievement results.
Jason’s 560 SAT goal represents a score improvement equal to .33 standard deviations. A goal of 582 would equal .53 SDs. It is interesting to note that when I did the same analysis two years ago using 2015 ACT scores, the required improvement for 68% of Jeffco students to meet the ACT’s more rigorous college and career ready benchmark in math was .68 SDs; for 75% of students to meet it, the required improvement was .88 SDs.
Frankly, I think that targets based on the most rigorous test and college/career ready standard are most relevant (and I note that unlike the ACT, the SAT doesn’t have a science test). So an improvement in Jeffco’s average SAT math score of between .33 and .53 standard deviations is really the minimum to which the district’s strategy should aspire.
Will Jeffco’s New Strategy Substantially Improve Math Achievement Results?
Having established the amount of improvement in average math achievement score that Jeffco needs to realize (over some period of time), the next step is to look at research findings about the achievement effect sizes for the twenty-seven initiatives that Jason has directed the district to pursue. In theory, Jeffco’s goal should be to pursue a mix of achievement improvement initiatives that maximizes the expected effect size that can be obtained using the district’s resources. Put differently, Jeffco should prioritize implementation of the most cost-effective achievement improvement initiatives, up to the limit of the resources that are available for this purpose.
Indeed, this approach is now mandated by federal law. As the California Department of Education has noted, “The Elementary and Secondary Education Act (ESEA) has consistently directed educators to implement interventions grounded in research. Under No Child Left Behind (NCLB), districts and schools were called to use “scientifically-based research” as the foundation for education programs and interventions. This has been replaced by ‘evidence-based interventions’ under the Every Student Succeeds Act (ESSA). This shift was designed to help increase the impact of educational investments by ensuring that interventions being implemented have proven to be effective in leading to desired outcomes, namely improving student achievement.”
At this point, we confront a problem: the lack of replication of many of the effect size findings reported by education researchers (e.g., see, “Facts are More Important than Novelty: Replication in the Education Sciences” by Makel and Plucker). Actually, this problem is much broader, as replicability of research findings in a growing number of disciplines is increasingly being called into question (e.g., see, “Research Reproducibility, Replicability, Reliability” by Ralph Cicerone, president of the United States National Academy of Sciences).
The solution to this problem is to not rely on the effect sizes reported in potentially unreliable individual research studies, but rather on those reported by so-called “meta-analyses”, which combine the results of multiple individual studies on the basis of the strength of the methodologies they use. Fortunately, when it comes to student achievement improvement initiatives, there is no shortage of meta-analyses we can use.
For example, consider the effect-sizes for five achievement improvement initiatives that are frequently encountered in district Unified Improvement Plans: Smaller Classes, Early Childhood Education, Free Full Day Kindergarten, Response to Intervention (a methodology for systematically delivering more instructional support to struggling learners) and more spending on Teacher Professional Development.
The Washington State Institute for Public Policy (WSIPP) is the non-partisan arm of the Washington State Legislature (similar to the federal Congressional Research Service). It is charged with conducting meta-analyses of the expected effect sizes and cost-benefit ratios of various policy proposals.
WSIPP’s analyses found an achievement improvement effect size (ES) of .019 of a standard deviation for reducing class size by one student in grades K-2, and just .007 for reducing class size by one student in grades 3–6. However, these were conclusions based on the large-scale implementation of these initiatives; higher effect sizes were found for class size reductions that were more limited in scope, and targeted at younger, at-risk, and male students. WSIPP’s general conclusion was as follows:
“First, the weight of the evidence indicates that, on average, class size is related to student outcomes — smaller class sizes improve outcomes, although the overall effect appears to be small. Second, the positive effect of lowering class size is much stronger in lower school grades and weaker in the upper grades. The bottom-line finding from our analysis of the evidence and economics of class size reduction is that in the earliest K–12 grades reducing class size has a high probability of producing a favorable outcome — that is, where the long-term benefits of reducing class size consistently exceed the costs. In the upper grades, on the other hand, reducing class size poses a substantial risk of an unfavorable outcome — that is, where costs may often exceed benefits” (see “K-12 Class Size Reductions and Student Outcomes”, and “Benefits and Costs of K-12 Educational Policies: Evidence Based Effects of Class Size Reductions and Full Day Kindergarten”).
There is a valuable lesson here, that is too often overlooked by school boards and education advocates: initiatives that are highly cost effective when carefully targeted often become cost ineffective when they are expanded to middle class entitlements.
It is also important to keep in mind that the impact of an intervention on the overall district achievement gap equals its expected effect size times the percent of district students affected by a given initiative. For example, an effect size of .20 for a group of students who account for 30% of a district would generate an expected .06 SDs reduction in the overall district achievement gap.
What about increased investment in Early Childhood Education? In their meta-analysis (“Early Childhood Education for Low Income Students”), WSIPP found an effect size on achievement outcomes (in grades K — 2) of .152 that faded to .085 by grades 6–9. Another WSIPP meta-analysis of full day kindergarten for disadvantaged students found an initial effect size of .12, which “faded out to nearly zero by grades two through five” (“Full Day Kindergarten: A Review of the Evidence and Benefit-Cost Analysis”).
Response to Intervention (RTI) is, at first glance, a very common-sense approach that uses assessment results to target additional instruction (“Tier 2” and “Tier 3”) at students who most need this support. Initial research on RTI pilots found very impressive effect sizes. For example, in “Meta-Analytic Review of Responsiveness-to-Intervention Research: Examining Field-Based and Research Implemented Models”, Burns et al found a median effect size of 1.09 for this approach (also see the Institute for Educational Sciences Practice Guides for RTI in Reading and Math). Given this initial promise, over 70% of U.S. school districts now use some form of RTI (which is also known as MTSS — Multi Tier System of Supports).
Unfortunately, scaled up implementations of RTI have often failed to live up to the results from the early pilots. For example, in November 2015 the U.S. Department of Education’s Institute for Education Sciences published an updated meta-analysis of RTI reading interventions (“Evaluation of Response to Intervention Practices for Elementary School Reading”) which found an average of zero to negative effect sizes for the 20%-30% of students who received additional Tier 2 support.
What about more district spending on teacher professional development (PD)? In “Teacher Compensation and Training Policies”, WSIPP’s meta-analysis found an achievement effect size of zero for general professional development programs, and .005 for content specific PD. These findings are consistent with other studies that have found that teacher professional development spending has at best a very minimal impact on student achievement results. For example, The New Teacher Project recently found that the districts they studied spent an average of $18,000 per year per teacher on professional development (counting direct costs and the value of teacher time) with no positive return (see their report, “The Mirage”).
Let’s now take a closer look at what research tells us about the effect sizes we should expect for the other initiatives that Jason is using to implement his Jeffco Generations vision for Jeffco schools.
In the case of some initiatives, I could locate no research about the expected achievement impact (e.g., “Embracing the Full Range of Human Experience” or “Build the Jeffco Brand”). In other cases, multiple initiatives seemed to be aimed at the same underlying student achievement driver (e.g., “Family and Community Engagement”, “Social Media to Expand Reach”, “Support School Efforts to Deepen Family-School-Community Partnerships”, and “Community Schools”). This is important, because expected effect sizes are only additive if they are aimed at different underlying student achievement drivers (e.g., student, family, instructional, and school factors).
In the case of other Jeffco initiatives, there is a decent body of research we can use to roughly estimate expected effect sizes. The cases of Early Childhood Education and Full Day Kindergarten have already been covered, as has the expected impact of smaller class sizes.
With respect to “Responsive Teaching”, “Professional Model of Teaching”, and “Educate for Professionalism”, research has found substantial effect sizes for the student achievement gains that result from having a highly effective teacher in the classroom instead of one who is just average. In “Measuring the Impacts of Teachers”, Chetty et al found that in New York, the effect size in elementary school was .12 for English Language Arts (ELA) and .16 for math, while in middle school it was .08 for ELA and .13 for math.
In a study of the impact of highly effective teachers in Los Angeles (“Validating Teacher Effect Estimates Using Changes in Teacher Assignments in Los Angeles”), Bacher-Hicks et al found effect sizes of .19 and .29 for ELA and math in elementary school, and .10 and .21 for ELA and math in middle school. Increasing the percentage of highly effective teachers in our schools is clearly an important leverage point for improving student achievement.
Unfortunately, researchers have yet to identify all the factors that drive superior teacher effectiveness. For example, in “Teacher Compensation and Training Policies”, WSIPP’s meta-analyses identified the effect size impacts of the following teacher factors on student achievement results:
· Having a master’s degree = (.004)
· Having a graduate degree in the subject taught = .023
· Individual Pay for Performance = .005
· Intensive Induction Programs = .07
Moreover, nowhere does Jeffco’s strategy contemplate getting poor teachers out of the district’s classrooms, including the 22% (1,130 teachers) who, according to the most recent US Department of Education data, are chronically absent for more than ten days each school year (the point at which DOE has determined the negative impact on student achievement becomes significant). Instead, the unspoken assumption in Jason’s strategy seems to be that more teacher professional development will turn Jeffco’s poor teachers into highly effective ones, despite what both research and Jeffco’s own history says about the likely ineffectiveness of this approach.
Two closely related Jeffco initiatives are “Technology to Transform Learning” and “1:1 Devices” — that is, a computer or tablet for every student. Thus far, there are a number of meta-analyses that reach contradictory answers about the impact of technology in schools. This isn’t surprising, as one of the key lessons from thirty plus years of technology investment by private sector companies is that expected benefits aren’t fully realized until major organizational changes are made to take full advantage of its capabilities.
That said, two recent studies more carefully looked at giving students laptops or tablets. Both concluded that the effect size impact on math achievement was neutral or negative, in both early grades and high school (see, “A Quantitative Investigation into the Impact of 1:1 iPads on Early Learners’ ELA and Math Achievement” by Bebell et al, and “The Relationship Between Student Participation in a 1:1 Laptop Initiative and Academic Achievement in a 9–12 Upper Middle Class Suburban New Jersey Public School District” by Brian Gatens).
Another Jeffco strategy initiative is “Schools as Community Hubs.” As described in a recent research evidence review (“Community Schools: An Evidence Based Strategy for Equitable School Improvement” by Oakes et al), “Community schools represent a place-based school improvement strategy in which schools partner with community agencies and local government to provide an integrated focus on academics, health and social services, youth and community development, and community engagement. Many operate year-round, from morning to evening, and serve both children and adults. Although the approach is appropriate for students of all backgrounds, many community schools serve neighborhoods where poverty and racism erect barriers to learning, and where families have few resources to supplement what typical schools provide.”
“Community schools vary in the programs they offer and the way they operate, depending on their local context. However, four features — or pillars — appear in most community schools:
1) Integrated student supports
2) Expanded learning time and opportunities
3) Family and community engagement
4) Collaborative leadership and practices”
While performance evaluations have found that community schools have significant effect sizes for some outcome variables (e.g., student attendance and graduation rates), the evidence for their impact on student achievement results is much more varied, and no comprehensive meta-analysis of these outcomes has been done. It is also important to note that the achievement effect size evidence cited in some community schools analyses is heavily driven by the impact of increased family/parent engagement, where meta-analyses have found an achievement effect size of .30 for these programs (e.g., see, “A Meta-Analysis of the Efficacy of Different Types of Parental Involvement Programs for Urban Students” by William Jeynes).
Some research reviews have been less enthusiastic about the impact of Community Schools. For example, in “Community Schools in Practice: Research on Implementation and Impact”, Jenkins and Duffy note that “While many advocates and education leaders tout the promise of community schools, skepticism remains, in part because the model is difficult to implement and sustain, and supporting research is scarce and mainly limited to comprehensive, long-running models… The variety inherent in community school goals, design, and implementation makes research and evaluation a challenge, and can limit the generalizability of findings when they are available.”
An evaluation of Community Schools by Paul Hill et al from the Center on Reinventing Public Education (CRPE) made a critical point in its title: “Comprehensive Social Services Can’t Substitute for Strong Schools”. As the authors note, “the links between even intensive services and student learning are weak and hard to find…An emphasis on social and health services can make teachers and principals think that instructional practices and conditions of teaching don’t have to change…[But] even when kids are healthier, schools that don’t challenge them and push teachers don’t get better results.”
Finally, a number of studies have concluded that the community school model is very expensive to implement, which raises important questions about its cost-effectiveness as a means of significantly improving student achievement outcomes.
A contrast to the single site community schools model being pursued by Jeffco is Massachusetts’ “Wrap Around Zone” model, which gains scale efficiencies by being implemented at a multi-school level (e.g., the district, or, in the case of Jeffco, the Articulation Area level). This model has been rigorously evaluated and shown to deliver impressive effect sizes for student achievement results — e.g., .24 SDs for math (see, “Focusing on the Whole Student: Final Report on the Massachusetts Wrap Around Zones” by Gandhi et al from the American Institutes for Research).
Closely related to community schools is the Jeffco initiative titled “Social-Emotional Supports”. Increased focus on developing students’ social and emotional skills is a promising student achievement improvement initiative. For example, in “The Impact of Enhancing Students’ Social and Emotional Learning: A Meta-Analysis of School-Based Universal Interventions”, Durlak et al found an average effect size of .32 for the impact of these programs on academic performance. Similarly, in “The Role of Noncognitive Factors in Shaping School Performance”, Farrington et al found an average effect size of .27.
However, given limited resources, an important distinction must be made between initiatives that address acute and chronic student mental health issues and those that focus on the long-term development of character traits like grit, resiliency, conscientiousness, and persistence. In addition to the obvious non-academic benefits of effective mental health initiatives (e.g., fewer suicides and lowered risk of violence), researchers have also found that there are academic benefits for all students in a classroom from reducing class disruptions. Two studies have found that repeated classroom disruptions by a classmate reduces peer academic achievement results by about .02 standard deviations (e.g., see “Disruptive Peers and Student Outcomes” by Kristoffersen et al, and “The Long Run Effect of Disruptive Peers” by Carrell et al).
The highest benefits from an integrated system of student supports are therefore likely to result when they address both short-term mental health issues (e.g., by adding social workers and psychologists, and improving coordination with outside agencies) and long-term character development (e.g., through social/emotional curriculum innovations). See, for example, “Integrated Student Supports: A Summary of the Evidence for Policymakers” by Moore and Emig.
In his public comments since taking over as superintendent of Jeffco Schools, Jason Glass has repeatedly referred to “transforming the student task” as one of the district’s key initiatives, along with “Student Directed Learning”. It is hard, however, to find research that gives one confidence that this will lead to a substantial improvement in aggregate student achievement results.
To be sure, it is self-evident that higher student engagement in learning will lead to better achievement results. For example, John Hattie’s meta-analysis research has found an effect size of .45 for “higher student concentration and engagement.” But as every teacher can tell you, the real challenge is how to generate that increased student engagement.
Meta-analysis of the effectiveness of a traditional method of teaching known as “direct instruction” has found an effect size of .56. This is so significant that some observers have called direct instruction the “Rodney Dangerfield” of education, as it gets too little respect. But direct instruction is not what Jason Glass is pursuing in Jeffco. Rather, he seems to be seeking higher student engagement through student-directed learning, project-based learning, and personalized learning.
While student-directed learning is often more enjoyable for the student, critics have questioned its effectiveness (see, “Why Minimal Guidance During Instruction Does Not Work” by Kirschner et al). As for project-based learning, the best one can say at this point is that the evidence is inconclusive (see, “Project-Based Learning: A Literature Review” by Condliffe et al from MDRC).
Personalized learning has been reported to generate effect sizes of .09 SDs for math achievement (see “Insights on Personalized Learning Implementation and Effects” by Pane et al from the RAND Corporation). However, other research has found that implementing personalized learning methods, particularly at scale in large districts, is very difficult (see “Personalized Learning at a Crossroads” by Gross and DeArmond from the Center on Reinventing Public Education).
The following table summarizes my assessment of the available evidence regarding the average student achievement effect sizes of the initiatives being undertaken to implement Jason’s “Jeffco Generations” strategy:
Based on this analysis of Jeffco’s strategic initiatives, what is the probability that they will achieve Jason’s unambitious (and quite likely inadequate) goal of a .33 standard deviation improvement in Jeffco’s average grade 11 SAT math score?
Even if Jeffco matches the average effect sizes achieved elsewhere for those of its twenty-seven initiatives for which there is comparable research, achieving its .33 SD math improvement goal will likely be a near run thing, particularly given how far behind so many Jeffco students are today, and that the data from NWEA MAPs shows that there has been no improvement in achievement results during Jason’s first year.
However, it is highly doubtful that Jeffco will match the average effect sizes achieved elsewhere. Years of root cause analyses in Jeffco’s Unified Improvement Plans make it painfully clear that the district’s culture has routinely resisted initiatives intended to address the perceived drivers of its achievement shortfalls. Indeed, this is a depressingly common phenomenon in K-12 achievement improvement programs across the United States (for example, see, “Improving Teacher Effectiveness, Final Report” by Stecher et al from RAND).
In addition to long-standing cultural resistance to change in Jeffco Schools (see, “Why Do So Many School Districts Fail to Improve?), the achievement improvement impact of Jason’s initiatives will also be reduced by the inevitable loss of focus and dissipation of effort that occur when an organization tries to simultaneously implement twenty-seven initiatives in pursuit of fourteen unprioritized goals.
Given these considerations, I conclude the probability that Jason’s strategy for Jeffco will succeed in reaching his math achievement goal within his three-year contract is less than 25%. And that is quite likely a generous estimate.
Does Jeffco Need More Money to Implement Its Strategy?
As I have written many times before, unless and until Jeffco can provide voters with an activity-based analysis of its costs, the answer must be no. Such an analysis — routine in the private sector — would show the activities that must be performed (and to what standard) to achieve Jeffco’s achievement goals, how they are to be measured, and their associated operating and capital costs. In the absence of this analysis (which, for example, would show how much is being spent on teacher professional development, and the returns on this spending), it is impossible to convincingly argue that Jeffco — which already receives a billion tax dollars each year — needs even more money to accomplish its goals.
The absence of this analysis also calls into grave question the basis for the claim in Jason’s first Jeffco UIP that a shortage of resources is a root cause of the district’s poor achievement results.
Finally, in private sector turnarounds it is usually the case that a company first has to demonstrate improved results using its current resources before it can ask investors for more funding. There is no reason why this should not also be the case for a billion dollar school district like Jeffco.
Conclusion
This analysis has been based on the assumption that in exchange for spending a billion tax dollars each year on Jeffco Schools, voters’ top priority is graduating students who meet objective college and career ready benchmarks and are well-prepared to succeed in the harshly competitive economy that awaits them.
I have used Jeffco’s new math achievement target as an example and shown that it is quite unambitious. While Jason seems to focus on setting goals that are relatively easy for the district to achieve (e.g., those being measured with employee surveys), I take a different approach, and focus on identifying the goals that Jeffco must achieve (however challenging they may be) if our children are to survive and thrive in the 21st century economy.
Despite ESSA’s focus on the use of evidence-based achievement improvement interventions, Jason has not provided any estimates of, nor research support for, the achievement improvement effect sizes or relative cost effectiveness of his twenty-seven initiatives.
When this analysis is done, it becomes depressingly clear that even the modest student achievement improvement goals that Jason has set for Jeffco have a low probability of being met.
These findings imply that, despite spending a billion tax dollars each year, Jeffco’s long struggle to improve student achievement will remain unresolved, and tens of thousands more children will graduate from district schools unprepared to successfully make their way in a highly uncertain and unforgiving world.
Tom Coyne is a business executive who has been involved in K12 performance improvement for 17 years. He is a co-founder of K12 Accountability Inc. (www.k12accountability.org)
