Using Academic ROI to Drive Student Outcomes
With education budgets highly uncertain in the current economic climate of COVID-19, it is more important than ever for school districts to be careful with how they are spending to maximize impact on student outcomes. In a previous post, we discussed the importance of the strategic budgeting process. Present in this process is the ability to monitor and quickly iterate investment priorities in favor of what helps students the most. To do this, K12 leaders need to measure the impact of a given program on student outcomes, a concept called Academic Return on Investment (A-ROI).
Academic ROI is a mechanism through which leaders can measure how much “bang for their buck” they are getting. It standardizes program outcomes and attempts to make them measurable so that various alternatives can be compared and chosen between. But this process needs to be carefully constructed to maximize the impact of a district leader’s decision.
Before embarking on this journey to measure value of a given program, a set of program goals that are working to help students along their educational journey need to be developed. These goals should be established in close collaboration between finance and instructional departments to ensure that they are in line with a district’s needs and are student oriented. Clear communication is crucial in this step. Instructional specialists need to communicate academic priorities to district leaders, and district leaders need to communicate financial limitations to instructional specialists. In this way, those closest to student learning can give initial recommendations of the most impactful programs for student learning within given budget constraints.
Program goals should be established to accurately measure program effectiveness. Based off these program goals, initial investments can be made. However, the true impact of a program cannot be known until it is observed in action. Launching several pilots of different programs at different sites can become unwieldy rather quickly. A more practical way of approaching this is either testing one program at a time at a small scale or encouraging school leaders to test program ideas of their own and supporting their efforts of measurement and evaluation. This allows different perspectives to attempt to achieve set program goals in unique ways that can be scaled up to a whole district level if they are successful.
Calculating Academic ROI
Once program goals are established, they act as measures to use in gauging program effectiveness. With this in hand, we’re on the road to calculating an academic ROI of each program. The formula for calculating the Academic ROI is:
By “Learning Increase”, the formula means the difference between the new program outcomes and what the projected natural increase would have been by the same point in time (using historical trends). Dividing this increase by the dollars gives you the academic impact per dollar spent of a given program.
Doing this across several programs at different sites, a district has a comparable measure to assess each investment opportunity. Before using this data as the single source of truth, districts should be sure to factor in any contextual information to temper A-ROI results and give us a more complete picture of each alternative.
Data Selection and Tracking
A big part of this process is being able to track costs accurately and comprehensively across programs. This means that any materials, supplies, curriculum, or even personnel time allocated should be accounted for in running this calculation to ensure that the A-ROI figure is accurate and can be used to make future investment decisions. Personnel time in particular is often missed in A-ROI assessments but is often the most expensive part of any new program.
Edstruments can help in this process by tracking expenses tied to specific program investments down to the individual transactions and helping with this A-ROI process. Our platform can be customized to make it easy to associate dollars with goals for any budget owner in a school district, and then compare investment and progress towards these goals across sites.
In addition to dollar tracking, district leaders need to be careful to select consistent and representative metrics for student learning. The parameters for measuring these metrics should be consistent across a given program whether they’re quantitative or qualitative. We need to know exactly what we’re measuring, how we’re measuring it, and be able to measure it consistently to compare alternatives.
Act on Results
As mentioned briefly above, immediate decisions based on the A-ROI of a given program should not be made alone. Important context needs to be considered before acting on this information such as student demographics, other changes in the environment that may have influenced outcomes, or other circumstances that could have contributed to the learning increase (or lack thereof).
If the return on a given program seems to be particularly low, poor implementation should also be examined as a cause before canceling a program. It is entirely possible that even with solid effort put into a program launch, some misstep was made that impacted the outcomes of the program. Scratching an option before examining implementation could cause a district to hastily move past an effective program and spend more time and money testing others.
When examining whether a program should be expanded, it is worth remembering that investment in a program is not all or nothing. If a particular program works well with specific student populations, it could be reasonable to expand that program to the scale of the applicable students and invest in another alternative to achieve the same goal for a different group of students. Leaders should be careful not to use this philosophy to unintentionally expand inequities between student groups within a district.
Finally, if implementation, context, and program results are examined and results are looking poor, it’s time to assess other alternatives.
This iterative process should be continuously used in school districts so that a district becomes adaptable in a rapidly changing environment.
Some Final Thoughts
When making a program investment, districts should strive to ignore sunk costs. Just because a bunch of money has already been spent toward a certain program doesn’t mean it must continue to exist. This mentality can lock schools into mediocre and stagnant practices which ultimately hurt students. One of the biggest sources of waste in districts is continuing programs that have limited positive/measured impact on students just because of inertia from the history of an organization.
Finally, a district should be careful not to ignore opportunity costs of a program investment. This is particularly applicable when looking at human capital allocation to a program. If a staff member working on one program is taking time away from another where they are needed more, the costs could offset any learning increases from the prospective program.
It's worth mentioning that A-ROI can be challenging to measure and that these are suggested guidelines for a process. It can be exceedingly difficult to connect a single program or set of programs to academic outcomes, but it’s worth making the best effort possible – particularly when introducing new programs or addressing novel problems. Developing a culture of innovation through iteration is the most consistently effective way to find the best solutions for students.
At the end of the day, leaders of school systems have been entrusted with a high responsibility: deploy the funds provided to them by taxpayers in the way that best serves the students in their schools. The next generation is America’s most precious asset, and it is crucial that they are given high quality and equitable opportunities to succeed and pursue the future they desire.
We would like to thank the Government Finance Officers Association (GFOA) and the Association of School Business Officials (ASBO) for informing the contents of this post. We hope it proves useful to anyone looking for effective decision-making strategies within school districts.
Are you a leader at an organization looking to more effectively allocate and deploy financial resources in service of your mission? Reach out to us today at firstname.lastname@example.org and we’d love to learn about your needs and share how we can help.