A Data-Driven Organization's Approach to Assessing The Quality of Program Delivery
Relative to our efforts to produce high quality instructional programs, we work with two different kinds of information, (1) outcome and (2) process. Generally speaking, outcome data refer to student achievement, typically test scores, percent of students meeting standards, and the like.
Process information, on the other hand, tells us about program delivery, the materials and equipment we use, the instructional and support activities in which we engage, and the roles, responsibilities, skill levels and attitudes of the staff responsible for instruction (teachers, aides, specialists, administrators, district office staff). Put all these together, we have process, the things we do.
"Process" has quality attached to it. We can have a high quality process or we can have a lesser quality process. What we must embrace as professionals is the premise that there is a direct relationship between the quality of our process and the quality of students' results. The better we are the more optimal will be our students' results. The less effective our process, the less will be our results. Acknowledgment of this truth is crucial!
Now, over which of these two types of information do we have control and over which do we not. We control our process. We cannot directly control the outcome. It's like an athletic event. A coach controls the preparation and efforts to enhance the skills of the players. The coach cannot control the outcome of the game.
I believe as educators we spend far too much time laboring over test scores, contracting a case of analysis paralysis along the way, and far too little time laboring over the strengths and weaknesses of our instructional programs. Regardless of how effective our programs are, those programs have weaknesses. Data-driven organizations know through data collection, formal or informal, the strong elements of their programs and the elements that need shoring up. The goal of data-driven organizations is to continuously improve the quality of delivery, knowing that by doing this, the outcomes will be affected. The goal of data-driven organizations is NOT to improve test scores. Let it go. It will happen!
If you believe this, then the next step is to define the elements of process. When we collect information about the quality of delivery, what do we collect? Over the past few years, in my work as an evaluator and a data guy, I have defined "process" using 15 elements. I believe that educational leaders should be able to discuss the contribution of each of the 16 to a program's overall quality. By doing this, honing in on the weaknesses becomes a natural extension of the data collection process.
Here are the 15 elements:
- Materials and equipment for teachers
- Materials and equipment for students
- Materials and equipment for parents
- Existing program content (standards)
- Time spent on instruction
- Teachers' daily schedules
- Diagnosing and prescribing instruction on an individual and group basis
- Assessing student results (outcomes)
- Managing the classroom
- Providing a positive classroom environment
- Physical facilities
- Teachers' knowledge and skills
- Teachers' receptivity to the instructional program
- Communication system among the principal, teachers, and parents.
- The school's staff development program
Each instructional program on a campus has these 15 elements. If we're considering a reading program, these 15 elements apply; the same goes for math, science, fine arts, physical education and all the others.
I recommend that a staff dialog, discuss, and reach agreement as to which of the 15 support the quality of a particular program and which of the 15 are deterrents. When the deterrents are known, I then suggest that they be ranked and the top three be identified. Then go after the improvement of those three.
I realize that this is a bit linear. And I realize that changing one part of a program changes something else. But I do believe that if a school does not have a process for looking at the quality of its process in some rather specific ways, this is an awfully good way to start.
Several years ago I designed a form called the Analysis of Process (AOP). The form directs the discussion in these 15 areas. It's a single sheet two-sided form that takes about an hour for a small group of educators to complete. Click here for a copy of the AOP form.
Other Columns by Jim Cox: