PROJECT EVALUATION

Under the leadership of Sean Smith, Mellisa Smith, and Iris Weiss, Horizon Research, Inc. (HRI) is conducting the external evaluation for the project.  HRI has over 15 years of experience evaluating mathematics and science education improvement projects, including relatively small and narrowly focused teacher enhancement projects, a number of Statewide Systemic Initiatives (SSIs), and materials development projects.  The evaluation differs from a more traditional approach due to the comprehensive research program the project includes; i.e., the types of data the project will collect for its own purposes include, and go beyond, those typically collected in an external evaluation.

Formative Evaluation

The formative evaluation component is focusing on several questions, all aimed at informing mid-course corrections the project might need to make:

1. To what extent does the course development process incorporate the knowledge, insights, and diverse perspectives of the members of the development teams?
2. How satisfied are the Mathematics Specialists with the training they receive?
3. How well-received is the statewide Master’s Degree program among college and university mathematics educators not directly involved in the project?
4. To what extent does the project use findings from its own research on preparation and deployment of the Mathematics Specialists to inform revisions to the program?
5. To what extent is the project able to use the results of its research on the policy environment to improve the “fit” (by modifying the program and/or bringing about policy changes) and increase the likelihood of going to scale with Mathematics Specialists statewide?

HRI will (1) review project progress and research reports annually; (2) interview project staff twice each year; (3) interview a sample of Mathematics Specialists in each cohort; (4) interview a sample of college and university mathematics educators external to the project; and (5) attend at least one project planning meeting each year in order to determine the extent to which the project utilizes research findings. 

Summative Evaluation

The project’s research component includes a controlled study of impacts resulting from the deployment of Mathematics Specialists.  HRI will play the role of methodological auditor (or meta-evaluator) in the summative evaluation.  The bulk of this work will take place in Year 5, when HRI will audit the project’s research findings in light of the data collected and the research methodology and assess the extent to which claims based on the research appear to be justified.  (Through the formative evaluation, HRI will remain grounded in the project’s work in Years 1 through 4.)

In addition to its own review, HRI will assemble a three-person panel with specific expertise in research methodology to review the project’s research on the effectiveness of the Mathematics Specialists and prepare a critique.   HRI will also convene a small group of mathematics education experts to critique the quality of the six courses for Mathematics Specialists.  Finally, a third review will focus on the quality and likely impact of the project’s dissemination efforts:  Are the research methodology and findings reported in sufficient detail and with enough clarity to enable others to decide if they are interested in replicating the program?  When interested people seek additional information, is the description of the program adequate to enable replication of components of the program?

HRI will report evaluation findings informally to the project staff through regular phone and email contact.  In addition, HRI will prepare one formative memo and one evaluation report each year detailing all evaluation activities and findings.