Christopher L. Smith
(3/06) The objective of this evaluation project is to evaluate whether or not the Ohio CMS Systems Change Project has met its stated objectives. The evaluation methods will include interviews, focus groups, document reviews, and surveys. Throughout all of these data collection methods, the goals outlined in the Ohio Project will inform the evaluation. To do so, four key questions (and sub-questions) will be asked throughout the evaluation process. These include:

1. Has the project improved the availability of useful information for individuals and families?
2. Has the project produced a comprehensive, statewide quality framework that outlines desired outcomes for the service system?
3. Does the new system provide information to determine
   • training initiatives
   • technical assistance methodologies
   • public policy
   • effective and efficiency measures
   • improved state and local resource deployment
   • statistically-based trends and analysis
4. Has the project produced an agile system for reporting information and discoveries?

Data Collection

The project will use two primary modes of data collection. These include qualitative evaluation and quantitative analysis of survey responses. The qualitative evaluation will further use three primary methods including interviews, focus groups, and document reviews. The purposes of these methods include: a) to collect information, and b) to validate findings and assumptions through triangulation using multiple sources. During each interview and focus group, the evaluator will elicit information regarding ay documents the participants feel should be a part of the evaluation. These documents will become a part of the overall document review process including the initial grant award documents and any updates as a part of the granting procedure. Finally, documents relating to any significant databases that house relevant data will be requested to augment the determination of progress and/or attainment of goals.

The survey data collection process will involve the identification of appropriate target groups for surveys, distribution of surveys, and collection of surveys. It is assumed that all surveys will be distributed electronically whenever possible, though traditional mailings will be possible as well. The evaluator will work with the AQ/QI Evaluation Project Manager as well as the Grant Advisory Committee to identify potentially meaningful target groups for ay surveys, as well as to agree on sampling procedures. All interviews, focus groups, document reviews, and surveys will focus on the evaluation questions outlined above.

Data Analysis

Analysis of qualitative information from interviews, focus groups, and document reviews will follow steps outlined in accepted qualitative analytic literature. These include preparing transcripts or written records, breaking down the records into units (the smallest meaningful pieces of information), categorizing the units into groups of units that have similar meanings, and creating a final category of units from all qualitative data sources. These units and categories are used in creating the evaluation reports including the Formative and Summative Reports. Units and categories will also be used to create a report called a “Member Check,” a document that is used in the final phase of the evaluation to elicit feedback from evaluation participants regarding initial evaluation findings. Survey data will be analyzed to create summaries of demographic indicators as well as respondents’ data. Survey questions will be created to address the initial grant outcomes, and data summaries will be prepared in both text and graphic forms to allow for quick access to assessment of progress and/or attainment of overall grant project outcomes. Survey data will be intended to be used to triangulate the findings from the interview, focus groups, and document reviews. Through all data collection processes, no identifying information will be collected and all participants will be asked to sign a consent form.