The purpose of this study was to examine the systematic classification and evaluation standard of early childhood education policies based on theories of early childhood education policy evaluation and the opinions of experts in an effort to develop early childhood evaluation policy evaluation criteria. The early childhood education evaluation criteria developed in this study are expected to make it possible to accurately evaluate early childhood education policies executed in this field and to offer useful feedback.
It's specifically meant to develop criteria available for early childhood education policy evaluation based on the views of experts in early childhood policy and evaluation, in early childhood education administration and in early childhood education and of kindergarten teachers, and to validate the developed evaluation criteria by testing their relative importance and usability.
To serve the purpose of the study, the following efforts were made:
1. Develop early childhood education policy evaluation criteria.
2. Validate the early childhood education policy evaluation criteria developed in this study.
The major findings of the study were as follows:
First, literature related to early childhood education policy evaluation criteria were reviewed to determine the right directions for the development of the criteria, and a focus group interview was conducted twice after eight experts were selected. SWOT analysis and content analysis were made to analyze the opinions that the experts gave in the focus group interviews. Besides, a primary Delphi survey was conducted to check the opinions of other selected experts on the right directions for the development of early childhood education policy evaluation criteria.
According to the results of SWOT analysis on the right directions for the development of early childhood education policy evaluation criteria, four categories were selected: strengths, weaknesses, opportunity and threat. When a SWOT matrix strategy analysis was made to formulate strategies for the development of the criteria, four strategies were suggested: 'satisfying the requirements of the times,' 'developing well-grounded criteria taking diverse theoretical perspectives,' 'addressing the needs of different members,' and 'developing sustainable, realistic and effective criteria.' And a content analysis of the focus group interview data was made to see whether the directions for early childhood education policy evaluation criteria would be right or not.
The primary Delphi survey was conducted on a panel of 33 members who were experts in early childhood education policy and evaluation, in early childhood education administration and in early childhood education. Open-ended and closed-ended questions were both used. A primary draft of the evaluation criteria was drawn up based on the opinions of the panel members, which consisted of four evaluation categories, 14 evaluation criteria and 55 evaluation elements according to the framework of evaluation category-evaluation criteria-evaluation element.
And then Delphi survey were conducted twice more on the panel of specialists. In the secondary Delphi survey, opinions were gathered from the panel about the primary draft of the evaluation criteria that was made based on the results of the primary Delphi survey. Their opinions on the representativeness, accuracy and comprehensiveness of the evaluation criteria, evaluation elements and evaluation indexes were analyzed. After questionnaires were handed out to 33 panel members, the answer sheets from 32 panel members were collected, and frequency analysis and descriptive statistics(mean, standard deviation, skewness and quartile) were utilized to analyze the collected data. Since the dominant opinion of the panel members was that the framework of evaluation category-evaluation criteria-evaluation index would be more adequate than the framework of evaluation criteria used in the primary Delphi survey, some of the terms were changed. When the appropriateness of the evaluation categories, evaluation criteria and evaluation indexes were analyzed, they gave a mean of less than four out of five in each index. The evaluation criteria and evaluation indexes to which less than four average marks were given out of five, which were positively skewed, or which the experts considered it necessary to modify were modified, and some of them were omitted or integrated. Thus, a secondary draft was drawn up, which consisted of 13 evaluation criteria and 44 evaluation elements.
In the tertiary Delphi survey, questionnaires that were designed to check the results of the secondary survey and reach a consensus were utilized, and frequency analysis and descriptive statistics(mean, standard deviation, skewness and quartile) were utilized. After the opinions of the panel were analyzed about appropriateness, a final draft was drawn up, which consisted of four evaluation categories, 13 evaluation criteria and 44 evaluation indexes. Out of the evaluation criteria, two were modified in part according to the opinions of the panel.
Second, AHP and usability analysis were conducted to check the validity of the early childhood education policy evaluation criteria. The 32 panel members who participated in the Delphi survey were asked to analyze weight by using AHP. The weight analysis method by AHP was composed of three-stage hierarchical structure, and each of the three stages respectively consisted of the four evaluation categories, 13 evaluation criteria and 44 evaluation indexes developed in this study. As a result of calculating the consistency ratio of the ratings of the individual panel members, the consistency ratio of the answers of 12 panel members out of 32 was above 0.1. So a second investigation was fulfilled by asking these panels whose consistency ratio was higher to rate once again. When the results of the first and second investigations were analyzed, the answers of 24 panel members whose average consistency ratio was below 0.1 were selected for final analysis.
And then the simple weight of the evaluation categories, evaluation criteria and evaluation indexes was converted into percentage. As for the evaluation categories, early childhood education policy publicness was given the most weight(34.2%), followed by policy suitability(31.7%), policy democracy(23.1%) and policy economic efficiency(11.0%). Afterwards, the simple weight of the evaluation criteria and evaluation indexes was respectively calculated within the superordinate dimensions, namely the evaluation categories to which each of them belonged. As a result of analyzing the total weight of the evaluation criteria and evaluation indexes, public benefit(.150) was found to be top priority for the evaluation criteria, followed by feasibility(.118), accountability(.102) and sufficiency(.090). As for the evaluation indexes, top priority was given to the level of contribution to the interests of preschoolers(.063), followed by the level of fulfilling social obligation(.056) and the level of long-tern benefit(.053).
In order to check whether the evaluation criteria developed in this study would be appropriate enough to assess early childhood education policies, 307 kindergarten teachers who served as early childhood teachers for three years or more were asked to assess the usability of the evaluation criteria and evaluation indexes. To be specific, their validity, reliability, response distribution for the evaluation indexes and the discrimination of the evaluation indexes were all checked.
As for the validity of the evaluation criteria, their content validity and construct validity were checked. As a result of calculating mean scores about the importance of the evaluation indexes, the kindergarten teachers placed high value on all the 44 evaluation indexes. Their mean scores ranged between 3.99 and 4.30. So the evaluation indexes of the evaluation criteria turned out to be composed of significant items to assess early childhood education policies, and the content validity of the evaluation indexes were quite good. To check construct validity, the teachers rated the appropriateness of the evaluation criteria. When the correlation between the evaluation criteria and total scores and between the evaluation indexes and total scores was analyzed, total scores were closely correlated to every subarea and every item. Thus, it's confirmed that the evaluation criteria and evaluation indexes developed in this study were valid enough to make a valid assessment of early childhood education policies. When internal consistency was checked to confirm the reliability of the subordinate evaluation indexes of the evaluation criteria, Cronbach alpha coefficient stood at .97. As a result of analyzing split-half reliability, Spearman-Brown coefficient stood at .93. The findings showed that the evaluation criteria were stable criteria to make a consistent assessment of early childhood education policies.
Regarding the usability of the evaluation indexes, the response distribution and discrimination of the evaluation indexes were analyzed. The response distribution was mostly negatively skewed, and the mean scores ranged from 3.85 to 4.14. Thus, none of the evaluation indexes were given extreme mean scores(M>4.50 or M<.50), and none of them had an extremely lower standard deviation(SD<.50), either. Therefore the evaluation indexes of the evaluation criteria were found to be appropriate for early childhood education policy evaluation. A group that was in the top 27 percent got higher mean scores in every evaluation index of the evaluation criteria than another group in the bottom 27 percent, and the differences between the two were statistically significant. Therefore the evaluation indexes of the evaluation criteria turned out to be excellent in discriminating between an upper-tiered group and a lower-tiered group.
Thus, the evaluation criteria and evaluation indexes were identified as a valid and reliable scale to assess early childhood education policies, and were found to be composed of usable items that can make a systematic assessment of the characteristics of early childhood education policies.