Development and Validation of Poksd-Porsd Protocol Assessment of Engineering in Elementary Classroom

The aim of this study was to develop and validate classroom observation instruments designed to reveal the emergence of engineering activities in primary school teachers in project-based learning. The instruments developed included the elementary school classroom observation protocol sheet (POKSD) and the elementary school engineering observation protocol assessment (PORSD). Task items were arranged based on indicators adapted from COPUS (Classroom Observation Protocol for Undergraduate STEM) items. The initial design of the instrument was consulted with three experts based on learning objectives. The instrument was then validated by three experts in the field of basic education. The instrument test was conducted on teachers and 5th-grade students of UPI Bandung Laboratory (N = 1). POKSD and PORSD were assessed by three raters. Scores from the three raters were then analyzed using two-way ANOVA. The results showed that the intra-class correlation of performance assessment instruments was adequate (ICC = 0.773). The findings of this study demonstrated that the instrument was reliable and could be used for the emergence of engineering activities in elementary school teachers.


INTRODUCTION
Globalization, with impressive technological developments, which can separate current geographical boundaries, is easily eliminated by various advances in information and communication technology. Acquiring new knowledge is easily obtained quickly, leading to the emergence of a digital-based economy that enables those who have the ability to obtain, process, and interpret various information, and this knowledge has the ease of responding to various challenges in the global community.
Thus, the concept of learning that should be directed at can equip students with 21st century skills so that they contribute to strengthening the ability to work together, solve problems, think critically, think creatively and innovatively that have the potential to sustain the economy. Learning with the STEM approach is one way to achieve the above objectives.
STEM becomes an alternative where the concepts, principles, and techniques of STEM are used in an integrated manner in the development of products, processes, and systems that are useful in later careers. Subsequently, STEM education has been adopted as an interdisciplinary approach to learning [1]. In learning with the STEM approach, students use science, technology, engineering, and mathematics in the context of real life that is connected between schools, the world of work, and the global world so that students can compete in the global job market.
The main aspects of STEM are the science process and the engineering design process, both of which are closely related to support learning. The science process is a tiered process consisting of the following stages: 1) ask questions or make observations; 2) arrange hypothesis; 3) develop an estimated answer; 4) conduct tests/experiments; 5) find and make conclusions. While the engineering process design is a cycle as follows: a) ask (explain the problem and identify the constraints); b) imagine (think ideas and choose the best); c) plan (draw diagrams and gather materials); d) create (follow the plan and test); e) improve (discuss the possibility of improvement and repeat steps 1 through 5) [2].
The science material for the STEM approach must undoubtedly be adjusted to the STEM learning characteristics [3]. In analyzing the material in the 2013 curriculum, we can identify various basic competencies (KD) in the realm of knowledge and skills related to design activities in the form of processes, systems, and products. If the appropriate KD has been identified and selected, then the next step is to formulate competency achievement indicators (IPK) as markers of achievement that can be measured and observed as a reference for subject assessment. Criteria that can be used in preparing the IPK are urgency, continuity, relevance, usability (UKRK). The next thing to do is STEM analysis on the chosen topic. In this analysis, process activities that are relevant to the four domains of science, technology, engineering, and mathematics must be identified. The results of the STEM analysis on science learning in the implementation of the 2013 curriculum are described in the student worksheet (LKPD).
To find out whether STEM is implemented in learning, it needs a measuring tool that can describe the achievements and learning process with the STEM approach, since so far, engineering has always been ignored in STEM implementation [4]. A measuring tool is needed that can reveal the implementation of the engineering process in STEM learning.

METHODOLOGY
The developed instruments consisted of a POKSD classroom observation assessment (elementary school classroom observation protocol) adapted from COPUS [5] and an elementary school engineering observation protocol assessment (PORSD). The aims of the task item construction in the assessment instrument were to objectively assess the activities of students in the class and note the actions of the teacher and students in learning. Task items were constructed according to indicators on the behavior of instructors and students on a short time scale.
Specifically, the observer identified from a list of 25 codes (12 and 13 codes for instructors and students, respectively) for which behavior occurred within each 2minute time frame. The instructor's behavior included teaching, asking questions, or writing on the board, while being a student, the behavior included listening, working in groups, and answering questions ( Table 1). Analysis of class periods via POKSD resulted in two pie charts (one for student behavior and one for teacher behavior) that illustrated the prevalence of each code. This prevalence was calculated by dividing the total number of 2-minute time blocks at which a particular code was used by the total number of codes used. Elementary school engineering observation protocol assessment (PORSD) was adapted from SEcLO [6] to objectively access the emergence of engineering activities undertaken by students and teachers in the classroom, which was explained in Table 2. After the task items were constructed, rubrics and scoring were made. The rubric was arranged in three gradations of performance. The highest level of gradation was given a score of 3, and the lowest level was given a score of 1. The assessment instrument was then consulted with three experts to obtain input. Based on this input, improvements were made. The performance assessment instrument was further validated by three experts in the field of elementary education. The assessment aspects for the validation of the construction of the assessment included: (1) sentences were easy to understand; (2) there was no waste of words; (3) ease of use to assess; (4) compliance with indicators. Scores obtained from experts for the four aspects were then analyzed by Intra-Class Correlation (ICC) two-way mix ANOVA consistency that emphasized the similarity of ratings between raters. In addition to being used to test reliability among raters, this correlation could be used to determine the validity of an assessment instrument based on how much the consistency of inter-expert judgment (ICC consistency) in assessing task items. Eventually, the instrument that had been validated was tested, which assessed the emergence of engineering in classroom learning. In this trial process, the POKSD and PORSD that had been undertaken by observers were assessed by three elementary education lecturers as raters. The results of the assessment of the three raters were then analyzed by intra-class correlation using two-way random ANOVA internal consistency to determine empirical reliability.

RESULT AND DISCUSSION
From the results of the two-way mix ANOVA statistical analysis, the intra-class correlation coefficient (ICC) consistency between experts for the aspects was as follows: (1) the ease of the sentence to understand was 0.962, (2) the wastefulness of words was 0.904, (3) the ease to judge was 0.895, and (4) compliance with engineering indicators was 0.916. Based on the ICC coefficient > 0.80, this showed good consistency among experts [7] for all aspects of the task item, indicating that the validity of the task item was high or in other words, this engineering assessment instrument could be used to assess the emergence of engineering students and teachers.
For the results of revisions and trials, generally, there were no revisions that were too meaningful. The revised engineering assessment instrument was used to assess the emergence of engineering by involving three people (teacher, lecturer, expert) as a rater. From the results of the two-way random ANOVA statistical analysis, the intra-class correlation coefficient (ICC) consistency between raters was 0.773, showing that the intra-class correlation of this assessment instrument was adequate [7]. This result indicated that the instrument was reliable to measure the emergence of teacher and student engineering. Quantitative analysis of the construction validation results showed that the engineering assessment instrument produced consistency among validators with the right expertise, showing that the engineering assessment instrument was valid to be used to assess the emergence of teacher and student engineering.
Quantitative analysis of the engineering scores of teachers and students from three raters showed that performance assessment instruments generated adequate inter-rater consistency, showing that this engineering assessment instrument had sufficient reliability for assessing the emergence of teacher and student engineering.

CONCLUSION
All aspects of the task item showed consistency between competent experts (ICC consistency > 0.80), meaning that the validity of the task item was high; in other words, this engineering assessment instrument could be used to assess the appearance of engineering in teachers and students. Intra-class correlation coefficient (ICC) consistency between raters was 0.773, which showed that the intra-class correlation of these engineering assessment instruments was adequate. These results showed that the instrument was reliable for measuring the emergence of teacher and student engineering.