Student Achievement Monitoring (SAM) Tool in Pedagogical Practice

Abstract

This paper presents two closely related researches that explore the use of the Student Achievement Monitoring (SAM) tool in pedagogical practice. The SAM tool is based on a three-level model of mastering cultural ways of action (formal, reflexive, and functional levels are distinguished). The researches discuss pedagogical strategies that allow school pupils to move from a low formal level to a higher - reflexive and functional - level. In order to find and describe effective pedagogical practices, the authors have conducted two researches of pedagogues. The first research was concerned with exploring different pedagogical strategies of teacher pedagogical actions and their connection with the educational outcomes of school pupils. It was aimed at correlating the results of SAM testing of students with the real pedagogical practice of the school in order to test the working hypothesis that high results at the second and third levels of students' mastery of learning materials are connected to a particular qualitative teaching model. The second research was related to the search for optimal ways to organize pedagogue professional development in mastering the level-based model of school pupil learning. During this research the main pedagogical difficulties and ways to compensate for them were analyzed. The result of the research was the construction of a matrix of pedagogical actions based on SAM methodology, typology of diagnostic assignments for teachers, description of the technology for training teachers to make three-level diagnostic blocks on their subject material and to design process maps of lessons with built-in three-level diagnostics.

Keywords: Developmental teaching, level-based learning model, pedagogical action strategy, student testing, SAM technology

Introduction

In recent years there has been much discussion about new educational outcomes for schools, but little discussion about how to achieve these outcomes. No matter what direction the teacher's role changes, he/she still plays and will continue to play a significant role in school education.

Problem Statement

A group of national experts in developmental teaching led by Elkonin and Nezhnov developed a level-based learning model that allowed the school itself to determine levels of quality in general education (Nezhnov et al., 2009). In order to assess these levels of cultural knowledge appropriation by school pupils, first of all, a special tool named SAM (Student Achievement Monitoring) was developed, which allows to determine the levels of students' learning material capturing at different stages of schooling (Nezhnov, 2018b). The proposed model distinguishes qualitatively different levels of mastering a cultural way of action:

• the first (formal) level is characterized by the fact that it allows action by example, without reliance on understanding the content of the concept underlying this way;
• the second (reflexive) level is shown only by those students who focus not on external, but on essential (entity) characteristics of the analyzed way of action;
• the third (functional) level assumes that students not only act with understanding, but also see the limits of the way, are oriented to the field of possibilities of application of this way.

Research Questions

In order for school pupils to reach the second and third levels of capturing knowledge, effective pedagogical practices are needed. And what do effective pedagogical practices mean? Who owns these practices in today's schools? And can teachers master such practices and solve today's educational problems?

Purpose of the Study

In order to answer these questions, two researches of adults (pedagogues) were conducted based on the development of this tool for assessing the level of educational outcomes of school pupils. One research was concerned with exploring different pedagogical strategies of teacher pedagogical actions and their connection with the educational outcomes of school pupils. The second research was related to the search for optimal ways to organize pedagogue professional development in mastering the level-based model of school pupil learning.

Research Methods

Findings

The primary expertise reveals the difficulties that teachers have in composing the blocks of tasks.

• More than 80% of the teachers in our experiments do not distinguish between qualitative levels (formal, reflexive, and functional) and levels of difficulty. As a result, the blocks of tasks compiled by them lose their diagnostic value: when tested, a decrease in the percentage of solvability is observed, but not due to the fact that students do not understand the concept at the reflexive/functional levels. All three tasks are not qualitatively different, and from the results of the diagnostics we can only say that students have difficulties in moving from simple to complex tasks. Thus, an important area of professional development for teachers is learning to equalize tasks by the level of difficulty within a single block (as a rule, it is recommended to set a minimum level of difficulty).
• Approximately 50% of teachers have difficulties with the structuring of learning content. This manifests itself in the fact that within one block the conceptual unity of the three tasks is not retained, as a result it is more difficult to diagnose the causes of students' difficulties. In the professional development courses we have to pay attention to the content analysis of subjects, the identification of cross-cutting content lines, key concepts.
• Difficulties are also caused by the distinction between functional level tasks and creative tasks. Since the professional development courses focus on distinguishing between the first (formal) level and the second (reflexive) level, we have no quantitative data about this difficulty. An indication that this difficulty is present is the inversion of the results during the transition from the second to the third level: instead of a decrease in the percentage of completion there is often an increase (sometimes a significant one). This means violation of the principle of leveling in the construction of the block of tasks, since those students who mastered the second (reflexive) level had to show statistically significant results at the third (functional) level. Inversion can mean that the proposed creative task can be solved in a non-normative way (intuitively, with reliance on worldly experience).
• Quite often teachers strive to move away from test-type tasks with a choice of one or more correct answers and formulate open-ended tasks (especially this concerns tasks of the third - functional - level). This facilitates the invention of tasks, but greatly complicates the testing procedure. Teachers should explain that open-ended tasks should be provided with all options of correct answers (detailed evaluation criteria should be attached to the task).

After the blocks have been revised and approved by the expert, teachers proceed to testing. Testing under different conditions may take place in different ways, in particular, the school may continue to work routinely, teachers enter the classroom for 15-20 minutes for testing. Less frequently, the testing requires an entire lesson.

Teachers record the results, indicating the percentage of completion of each level for each block of tasks. When analyzing the results, the experts adhere to the following approximate guidelines: first-level tasks should be solved by at least 70% of students, second-level tasks by about 50%, and third-level tasks by at least 15%.

The main thing the experts pay attention to is the absence of inversion (when more students solve higher-level tasks than lower-level ones). It is not uncommon for first-level tasks to be solved by 90-100% of students; this is quite acceptable. In elementary school it often happens that no one solves third-level tasks, which is also acceptable because the third level is not the target level in elementary school (in elementary school the target level is the second level).

In the case of a strong deviation from the set indicators (for example, very low percentage of solving first-level tasks, or too sharp difference between the first and second levels, or inversion of results), first of all the experts should return to the analysis of tested blocks and make sure that they meet the content criteria. Namely: all tasks of the block are based on the same concept (the same content of educational material), the tasks meet the criteria of the corresponding levels, the answers minimize the probability of guessing, the intuitive solution is excluded, etc. Then a more in-depth analysis of the textual design of the task is conducted, the reasons for low percentage of completion in complex or ambiguous formulations are looked for. The expert also draws attention to multiple-choice tasks, since quite often teachers do not specify exactly how many correct answers should be marked (this disorients students and leads to a sharp decrease in the percentage of completion).

Depending on the context in which the professional development courses take place, teachers are recommended different ways to revise the blocks of tasks. For example, it may be advisable to offer this block not to a class or a group, but to individual students (both who show high performance in a given subject and those who demonstrate low success). Discussion with students about how to solve these tasks often suggests the direction in which the block should be revised.

This specified sequence of stages in the development and testing of the SAM tool is not strictly set: testing and revising tasks can be done many times. In our practice there were cases when some blocks of tasks were developed, for example, for students of the 4th-5th forms, and testing showed that they were accessible only for students of 7th-8th forms (this was found out by testing in all age groups).

Special attention should be paid to the professional development of teachers engaged in developmental teaching (the system of D. B. Elkonin - V. V. Davydov), which are equipped with special means of formation and diagnostics of learning activity (Vorontsov & Lvovsky, 2020).

Conclusion

Our research has shown that SAM methodology is expedient to use not only in teaching teachers to develop and apply three-level tasks in pedagogical practice, but also for designing lessons with built-in diagnostics. The design, subsequent implementation, and analysis of a lesson with built-in diagnostics should be presented as a detailed and multidimensional process. The main tool is a special lesson process map (LPM) of the activity-based type, which is fundamentally different from the conventional LPM, primarily because of its non-linearity. That is why it is literally a map and not a preconceived plan.

The LPM focuses on several key points. First, it contains a task (let's call it conditionally a learning task), which must lead the participants of the lesson to a new way of action (in SAM logic, it may be a second or third level task). In many cases this "learning task" is preceded by a task which allows the teacher to evaluate the class's readiness to the next step (it may be a task which evaluates both curricular and meta-disciplinary competencies). The culmination of the lesson in the task-based approach should be the situation of problematization, in the design process it is necessary to provide different routes leading to the problem situation, as well as those indicators using which the teacher can promptly assess the children's actions in this situation. The map should include various subtasks, which allow, first, to slow down the general movement and contribute to the awareness of most students of the problematic situation, and secondly, help to find a way out of the problematic situation.

In order for LPM with built-in subject and meta-disciplinary diagnostics to become a tool for teachers to master ("make their own") the activity-based approach to learning, a number of conditions must be met. LPM must not become a formal demonstration product; it is usually designed in teamwork of several teachers under the guidance of a trainer-technologist "for themselves" (not for controlling bodies, not for satisfying someone's formal requirements). LPM is not needed in every lesson; its design is no simple matter, but it is important for the successful progress of a teacher on the path of activity-based pedagogy. The trainer-technologist should strive to ensure that at some point the teacher starts to give productive lessons that include activity-based educational practices, acting spontaneously and effectively without the "scaffolding" of LPM.

LPM is contradictory: on the one hand, it implies both certain actions of the teacher and certain actions of the student (thus limiting the teacher's freedom of maneuver), but on the other hand, it should be so variable as to become a teacher's assistant in various situations that may arise during the implementation of activity-based practices. In fact, LPM is a powerful accelerator for the emergence of new pedagogical intuition, which is nothing less than curtailed experience. Designing, conducting, and subsequent analysis of a lesson with feedback (a lesson where a teacher sees, feels, and understands students' actions in a problem situation) brings together, curtails, and twists key pedagogical skills: to set a "learning task" and organize the process of its solution, to select and use subject tasks of different levels as per SAM as formative and diagnostic ones, to organize metadisciplinarity assessment and development situations.

Acknowledgments

The SAM tool was created with financial support from the World Bank and informational and organizational support from the Center for International Cooperation in Education Development (CICED).

References

• Agapov, A. M., & Lvovsky, V. A. (2018). Formirovanie novyh kompetencij pedagogov v usloviyah perekhoda na deyatel'nostnyj podhod [Development of new teacher competences in the conditions of transition to activity-based approach]. Obnovlenie soderzhaniya osnovnogo obshchego obrazovaniya: Teoriya i praktika [Renovation of the contents of general education: theory and practice] (pp. 126-136). Avtorskij klub.

• Agapov, A. M., Mysina, T. Yu., Zotov, S. V., Lvovsky, V. A., Goncharova, M. A., Reshetnikova, N. V., & Ushakova, E. G. (2018). Diagnostika i formirovanie novyh obrazovatel'nyh rezul'tatov: rukovodstvo dlya uchitelya [Diagnosis and the formation of new educational results: a Guide for teachers]. Avtorskij klub.

• Khakhalina, O. M., & Nezhnov, P. G. (2021). Evaluation of educational achievements on the transition to the basic school. In V. A. Lvovsky, (Ed.), Sbornik materialov III s’ezda trenerov-tekhnologov deyatel'nostnyh obrazovatel'nyh praktik. III s’ezd trenerov-tekhnologov deyatel'nostnyh obrazovatel'nyh praktik [Collection of materials of the III congress of trainers-technologists of activity educational practices. The III congress of trainers-technologists of activity educational practices] (pp. 186-201). Avtorskij klub.

• Lvovskaya, A. N., & Nezhnov, P. G. (2021). Issledovanie svyazi mezhdu predmetnymi i metapredmetnymi obrazovatel'nymi rezul'tatami [Investigation of the connection between subject and meta-subject educational results]. In V. A. Lvovsky (Ed.), Sbornik materialov III s’ezda trenerov-tekhnologov deyatel'nostnyh obrazovatel'nyh praktik. III s’ezd trenerov-tekhnologov deyatel'nostnyh obrazovatel'nyh praktik [Collection of materials of the III congress of trainers-technologists of activity educational practices. The III congress of trainers-technologists of activity educational practices] (pp. 158-168). Avtorskij klub.

• Lvovsky, V. A. (2019). Indikatory stanovleniya uchitelya v deyatel’npstnoi pedagogike [Indicators of teacher formation in activity pedagogy]. In A. M. Aronov, & I. Yu. Plyasova (Eds.), Deyatel'nostnyj podhod v obrazovanii: Monografiya. Kniga 2 [Activity approach in education: Monograph. Book 2] (pp. 240-244). Avtorskij klub.

• Lvovsky, V. A. (2021). Put' uchitelya v deyatel'nostnuyu pedagogiku [The Teacher's Path to Activity Pedagogy]. Nauchno-pedagogicheskij zhurnal «Uchitel' Altaya» [Scientific and pedagogical journal “Teacher of Altai”], 1(6), 12-22.

• Lvovsky, V. A., Morozova, A. V., & Ulyashev, K. D. (2015). Deyatel’nostnyi podhod k perepodgotovke uchitelei. Praktika razvivayushchego obucheniya (seriya) [An activity-based approach to teacher retraining. (Series: The practice of developing learning)]. Avtorskij klub.

• Lvovsky, V. A., Mysina, T. Yu., & Ushakova, E. G. (2018). Model' podgotovki specialista po soprovozhdeniyu deyatel'nostnyh obrazovatel'nyh praktik [Model of Training a Specialist in Support of Activity-based Educational Practices]. MCU Journal of Pedagogy and Psychology, 2(44). 31-41.

• Lvovsky, V. A., & Sanina, S. P. (2018). Problemno-zadachnyi podhod k obucheniyu v shkole I vuze [Problem-task approach to teaching at school and university]. In S. N. Vachkova (Ed.), Sovremennoe obrazovanie v megapolise: vektory razvitiya [Modern education in megapolis: vectors of development] (pp. 75-88). Econ-Inform.

• Nezhnov, P. G. (2018a). Deyatelnostnuj podhod v obrazovanii: problemy i opory [Activity approach in education: problems and supports]. In A. M. Aronov, I. Yu. Plyasova (Eds.), Deyatel'nostnyj podhod v obrazovanii: Monografiya. Kniga 1 [Activity approach in education: Monograph. Book 1] (pp. 158-192). Avtorskij klub.

• Nezhnov, P. G. (Ed.). (2018b). Testy SAM (Student Achievements Monitoring) v obrazovatel’noi praktike [SAM (Student Achievements Monitoring) tests in educational practice]. Avtorskij klub.

• Nezhnov, P. G., Frumin, I. D., Hasan, B. I., & Elkonin, B. D. (Eds.). (2009). Diagnostika uchebnoi uspeshnosti v nachal’noi shkole [Diagnostics of educational success in primary school]. Razvivayushchee obrazovanie.

• Nezhnov, P. G., Gorbov, S. F., & Sokolova, O. V. (2018). Diagnostika uchebno-predmetnykh kompetenstii [Diagnostics of educational subject competencies]. Avtorskij klub.

• Slobodchikov, V. I., & Zuckerman, G. A. (1996). Integral’naya periodizatsiya obshchego psikhicheskogo razvitiya [Integral Division into Periods of General Psychic Development]. Voprosy Psychologii [Issues of psychology], 5, 38-50.

• Vorontsov, A. B. (2018). Formative assessment: approaches, content, evolution. A short guide to activity pedagogy. Part 1. Avtorskij klub.

• Vorontsov, A. B. (2019a). Instrument otsenki pedagogicheskhih deistvii [A tool for evaluating pedagogical actions]. In A. M. Aronov, & I. Yu. Plyasova, (Eds.), Deyatel'nostnyj podhod v obrazovanii: Monografiya. Kniga 2 [Activity approach in education: Monograph. Book 2] (pp. 234-239). Avtorskij klub.

• Vorontsov, A. B. (2019b). Unifikatsiya ili variativnost; v obshchem obrazovanii: problema vybora [Unification or variability in general education: the problem of choice]. MCU Journal of Pedagogy and Psychology, 3(49), 33-51.

• Vorontsov, A. B., & Lvovsky, V. A. (2020). Novye proekty razvivayushchego obucheniya [New projects of developing education]. Psychological science and education, 25(5), 83-94.