Validity and reliability are central criteria in ensuring the utility of assessment. Besides validity and reliability, van der Vleuten’s (1996) utility model includes the impact on learning, the acceptability of assessment to stakeholders, and the cost of assessment in terms of required resources.
In the proposed keynote I will focus on the validity of e-assessment related to Mathematics courses, supported by learning design (LD) and learning analytics (LA). Our team has developed a comprehensive LD concept and a tool (learning-design.eu) based on research findings, to support balanced LD planning (BDP). The student-centred BDP concept and tool inter alia focus on learning outcomes (LOs) and student workload, ensuring constructive alignment and assessment validity, enhancing LD by using LA. I will present how the BDP concept and tool are applied in Mathematics courses.
The assessment programme consists of continuous formative and summative assessment tasks, which have been performed in an online setting since the onset of the COVID-19 pandemic. The formative tasks are individualized for each student. Assignments include computational tasks, assigned to students by random selection from the assignments database on Moodle, prepared by the course teachers. Unlike the assignments, quizzes are focused on understanding concepts, basic terminology, and solving tasks that help students comprehend concepts. The quizzes present automated formative assessment, with an automated grading system and feedback. The weekly formative assessment serves as a preparation for exams that are taken three times per semester. Students can use tools in calculation, but should provide a theoretical explanation of the procedures and results of the computational tasks. Relatively large databases (several thousands of assessment items) are used, with exercises and questions for formative and summative assessment. The exercises in the databases are classified according to the mathematical topic and three levels of difficulty based on the Math KIT taxonomy (Cox, 2003). Based on that, a student gets an individual exercise, randomly chosen from a certain task category that usually contains between 50 and 500 different, even though similar exercises. Most of these categories have been populated with automatically generated tasks, programmed in Python and R. To a certain extent, this prevents copying the results and cheating on exams. Besides the above-mentioned tasks, a part of the assessment programme is a problem-solving, project-based learning and/or essay-writing task.
The analysis of students’ results suggested that types and levels of assessment tasks were well aligned with the intended LOs. It showed a high level of alignment between the relative weights assigned to LOs based on their importance, and actual assessment points, but also pointed to certain limitations related to modes of delivery and available educational resources.
References:
- Cox, W. (2003). A MATH-KIT for engineers. Teaching Mathematics and Its Applications: An International Journal of the IMA, 22(4), 193–198. https://doi.org/10.1093/TEAMAT/22.4.193
- Divjak, B., Grabar, D., Svetec, B. & Vondra, P. (2022). Balanced LD Planning: Concept and Tool. Journal of Information and Organizational Sciences, 46(2). Available: https://www.researchgate.net/publication/359320864_Balanced_Learning_Design_Planning_Concept_and_Tool
- Divjak, B., Žugec, P., & Pažur Aničić K. (2022). E-assessment in mathematics in higher education: a student perspective. To be published in the International Journal of Mathematical Education in Science and Technology.
- van der Vleuten, C. P. M. (1996). The assessment of professional competence: Developments, research and practical implications. Advances in Health Sciences Education, 1(1), 41–67. https://doi.org/10.1007/BF00596229