Many teachers have recognized the rapid development of artificial intelligence (AI) and its abrupt appearance as a threat to the fairness and reliability of assessment. Some of them see the opportunity for change and are introducing more meaningful assessment programs. In general, both groups have to respond to the contemporary AI challenge.
Assessment in mathematics usually consists of two types of tasks: routine tasks and problem-solving tasks. Routine tasks can be part of summative assessment today only with a strict proctoring system and instructing students about academic integrity and consequences of misconduct.
On the other hand, problem-solving tasks can still be valuable part of student assessment in mathematics in at least two formats: smaller problem-solving tasks that can be part of a time-limited summative test, and more demanding tasks that students work on at home and then report on in the form of an essay or an oral presentation.
The latter has been our major topic of interest related to the assessment of IT students in mathematical courses. Our main idea has been to give students an opportunity to openly partner with a chatbot and critically analyse the results.
Students are given a description of their individualized problem-solving exercises. A typical exercise consists of theoretical background research, and based on that, students have to solve a (not too difficult) problem.
The learning outcomes linked to that task include individual problem-solving (recognizing problem-solving phases, trying to generalize results, or coming to a result based on mathematical deduction), but also structuring a mathematical text, using tools for mathematical text editing (Latex) and checking results, as well as correctly listing and correctly using references.
This year we are giving additional instructions to students on how to use ChatGPT (or a similar chatbot) to get into the theoretical part and basic references. They need to report on the results of their interaction with the chatbot, detect potential mistakes, and further research a mathematical topic to solve a problem. We are asking students to be careful with terminology (Croatian standard mathematical terms), definitions and theorems.
Finally, students are supposed to analyse the problem solution(s) as well as critically evaluate of their interaction with the chatbot.
Students’ essays have been assessed using the scoring rubric incorporating all the above-mentioned criteria. We have also collected feedback from students related to this exercise.