Cheating, ClickMaps, and the importance of question randomisation: Case studies in the 2024 MRCP(UK) written examinations
Chris McManus (UK)1; Liliana Chis (UK)2; Albert Ferro (UK)2; Ben Gillon (UK)2; Stuart Hood (Thailand)2;
1 - University College London; 2 - Federation of the Royal Colleges of Physicians of the United Kingdom;
Keywords: Cheating; Computer-based testing (CBT); Remote online proctoring (ROP); ClickMaps; Question order randomisation;
Themes: Assessment misconduct
Format: Short Presentation
Type of Presentation: Short Presentation

Background

Cheating has always occurred in examinations, and new technologies provide newer ways of cheating. Cheating, such as copying and collusion, threatens exam validity, and Boards of Assessors must detect anomalous and suspicious events. Computer-based testing (CBT) can aid detection in the form of ClickStreams and ClickMaps [1]. CBT also allows randomisation of question (and answer) order which should make cheating more difficult. Here we describe a ‘natural experiment’ in which a test provider inadvertently failed to randomise question order and several cheating incidents occurred.

Summary of Work

Eight diets of MRCP (UK) were taken in 2024, four of Part 1 and four of Part 2, using CBT for almost all candidates, sometimes in invigilated centres, and sometimes on candidates’ own computers with remote online proctoring (ROP). The program AcinonyxR [2] showed a case of probable cheating in an ROP diet of Part 2, and ClickMaps then revealed that question order had not been randomised. Two diets of Part 1 were also affected, with four events occurring involving nine candidates in invigilated centres.

Summary of Results

In the cases where anomalous pairs were detected by AcinonyxR, detailed examination of ClickMaps, as well as of the physical layout of invigilated centres, provided insights into probable mechanisms of cheating. It is important to note that there were no anomalous events in the five other 2024 diets in which question order randomisation was implemented properly, showing the importance of randomisation of question order.

Discussion & Conclusion

Randomisation of question order in CBT is important for reducing the prevalence of cheating in online examinations, and ClickMaps allow investigation of mechanisms. Although invigilation in centres is widely believed to be more secure than ROP, cheating was detected amongst candidates taking the examinations in  both modalities.

Take-Home Message

Question order randomisation is an effective method for preventing cheating in computer-based testing, whether in-centre or ROP, and new technologies allow better analysis of candidate behaviour.

References

1.    McManus IC et al: Visualising candidate behaviour in computer-based testing: Using ClickMaps for exploring ClickStreams in undergraduate and postgraduate medical examinations. medRxiv 2023:2023.2006.
2.    McManus IC et al: Detecting cheating in written medical examinations by statistical analysis of similarity of answers BMJ 2005, 330:1064-1066.