Does fraud occur in CA exams?

Security of online proctoring - Part 3 of the interview with Matthias Baume

How can in times of#CoronaCampus digital exams are taken? Online proctoring offers one approach to overseeing remote audits. In this article, our employee Gino Krüger interviewed Dr. Matthias Baume from the Technical University of Munich. This time, the focus is on the security of digital formats used by exam supervisors.This is the last part of our series of interviews on the subject of online proctoring, parts 1 and 2 of the interview can be foundhere.

How secure are digital forms of exam supervision? Image: [https://unsplash.com/photos/d9ILr-dbEdg/ Agence Olloweb]

University forum digitization: How can secure and reliable exams be achieved using online proctoring?

Dr. Matthias Baume:I think that we have to distinguish between two things in this context. Namely on the one hand the security and on the other hand the reliability that it really works. The latter depends in particular on proper planning and the forward-looking knowledge of all those involved. For this purpose, it is advisable, among other things, to design and conduct a demo exam in advance so that all those involved - especially the lecturers and students - can gain concrete experience with the system and develop a sound attitude based on it. In addition, such a demo test is a practicable option for determining any technical problems in advance and then solving them accordingly, so that technical reliability can be guaranteed.

On the other hand, it is about us choosing appropriate tools to ensure that the supervisory standards of the respective audit context are met. However, the decision for or against certain supervisory tools always involves a certain “balancing act”. Because the more tools are used in combination, the less likely it is that attempted fraud will go unnoticed. At the same time, however, the reliability of the test scenario also decreases because the system may disqualify participants from the test due to actually unproblematic behavioral patterns. This could happen, for example, if an automated system supervises the desktop activities of test participants and a tested person clicks away the update notification from Windows or the like. It is therefore of central importance that a robust set of supervisory instruments is selected for the respective audit scenario, i.e. one that ensures that the reliability and security standards do not collide with one another in as many audit cases as possible.

Another feasible approach could be to change our established audit approach so that fraud in the strictest sense becomes extremely difficult. Namely by taking exams - in the sense of so-called Open book exams - Conceived in such a way that a certain range of aids is permitted and the focus is more on transfer services than on memorization. Because this would make it possible to reduce the necessary supervision to the determination and guarantee of the identity of test participants. However, such examination formats are associated with a high degree of preparation and evaluation effort, so that they are more suitable for smaller cohorts. However, efforts are already being made to try to improve the scalability of open book exams by means of automation and modularization.

Is fraud more likely online than in the classroom? TUM is testing this in a 'cheating contest'. Image: [https://unsplash.com/photos/IBrzpD7scOA/ Gaëtan Werp]

Hochschulforum Digialisierung: How secure is online proctoring compared to regular on-site examination procedures?

Dr. Matthias Baume: From my point of view, a direct comparison is not that easy for several reasons. First of all, there is no 100% guarantee that there will be no cheating, either in conventional or online supervised exams. The fact is that there are few or no reliable studies on fraudulent behavior in online exams, while studies on fraudulent behavior in conventional examination contexts indicate that approx. 40-80% of the people surveyed already attempted fraud in some form have undertaken. Consequently, it can be assumed that fraud - without knowing more precisely about the success or failure of the attempted fraud - is not an exception in regular examination procedures. [1]

Furthermore, there are two different forms of online supervision and therefore also different security profiles. On the one hand, there are services in which supervision is carried out digitally by other people and, on the other hand, there are services in which supervision is automated. In the case of the former, security depends heavily on the person supervising and the specific number of test participants they monitor and is therefore subject to a high degree of variability. Of course, this also applies to the regular supervision of on-site examination procedures. An experienced supervisor can identify attempted fraud within a very short time and if sufficient human resources are available, this can also be guaranteed for examinations with a large number of participants.

With automated surveillance solutions, security now depends on the choice and combination of surveillance tools. If, for example, only a video recording is carried out via webcam, there is the risk that the blind spot of this observation mode will be recognized and systematically exploited in order to place a spicker or the like there. On the other hand, by coupling various AI-supported observation tools, you have the option of seamless and almost unlimited scalable supervision - provided that the necessary processing capacities function accordingly. After all, such systems are always online and have everything in view, while the focus of a human proctor is limited to individual participants.

At the Technical University of Munich we are currently conducting a study on fraud in online-supervised exams as part of a bachelor's thesis, the 'TUM Cheating Contest'. In this competition, the participants try to deceive the automatic supervision of a proctoring solution under controlled spatial conditions within the framework of a realistic test scenario and to cheat as inconspicuously and effectively as possible with unauthorized aids. The methods used are then precisely analyzed and documented together with the participants. And of course there is also a prize at the end for the best and most creative attempts at deception! The results of the study are expected to be available in September 2020.

Overall, it can be said that every procedure has its pros and cons and therefore no general statements can be made at the moment about whether online supervision is significantly safer or less secure than regular on-site procedures. Therefore, such judgments have to be patient until there are reliable studies that give a clear picture in this regard.

Demo exams help to identify technical uncertainties in advance. Image: [https://unsplash.com/photos/kwzWjTnDPLk/ NESA by makers]

Hochschulforum Digialisierung: What uncertainties arise with online proctoring and how can they be reduced?

Dr. Matthias Baume:First of all, the uncertainty of the identity of test participants must be dealt with. Because the exams are not taken on campus, there is uncertainty as to whether the person who registered for the exam is the one who will ultimately take the exam. This uncertainty can be handled in such a way that an authentication phase is always carried out before each test. In this, the test participants must visibly place a recognized identification document next to their face so that a webcam can be used to determine whether it is the same person.

In addition, there are technical uncertainties that feed from the ongoing functionality of the technical infrastructure. These can be reduced by carrying out demo tests in advance with exactly the same technical framework in order to identify and resolve any problems at an early stage. In addition, online proctoring providers usually offer analysis options that can be used to test in advance whether the computer used is sufficiently powerful and has the necessary software, the camera and microphone are functional and the available downstream and upstream of the Internet are sufficient etc. to run an exam smoothly. This is not always 100% successful, but the aforementioned analytical instruments have now matured to such an extent that they allow reliable diagnoses in the vast majority of cases. In this context, it is also advisable to develop a protocol for dealing with malfunctions during the respective test scenario, so that an ongoing test does not immediately lose its validity due to brief disruptions - e.g. if the Internet connection breaks down briefly. In addition, some online proctoring providers also provide live support, which can be consulted during the test in order to deal with any problems of a technical nature.

Much of what would be potentially verifiable nowadays is now also regularly tested in advance and supervised during the test. However, this has the consequence that more and more interventions in the hardware of the end user are necessary and therefore require authorization. Which in turn brings with it the uncertainty that the interventions, due to their scope and / or nature, appear unacceptable to end users. Against this background, it is important to systematically determine in a dialogue with the students which interventions are eligible for approval for examination purposes.

Another uncertainty arises from the classification of the behavior of test participants as 'fraudulent' or 'suspicious'. Because beyond obvious behavior - e.g. using a prohibited source of information or interacting with third parties - it is an open question whether and how a certain activity should be interpreted as attempted fraud or unproblematic behavior. After all, a multitude of unconscious behaviors - such as staring into the air for a moment or scratching your head - could be misinterpreted as attempted fraud. Furthermore, this problem arises regardless of whether one works with human or machine online supervision. It just varies the type and timing of the decision. To deal with this uncertainty, one could fall back on studies that describe the different characteristics and facets of human behavior in exam situations in order to condense a well-founded and comprehensible catalog of unproblematic behavior patterns. But even in such a way, the aforementioned uncertainty cannot be completely resolved. [2]

Ultimately, it is important to deal with the uncertainty of the examination conclusion, because it is not uncommon for an examined person - mostly out of nervousness and the like - forgets that the submission of the examination must be explicitly confirmed and the examination performance thus rendered becomes invalid. A problem that we regularly observed in Moodle. However, this uncertainty can be handled by automating the submission of the test.

HFD: Thanks for the interview!

To the first and second Part of the interview on the HFD blog. In these 6 recommendations the findings from our series on the subject of online proctoring are summarized.

 

[1] See also: Rodchua, Suhansa (2017): Effective Tools and Strategies to Promote Academic Integrity in e-Learning. In: IJEEEE 7 (3), pp. 168-179. Available online at http://www.ijeeee.org/vol7/424-EE005.pdf , last checked on July 3rd, 2020 and Sheard, Judy; Dick, Martin (2012): Directions and dimensions in managing cheating and plagiarism of IT students. In: Computing Education 2012.

[2] see Kolski, Tammi; Weible, Jennifer (2018): Examining the Relationship between Student Test Anxiety and Webcam Based Exam Proctoring. In: Online Journal of Distance Learning Administration 21 (3).