ABSTRACT

Online assessment in medical education is relatively new and there is a paucity of literature available on this topic. The objective of this scoping review was to conduct a systematic search of the published literature on the challenges of conducting online assessments in undergraduate medical and dental curricula. The literature was searched in the recommended electronic databases. The search terms used were “online assessments, e-assessments, computer-assisted assessments, online assessment challenges and opportunities, online assessments in medical education, reliability and validity of online assessments, and online assessments during COVID-19 pandemic” in PubMed Central, Google Scholar, Wiley Online Library, SpringerLink, and Ovid. After screening, 21 full-text articles were included in this scoping review. The data were charted independently and synthesized to present the findings.

Most of the studies included in this review were published in the year 2020 and 2021(till September). Half of the studies (50%) reported that cheating and plagiarism are the most common challenges in online assessments, while a similar number of studies highlighted the issues faced by the teachers in making and conducting online assessments. Similarly, 22% of the studies emphasized the concerns raised by the students, while another 22% reported the technical and financial issues faced by the institutions in conducting online assessments. It was revealed that during COVID-19 the biggest challenge for the educational system was to switch to online assessments, as the educational system was not ready for it. Few studies (16%) reported that the issues of reliability and validity of online assessments are still to uncover.

This scoping review may help to lay the foundation for more empirical future studies on the challenges of conducting online assessments in the undergraduate medical and dental curricula in Pakistan.

Keywords:

Online assessment, challenges, medical education, dental education, curriculum.

Introduction

Assessment is the heart of medical education and a core component of learning. It is a fundamental part of an educational experience and students cannot spare themselves from the effects of poorly designed assessments.¹ Due to the expanded growth and penetration of new digital technologies and the Internet, the opportunities for adopting online assessment methods are increasing.² The purpose of online assessment is not different from traditional assessments, both are used to assess whether the students have achieved the learning outcomes or not, to provide effective feedback to students, and for grading or awarding degrees and certifications to students.¹ Online assessments mainly focus on three domains of learning, i.e., cognitive, affective, and behavioral.³ Online assessment tasks, if designed appropriately, have the potential to assess students’ understanding, knowledge, and skills.⁴ The advantages of online assessments are that it provides instant feedback to learners; instant monitoring of students’ performance is possible; less or no paperwork is needed; and automated marking not only reduces the burden on the faculty and administration but also turns out to be time effective.¹

In medical education, the assessment is mostly conducted offline because it is experiential learning and most assessments are carried out using direct observations.¹ Online assessment is mostly used for formative purposes in medical education whereby effective feedback is provided to the students to improve their learning.¹ There are multiple methods used for formative online assessment; some of these are short answer questions, true or false questions, multiple-choice questions (MCQs), and matching questions. The most commonly used tool is MCQs.⁵ For summative purposes, online assessments are restricted to “digital portfolios.”¹ Online assessment should be designed in such a way that it encourages a deep approach to learning in students.⁶ The application of online assessment depends on how assessment results are used to enhance students’ learning, including the mode of feedback to the students and adjustment and revision of teaching strategies to respond to students’ learning needs and issues.³ But the current COVID-19 crisis, throughout the world, has opened the doors for online assessments in all professions, including medicine.¹ In the medical profession, it is not possible to assess every domain online but an attempt can be made to assess a larger part through online assessments during this crisis period.¹ Online assessment in medical education is relatively new.¹ Most of the faculty and medical students are not aware of the mechanisms and strategies of delivery of online assessments, therefore requiring appropriate relevant training of the faculty and students.¹ There is also a lack of a reasonable number of published research data on this topic, which leads to the slow adoption of online assessment methods in the medical profession.²

Some countries with advanced technology and people having great command of information technology (IT) are already working on e-learning, but it is a huge challenge to introduce an online learning system in a third-world country like Pakistan. The biggest challenge was the lack of proper IT support along with training of the staff.⁷ In Pakistan, it was a whole new experience even for the teachers to deliver the lectures and conduct assessments online. There were no proper guidelines or training provided to the teachers or students, and Internet issues, especially in rural areas, were a huge obstacle.^8,9

Like many other countries, the universities in Pakistan also opted for Zoom, Microsoft Teams, and other software for online learning to facilitate the students, but the medical and dental students were unable to attend the clinical training component through such tools, thus raising a grave concern regarding their competencies.¹⁰

This scoping review enables us to understand the challenges that arise during the conduction of online assessments in undergraduate medical and dental education in Pakistan. A systematic search of the published literature on the challenges to conduct online assessments in undergraduate medical and dental education was carried out to identify and address the local challenges, thus attempting to run and implement the online assessment smoothly among our institutes.

Methods

The methodology of this scoping review was based on the framework outlined by Arksey and O’Malley that is cited in literature.^{11, 12}

Identifying the research question

The research question guiding this scoping review is “What are the challenges for the institutions and faculty for conducting online assessments in undergraduate medical and dental education?”

Identifying relevant studies

A robust search was carried out to search the relevant studies on this topic. Data were searched using recommended electronic databases: PubMed Central, Google Scholar, Wiley Online Library, SpringerLink, and Ovid. The authors initially selected 38 articles published on the online assessment from 2012 till September 2021 (Figure 1). The search terms used were “online assessments, e-assessments, computer-assisted assessments, online assessments challenges and opportunities, online assessments in medical education, reliability and validity of online assessments, online assessments during COVID-19 pandemic.” These terms were taken from medical subject headings. All the searched articles were collected in Mendeley software. Duplicate citations were removed manually.

Study selection

The researchers included cross-sectional studies, both quantitative and mixed methods. The inclusion criteria were the articles written in English language, both review and original articles with full text available. Books, book reviews, commentaries, letters to the editor, newspapers, dissertations, and conference proceedings were excluded.

Charting the data

The data were charted independently. Arksey and O’Malley’s “descriptive analysis” approach was applied to data extraction, summarizing information from the selected articles and recording the data. This helped to analyze the selected articles through a common framework. The review process was iterative, i.e., the authors added and edited rows as necessary throughout the process. The general characteristics of studies included in this review are given in Table 1.

Figure 1. This bar chart shows the number of studies included in the review that were published from 2012 till September, 2021

Table 1. General characteristics of the studies included in the review.

Collecting, summarizing, and reporting results

The results were collated, summarized, and reported. Furthermore, the information extracted from the studies were refined systematically into manageable groups.

Results

The researchers retrieved a total of 38 articles by using 5 electronic databases, out of which 9 (24%) articles were from PubMed, 10 (26%) from Google Scholar, 9 (24%) from Wiley Online Library, 5 (13%) from SpringerLink, and 5 (13%) from Ovid. Six duplicate articles were removed manually (n = 32). Twenty-nine articles were identified after the first round of screening for title and abstract. However, five articles containing irrelevant and insufficient information were excluded (n = 24). In the second round of screening, articles were assessed for eligibility based on full text and 26 articles were retrieved. Five articles were excluded during data extraction because of disconcordance with the inclusion criteria (n = 21). Finally, a total of 21 relevant, full-text articles were included in this scoping review according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines (Figure 2).

About 50% of the studies included in this scoping review revealed that the most commonly reported challenges in the online assessment were cheating and plagiarism. About 55% of the studies highlighted the issues faced by the teachers in making and conducting online assessments; 22% of the studies confirmed that students’ performance is affected during online assessment due to technical problems and lack of digital skills and students feel stressed and anxious. Another 22% of the studies reported the challenges faced by the institutions which also included the huge financial burden on the institutes for conducting online assessments, while 33% of the studies highlighted the technical and electrical challenges due to different reasons while retaining the online connections during online assessment. A study revealed that it was the biggest challenge for the educational system to switch to online assessments during COVID-19, as the educational system was not ready for it. Furthermore, 16% reported that the issues of reliability and validity of online assessments are still to uncover.

Figure 2. PRISMA-ScR flow diagram demonstrating the articles selection process.

This scoping review reveals a developing body of literature outlining the challenges of online assessments in medical education. According to the literature reviewed, the most commonly reported challenge in online assessments is the issue of academic misconduct which includes cheating and plagiarism.^1,4,5,13-16 The increase in cheating and plagiarism is due to the ease of copy and paste options, the easy access to the cheating literature, the students can easily purchase materials and then claim it as their own work, etc.^13,14 To prevent or avoid fraud in online assessments is a main challenge.¹⁷ Most of the teachers do not have a clear concept and understanding of the nature of online assessment.³ Teachers find it very difficult to assess affective domains and skills through online assessment.¹ Designing online assessments is very time-consuming for the teachers⁶ as most of them do not have the necessary digital skills to deliver, conduct or manage feedback to a large number of students at a time during online assessment.^18,15 It is challenging for a teacher to solve a student’s problem which may be either due to software or hardware malfunctioning or inability to understand the question or the supporting material during online assessment.^15,19,20 Most of the teachers show resistance to adopting online assessment methods, particularly for high stake summative exams ²¹ as they do not trust the results of online assessments.²² Students suffer from anxiety during the online assessment because of the apprehensions related to the technical issues that adversely affect their performance.^2,4,6,14 The biggest challenge for an institution in online assessment is to protect assessment from viruses and hackers¹ and to ensure data secrecy and authentication of student’s identity and maintaining his/her privacy as the student is not face to face.^1,6,13,16,13 The huge cost of online assessment is a burden on the institutions as it requires a well-established infrastructure setup, servers, software licenses, well-trained support staff, a large number of electronic devices, and adequate space.^{1,11,19,22,23} Online assessments may also result in breach in security and loss of data which is potentially disastrous.²³ Students can deliberately crash or break their Internet connection to gain extra time or to retake an assessment.⁶ The objective of this study was elaborate the issues and challenges faced by the academic community in the field of medical and dental teaching and assessment in the local and international context. More studies can further explore the impact, reliability, and validity of online assessment in medical education.^2,21,24

This study concludes that the majority of the developed world has progressively shifted or is in a process to shift to online assessment because of adequate resources. Developing countries like Pakistan are still struggling to shift their academic activities to online learning mode; however, this transition appears tremulous and less productive. Medical and educational institutes in Pakistan, during the COVID era, were not prepared to embrace this demand due to the lack of adequate material, facilities, and training of the teaching faculty. The study highlights the challenges identified that should be sought out for the smooth implementation of routine online assessments and to enhance the quality of online teaching from the experiences of the developed world.

Limitations of the study

The scoping review does not provide answers to a specific question or give a synthesized result; it only provides an overview of the available literature. Furthermore, the paid articles without full texts and the articles in a language other than English were not retrieved. The quality of published articles included in the study was not assessed, except for their availability in the recommended search engines.

Future recommendations

• Studies should be conducted on how to achieve academic integrity in online assessments in the medical profession in Pakistan.
• Studies should be conducted on the reliability and validity of online assessments in medical education in public and private sector medical universities.
• Additional studies should be conducted to compare online assessments with face-to-face assessments in medical education in Pakistan.

Conflict of interest

None to declare.

Grant support and financial disclosure

None to disclose.

Ethical approval

Not required.

Authors’ contributions

GMJB: Concept and design of the study, acquisition, analysis of data, and drafting of the manuscript.

JS and SAJB: Interpretation of data and drafting of the manuscript.

MI and MSB: Acquisition of data and critical revision with intellectual input.

ALL AUTHORS: Approval of the final version of the manuscript to be published.

Authors’ Details

Gul Mehar Javaid Bukhari¹, Javeria Saleem², Syed Abdullah Javaid Bukhari³, Muhammad Ishaq⁴, Muhammad Salman Butt⁵

1. Assistant Professor, Department of Community Medicine, Federal Medical College, Islamabad, Pakistan
2. Assistant Professor, Department of Public Health, University of the Punjab, Lahore, Pakistan
3. Consultant Medical Oncologist, Oncology Department, Shaukat Khanam Hospital, Lahore, Pakistan
4. Assistant Professor, Institute of Social and Cultural Studies, University of the Punjab, Lahore, Pakistan
5. Clinical Trial Coordinator, Department of Public Health, University of the Punjab, Lahore, Pakistan

References

1. Joshi A, Virk A, Saiyad S, Mahajan R, Singh T. Online assessment: concept and applications. J Res Med Educ Ethics. 2020;10(2):49–59. https://doi.org/10.5958/2231-6728.2020.00015.3
2. Hewson C, Charlton JP. An investigation of the validity of course-based online assessment methods: the role of computer-related attitudes and assessment mode preferences. J Comput Assist Learn. 2019;35(1):51–60. https://doi.org/10.1111/jcal.12310
3. Pu S, Xu H. Examining changing assessment practices in online teaching: a multiple-case study of EFL school teachers in China. Asia-Pacific Educ Res. 2021. https://doi.org/10.1007/s40299-021-00605-6
4. Northcote M. Online assessment: friend, foe, or fix? Maria Northcote. Br J Educ Technol. 2002;33(5):623–6. https://doi.org/10.1111/1467-8535.00335
5. Hassan B, Shati AA, Alamri A, Patel A, Asseri AA, Abid M, et al. Online assessment for the final year medical students during COVID-19 pandemics; the exam quality and students’ performance. Onkol i Radioter. 2020;14(6):1–6.
6. Mate K, Weidenhofer J. Considerations and strategies for effective online assessment with a focus on the biomedical sciences. FASEB Bio Adv. 2021. https://doi.org/10.1096/fba.2021-00075
7. Manan S, Daterdiwala NF. Challenges of online medical education in Pakistan during COVID-19 pandemic. J Pak Med Assoc. 2022;72(6):1267. https://doi.org/10.29271/jcpsp.2020.Supp1.S67
8. Ansar F, Ali W, Khattak A, Naveed H, Zeb S. Undergraduate students’ perception and satisfaction regarding online learning system amidst COVID-19 Pandemic in Pakistan. J Ayub Med Coll Abbottabad. 2020 [cited 2022 Jul 3];32(1)(4):S644–50. Available from: http://www.jamc.ayubmed.edu.pk
9. Sabqat M, Aziz A, Mirza TI, Adeel B. Impact of COVID-19: faculty suggestions regarding online medical teaching. J Ayub Med Coll Abbottabad. 2022;34(2):304–8. https://doi.org/10.55519/JAMC-02-8997
10. Latif MZ, Hussain I, Saeed R, Qureshi MA, Maqsood U. Use of smartphones and social media in medical education: trends, advantages, challenges, and barriers. Acta Inform Medica. 2019;27(2):133–8. https://doi.org/10.5455/aim.2019.27.133-138
11. Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5(1):1–9. https://doi.org/10.1186/1748-5908-5-69
12. Sucharew H, Macaluso M. Methods for research evidence synthesis: the scoping review approach. J Hosp Med. 2019;14(7):416–8. https://doi.org/10.12788/jhm.3248
13. Reedy A, Pfitzner D, Rook L, Ellis L. Responding to the COVID-19 emergency: student and academic staff perceptions of academic integrity in the transition to online exams at three Australian universities. Int J Educ Integr. 2021;17(1):1–32. https://doi.org/10.1007/s40979-021-00075-9
14. Okada A, Whitelock D, Holmes W, Edwards C. e-Authentication for online assessment: a mixed-method study. Br J Educ Technol. 2019;50(2):861–75. https://doi.org/10.1111/bjet.12608
15. Guangul FM, Suhail AH, Khalit MI, Khidhir BA. Challenges of remote assessment in higher education in the context of COVID-19: a case study of Middle East College. Educ Assess Eval Acc. 2020;32(4):519–35. https://doi.org/10.1007/s11092-020-09340-w
16. Meccawy Z, Meccawy M, Alsobhi A. Assessment in ‘survival mode’: student and faculty perceptions of online assessment practices in HE during COVID-19 pandemic. Int J Educ Integr. 2021;17(1):1–24. https://doi.org/10.1007/s40979-021-00083-9
17. Almeida F, Monteiro J. The challenges of assessing and evaluating the students at distance. 2021;5(1):3–10. Available from: http://arxiv.org/abs/2102.04235
18. Raaheim A, Mathiassen K, Moen V, Lona I, Gynnild V, Bunæs BR, et al. Digital assessment–how does it challenge local practices and national law? A Norwegian case study. Eur J High Educ. 2019;9(2):219–31. https://doi.org/10.1080/21568235.2018.1541420
19. Alruwais N, Wills G, Wald M. Advantages and challenges of using e-assessment. Int J Inf Educ Technol. 2018;8(1):34–7. https://doi.org/10.18178/ijiet.2018.8.1.1008
20. Kearns L. Student assessment in online learning: challenges and effective practices. JoltMerlotOrg. 2012;8(3):198–208. Available from: http://jolt.merlot.org/vol8no3/kearns_0912.htm
21. Hewson C. Can online course-based assessment methods be fair and equitable? Relationships between students’ preferences and performance within online and offline assessments. J Comput Assist Learn. 2012;28(5):488–98. https://doi.org/10.1111/j.1365-2729.2011.00473.x
22. Perwitasari F, Astuti NB, Atmojo S. Online learning and assessment: challenges and opportunities during pandemic COVID-19. Proc Int Conf Educ Assess Policy. 2021;545(4):133–7. https://doi.org/10.2991/assehr.k.210423.077
23. Er HM, Nadarajah VD, Hays RB, Yusoff N, Loh KLY. Lessons learnt from the development and implementation of an online assessment system for medical science programmes. Med Sci Educ. 2019;29(4):1103–8. https://doi.org/10.1007/s40670-019-00802-5
24. Arnold SD. Assessing student learning online: overcoming reliability issues. IADIS Int Conf Cogn Explor Learn Digit Age. 2012:189–96.