I spent a lot of time thinking about how to design my exam after a mid-semester pivot to online learning due to COVID-19. In the end, my exams this term are open book with no time limit. They have a due date but are open for a week or more. I am even open to extending it. Students can go in and out of the exam as many times as they want. I also provided them with a downloadable copy of the exam so they could work on it outside of Moodle. The response has been overwhelmingly positive and my decisions are founded in good pedagogy.
Integrity
Over the past couple of years, I have been fortunate enough to have engaged in reading, discussions, and professional development around cheating. Cambrian invited Dr. Lang to talk about his book and facilitated a book club where I had the opportunity to reflect on how environmental factors can prevent or encourage cheating.
The facts of this situation lead me to believe that there is a high risk of cheating if educators take a traditional and punitive approach to exam delivery. We cannot proctor in-person exams. Students are under unprecedented levels of stress. Lang (as cited by Killam, 2018) was very clear that most people will cheat if the circumstances are right. Students are desperate to pass and trying to manage so many things right now that these circumstances are definitely right for cheating.
The immediate reaction of faculty who hear how my exam is designed is to question academic integrity. I actually think that my approach to their exams decreases the chance of cheating. Instead of trying to police them, I have clarified my expectations and encouraged students to discuss the exam. They cannot cheat by talking about the exam or using their resources because the exam was designed to make them do both of those things.
Being explicit about what is considered cheating (or not) is something that Lang and the Cambrian Teaching and Learning Innovation Hub advocate for. Before constructing the exam, I began by developing a description for my year three students then adapted it for year one. They are quite similar but customized to the learning objectives and terminology of each year. Before attempting the exam students are forced to agree to each section in a Moodle quiz. I have also talked them through it and answered their questions in a live session and asynchronous video.
Design
My exams were designed not to test student recall but to assess how they would apply knowledge to practice. The questions prompt students to critically think about course content, discuss it, and answer the questions. In the course content that is posted on Moodle, students can find the information they need to answer questions. However, the questions mostly require students to use more than one piece of information to answer these questions. My intent is to create more meaningful dialogue among students that will help them internalize course content for future use in practice.
I have taken three courses on how to write NCLEX style questions recently, the last of which was on testing critical thinking. The exams contain a mixture of questions written at different difficulty levels. I made the difficult level the same as I would have for a proctored in-person exam. Why? There are many reasons, but the two that stand out are authenticity and fairness.
Authenticity
Authenticity in assessment is important. When I think about what I am testing, the most genuine way to assess if students can make good decisions is by allowing them the same access to resources they would have in practice. In one course the exam was already planned to be semi-open book. I was going to provide them with all the course readings in a Google folder during the exam. When I really thought about it, in the courses I teach open book exams make the most sense.
The questions on the exam also need to be well written and reflective of what they will see in practice. In clinical practice, my students will not be asked to simply recall information. The questions on any test in nursing should not ask students to simply “select the definition of resilience.” Instead, they should recognize and be able to apply concepts to practice. Yet, we need to be careful not to make the exam too hard.
Fairness
It is not fair to make it more difficult for students to pass a class in the midst of a pandemic. A key take away from the NCLEX courses I took was that assessment needs to match content delivery. I have been teaching at an application level, which means I can assess at that level. However, if your class is lecture and content-heavy it is not fair to now ask students to move up to an application-level without any guidance.
… open book does not automatically equate to inflated grades.
A very smart colleague of mine
Educators erroneously assume that just because an exam is open book students will excel at it. Early this semester I had the opportunity to watch year one students in a computer lab working on a quiz worth 5% of their final grade. I thought for sure they would ace the test because the collegial discourse I was overhearing in the lab showed that they were thinking through the questions. They had all the answers in two main course readings. I wanted them to ace it. The quiz was written at an application or higher level of Bloom’s taxonomy. I was wrong – they struggled. The average was low for several reasons, including stress, over-thinking test questions, lack of practice with application style questions, and the absence of test-taking strategies.
In the nursing program, we are directed to have our questions written at an application or higher level of Bloom’s taxonomy in order to prepare students for the NCLEX (the exam students write to become an RN). My observations of this test coupled with my observations of a third-year group of students writing an application or higher level exam have shown me that there should be a variety of question difficulties on a test. Even under normal circumstances, stress influences our ability to focus and think. High functioning individuals are struggling to maintain focus. Resist the urge in the midst of a pandemic to make too many test questions at the analysis or evaluation level of Bloom’s taxonomy. Application and analysis level questions prompt discussion and critical thinking.
Flexibility and Choice
To mitigate the various contextual factors that we as educators cannot even begin to predict, flexibility is needed now more than ever. Timing of exams may need to be adjusted because a student cannot focus because they are being kicked out of their home, they think they were exposed to COVID-19, their loved one is sick, or they are the sole caregiver of kids at home. Is it fair to make them do their exam on one specific day and time regardless of these circumstances? Is that a valid measure of student knowledge?
When I was a student I was a single mom with a good support system. If I picture myself in the position of a single mom of a toddler who is being forced to do an exam at 11:30 on a Wednesday that is time limited I am pretty sure that I would be unable to focus. Maybe the child is hungry constantly (my boys ask for food every 12 minutes), needs their show changed, is refusing to nap that day, or is sick. When educators pick a specific time for an exam it creates a lot of unknown issues. I can pretty much guarantee that one time will not work for everyone. Even if a single mother thinks that once their child goes to bed they can focus the child may unexpectedly get up once the mother starts the exam. Childcare is hard to find and risky to seek in the midst of a pandemic.
My students work in health care. They are not nurses yet, but some volunteer at the hospital, work in long term care or are ward clerks. Only after I released the plan for my exam did someone tell me that due to the pandemic long term care homes are expecting students to do more shifts than normal. Because of that students may be forced against their will to stay for a 12 or 16-hour shift when they were only scheduled for 8 hours. I use to work in long term care and I know that is true. That could severely impact a student’s ability to focus on an exam for three hours.
The only way to mitigate these problems is by allowing students choice. At the very least, let them choose the time of day they write an exam.
Laura Killam
The reason for limiting the amount of time a student has to complete a test is usually to prevent “cheating” by referring to resources or talking to a colleague. If the exam is designed in a way that accounts for collegial discourse and use of resources that problem is solved. I think an open book test with the least technology possible is the simplest solution – and simple solutions are normally better. Online proctoring tools cannot prevent 100% of cheating. Creating an exam were using resources and talking to colleagues is encouraged, eliminates the possibility of cheating. However, that may not always feel possible.
If you want your exam to be closed book one strategy to limit the use of resources or collaboration is to limit student time to complete the test. In this context that is still problematic. If your goal is to provide a fair and accurate assessment of knowledge there are many things to consider. If a student approaches you to say that they could not finish the test in the time allotted because their child started unexpectedly vomiting and they were the only adult in the house … what would you do? There are many reasons that a student may not be able to complete a test in an allotted amount of time. Maybe they have anxiety and a question triggers them. Our students are working without the normal supports that they have on campus. Is it equitable to give a student on a personal computer with two screens the same amount of time as one who is forced to work on their phone? I think the answer is no. Personally, I work much faster on larger screens.
Hypothetically, if a student can choose any time of day to do the test that opens there could still be some cheating. If the test is open for a full day there is a high risk that discussion about the test and use of resources will occur. The easiest way to deal with that is to design the test with that in mind. Ask yourself: Do you want students who are willing to cheat to score better than those who are not? If your goal is to be fair then it seems to me that the best solution is to permit the use of resources.
Now is not the right time to test new proctoring technology. I have used Lockdown Browser before and it may help, but it does not prevent all cheating. It causes a lot of problems that we cannot fix right now – just Google it and you will see. It may be a good solution if we plan ahead for using it, but now is not the time.
If you need another reason, consider this: Flexibility reduces the chance that a student can appeal a course due to contextual factors that we did not predict. If, like me, you want to know if students are competent in the course outcomes a flexible exam is the best way to know. It will prevent many appeals if a student is not happy with their grade.
Choice
I have some creation level exam questions that ask students to draw on scholarly literature to construct a paragraph (I teach research). In these questions, I have given students a limited choice about what they write about. Too much choice adds stress, but generally speaking, asking students to write about one of three things is helpful.
Construct a paragraph critiquing the use of two different specific statistical tests in [insert citation]. Be sure to name the different statistical tests used in this study during data analysis. Critically evaluate the appropriateness of the statistical tests.
You must cite your ideas and are expected to use external resources. You may use course readings. Integrate scholarly writing techniques of a clear topic sentence, logic, flow, and a clear concluding sentence. A reference list is not needed in APA format, but if they are not course resources please provide a link or rough reference list entry.
Example Question
PS: I also clearly articulate how they will be graded on these questions. The guide is based on Bloom’s taxonomy but I am not sure if I can post it.
Students Matter
I believe that allowing students to do the exam all at once or in segments not only reduces their stress but helps them focus more. It is simply a more accurate way to assess knowledge. More importantly, it demonstrates professional caring in a program that is founded on caring as a theoretical framework. Students need to know that we care about them not only their success. The reaction of my students to my exam design has overwhelmingly positive – some version of “Thank you this means so much.” By demonstrating that I care students have reached out to me and talked about their context in unprecedented ways. I feel that is part of an educator-student relationship that promotes resilience and role models what a professional nurse should be.
Trust
This exam design requires that I trust my students. I need to trust that they see the value in my course content and follow the guidelines I suggested. I also need to trust that they will do as I asked and delete the offline version of my exam when the course is over. I am not going to police them. Instead, I am going to trust that over the course of the term we have developed a professional educator-student relationship. They are future nurses who will be held accountable for their own practice. We have talked about the value of integrity in class and how it connects to patient safety. I can’t police them. If they are going to be autonomous nurses one day they need to regulate their own integrity.
Approachability
I feel it was important after the school pivoted to online learning that I acknowledged how difficult this transition was. Every time I communicate with them, live or asynchronously, I let them know that I care about their struggle. I tell them it is okay if my class is not their priority. In fact, I have already extended the time frame for one of my exams because the dates I chose were not good for some students. Good communication is what will help everyone get through this term effectively.
Despite having a good relationship with students prior to the transition, they still feel bad about reaching out for help. I constantly need to tell them that it is okay – I am here to help. I keep saying “No really, stop apologizing for reaching out … that is my job. I am here for you. I care.” It saddens me that they still feel bad for asking for the help they need. I think that we need to keep in mind that they may be feeling bad about needing help and be even more proactive about offering it. Over and over.
Well-Informed
I also need to be able to defend my decisions to those who may be skeptical. There are people who think that because I teach in an accredited program this exam design may be viewed negatively by colleagues at other sites of the program or accreditors. I looked at our accreditation standards. What they seem to want is evidence that students are meeting learning outcomes. To be clear, I am NOT giving everyone an A just for trying. My standards remain the same. That is a completely different blog post, but the idea of giving everyone an A in nursing is absurd to me. Instead, I am designing an assessment that allows students to flexible ways to demonstrate their competence. It is just good pedagogy.
Universities embrace collegial discourse and academic freedom. Based on my conversations with a colleague from Laurentian around Senate decisions, I am confident that my peers may not all like how my exam is delivered but since I am armed with pedagogy and evidence they will respect it. I have also clearly articulated how my exam design is connected to my learning outcomes. One of them actually has “Participate in professional collegial discourse and sharing of ideas to support a culture that supports critical inquiry” as an end in view (course outcome).
In the above screenshot, the agreement quiz is followed by an exam outline that contains a list of the course outcomes and objectives that are tested as well as the number of marks for each. After that, the exam in Moodle is there with a clear indication of the due date and marks on it. The description also encourages students to reach out. After that, the word version of the exam is posted followed by a PDF. A Moodle announcement also contains a video showing them how to navigate the exam in Moodle and Word.
0 Comments