JET Logo

  The Journal of Effective Teaching
an online journal devoted to teaching excellence

 


Journal of Effective Teaching, Vol. 16, No. 1, 201   PDF Version

Green, S. G., Ferrante, C. J., & Heppard, K. A. (2016). Using Open-Book Exams to Enhance Student Learning, Performance, and Motivation. The Journal of Effective Teaching, 16(1), 19-35. [Abstract]


 

Using Open-Book Exams to Enhance Student Learning, Performance, and Motivation
 

Steve G. Green[1], Claudia J. Ferrante, and Kurt A. Heppard
USAF Academy, CO 80840

 

Abstract 
 

This study investigated an alternative testing protocol used in an undergraduate managerial accounting course. Specifically, we assert that consistent open-book testing approaches will enhance learning and better prepare students for the real-world decision-making they will encounter. A semester-long testing protocol was executed incorporating a mix of open-book and closed-book pre-quizzes, and open-book major exams. Findings indicated that students taking open-book pre-quizzes performed better on open-book final exams, but not other major exams. Our research approach also revealed preliminary indications that our students value their textbooks more, and used them more frequently and extensively, to prepare for class using open-book testing protocols as opposed to using traditional closed-book testing procedures. Also, preliminary indications reveal that alternatives to traditional closed-book testing enhance student satisfaction with courses and textbooks, and provide the potential to improve students’ experiences in the workplace. We encouraged future quantitative studies with robust research designs dedicated to addressing these preliminary indications and provide several suggestions for future research.

 

Keywords: Open-book exams, closed-book exams, assessment, assurance of learning, education.

 

 

Among educators, an excellent way to incite a debate is to discuss the strengths and weaknesses of any particular testing protocol.  In an era of unprecedented change to the educational landscape, including curricula innovations, new approaches to teaching and assessment, and an emphasis on process improvement have created an environment that is often referred to as learning-centered (Ramaley & Leskes, 2002).  Additionally, the proliferation of distance education and the abundance of credible on-line degree and certification programs have highlighted student interest and motivation. 

 

In this dynamic educational environment there has also been an increase in concerns about testing protocols, addressing learning outcomes, and assessment of student performance (Yang & Cornelious, 2005).  These realities coupled with the inexorable transition from traditional pencil-and-paper exams to computer-mediated exams have authors investigating many different aspects of various new testing protocols including; test anxiety, preparation, and how students feel about various exam modes (Alltizer & Clausen, 2008).  With alternative educational approaches, there are often concerns about cheating and plagiarism (Damast, 2007) and discussion on how to address them (Williams, 2006) continue as well.  These pressures for transformation have also motivated educators to investigate a wide variety of improvement opportunities including testing, assessment, and assurance of learning.  We feel examining open-book exams, and making the findings available, will encourage educators to investigate and identify opportunities for educational process improvement.

 

Testing, Assessment and Assurance of Learning

 

Approaches to assessment are as varied as the educators that use them, and the academic disciplines they represent. The act of “grading” was long-viewed as the best, or at least adequate, means of assessing students’ learning.  As researchers consistently determined that tests and quizzes measure retention as well as knowledge (Roediger & Karpicke, 2006a), and as the educational landscape continued to change, performance measurement with alternative forms of valid assessment became the holy grail for educators. 

 

For many business schools, a major change occured with the passage of the Association to Advance Collegiate Schools of Business International’s (Association to Advance Collegiate Schools of Business International [AACSB International], 2007) standards and requirements for a more structured approach to assurance of learning and the outcomes assessment process (Anderson-Fletcher, 2005).   These new AACSB standards forced a re-examination of educational processes at business schools and encouraged teachers to continuously improve the quality of education.  At our institution, we are keenly interested in investigating improvements to our educational processes and encourage sharing results with the academic community.

 

Our investigation of the relatively unorthodox assessment approach of using open-book exams is an excellent example of an attempt to enhance learning by continuously improving our educational processes in the spirit of the AACSB standards.  Specifically, our study investigated whether consistent open-book testing would improve student performance on major exams and ultimately better prepare our students for real-world operational decision-making environments they will encounter.  It also examined whether open-book testing would improve students’ overall satisfaction with courses, justification of required textbooks, and enhanced motivation to learn.

 

Significance

 

From a general education perspective, there is an on-going movement to help students become “intentional learners” who are capable of adapting to new environments, integrate disparate knowledge, and experience continuous learning throughout their lives (Ramaley & Leskes, 2002).  For the last two decades, practically all stakeholders in the educational process have been demanding that the educational community constantly search for improvements to student learning and success (Barr & Tagg, 1995).   Given the exponential growth in readily available knowledge, the assertion that technology is making information increasingly easy to access is also significant.  This changing technology has had an important influence on pedagogy as students’ behavior transform and adapt to contemporary realities such as digital textbooks or e-textbooks (Weisberg, 2011). 

 

At our university, the United States Air Force Academy (USAFA), we feel the use of open-book exams may align better with our stated learning objectives and the technology that our students  access.  More importantly, we feel open-book exams will be more representative of the professional setting our students will encounter upon graduation.  Investigation of the benefits of open-book exams will address in part the persistent call for relevance and “real world” application (Collett, 2000) in higher education.  Our study addresses and compliments many academia-wide initiatives on this topic.

 

Our premise is that traditional time-proven pedagogy, instructional activities, and assessment techniques might not be optimal approaches for all disciplines.  Specifically we feel this may be true for our study involving the relatively structured discipline of managerial accounting.  In a contemporary dynamic workplace, we feel decision-making is essentially an “open-book” activity where managers do not rely upon memorized information to act effectively.  There are many professional bodies that agree with this premise.  For example, in 2005 the National Association of Communication Systems Engineers (NACSE) call closed-book testing “archaic” and “not reflective of the real world” and subsequently changed its certification examinations and training to open-book testing (Sosbe, 2005, p. 4). 

 

For years the same growing void that we recognized in education and practice has also been identified in the accounting community (Albrecht & Sack, 2000), and has been extensively documented (Apostolou, Watson, Hassell, & Webber, 2001), researched (Paisey & Paisey, 2004; Phillips & Phillips, 2007), and promulgated (Accounting Education Change Commission [AECC], 1990, 1992). Stakeholders, such as students and employers, have exasperated the so-called capabilities-gap, by demanding what they want from higher education and the realities of what universities can provide. As a result, formal outcome assessment of accounting programs has become increasingly significant as accreditation bodies require evidence of assurance of learning (AACSB, 2007).

 

Similarly, we feel that the type of learning the undergraduates at our institution experience can improve if we attempt to replicate the modern dynamic workplace they will encounter after graduation.  Therefore we contend that open-book exams would be a closer representation to what graduates would encounter “on the job” including for example being a pilot, program manager, or an accountant.  Feller (1994) felt closed-book exams test what students can memorize while open-book exams better represent real-life situations where considerably more resources are available.  Granted, a pilot needs to memorize certain emergency procedures. However, since each emergency a pilot might encounter is by definition unique, they must be able to assess the situation and adapt appropriately.   Even if our graduates do not fly, to improve learning, we assert that consistent open-book quizzing and examination protocols will better prepare students for the real-world operational decision-making they will encounter. 

 

Open-book testing also addresses the discontent associated with textbook purchase for courses.  Students get frustrated when they pay large sums of money for textbooks that are either  sparingly used during a course, or find that success in the course is not dependent on their use of the textbook.  Students spend billions of dollars each year on textbooks with legitimate complaints of too frequent revisions and needless bundling.  There are reports that textbook prices tripled from 1986-2004 (US Government Accountability Office [GAO], 2005).   Economic realities have undoubtedly forced some students to choose courses based on whether a textbook is required or even based on the cost of the textbook.  Authors are addressing this textbook crisis by studying alternatives to required textbooks such as library reserves (Pollitz, Christie, & Middleton, 2009).   Finally, the federal government has tried to alleviate some of the textbook cost burden by increasing direct aid and suggesting that textbook costs be tax deductible for eligible filers (Supiano, 2009).  We argued that there are few better ways to illustrate the value of an expensive textbook than to allow students to reference it during open-book exams, and optimally retain it for future use.

 

Finally, we believe there may be valuable insights into open-book testing protocols that many educators may have dismissed in the past.  We feel this is especially true in education communities that culminate the learning experience with closed-book computer-based certification tests such at the Certified Public Accounting (CPA) and Certified Management Accounting (CMA) exams.  However, even as educators embraced computer-based exams in their courses as improvements to their assessment portfolios, research indicated no significant difference in student performance on computer-based exams versus traditional paper-based tests (Anakwe, 2008).  We feel by including open-book exams into their assurance of learning repertoire, whether they are computer-based or traditional paper-based, educators will likely enhance student learning while addressing the needs of future employers.

 

Study Landscape

 

Curriculum improvement is an integral part of the mission of our institution, USAFA.  We are very circumspect as to how we select and how we approach our improvement efforts.  We take great strides to ascertain that any study that directly involves students receives particular scrutiny. The design of this experiment ensured that our learner-focused institutional goals and objectives were not compromised, our assurance of learning and assessment processes were enhanced, and our students received equitable treatment regardless of the testing methodology.   To accomplish this we offered all of the major exams, including the final exam, for every student in the course, in an open-book format.  We used the pre-exam quizzes solely as the testing vehicle and alternated open-book and closed-book versions depending across sections of the course.  Also, the pre-exam quizzes represented only 10% of their course grade ensuring that motivation for accomplishing the pre-exam quizzes existed, but the overall impact on the final grade was nominal.

 

Review of the Literature

 

Since assessment is not unique to any specific academic discipline, the review of the literature on open-book versus closed-book testing we chose was somewhat eclectic.  We relied upon a broader education-oriented body of knowledge for relevant discussions of open-book and closed-book testing protocols.   However, we feel this approach enhanced rather than diluted any finding or indications associated with the study.

Numerous research efforts across several different academic disciplines have studied, measured and reported on the efficacy of open-book versus closed-book exams.  A cross-disciplinary review of the literature appears to be inconclusive as to whether “better” learning occurs, varied on whether “better” preparation occurs, and consequently is diverse on which approach is superior.  For example, in an introductory biology course, Moore and Jensen purported that open-book exams actually impede long-term learning (2007).  For an introductory statistics course, Block (2012) discovered that in addition to a reduction in anxiety, the use of open-book exams increased student enjoyment while encouraging deeper student learning.  Also, a new dimension was investigated in several psychology courses with the addition of “cheat-sheets,” or student produced notes that are available during an exam. Results showed that students performed slightly better on open-book exams versus closed-book exams, but for students that predicted they would do better with open-book versus cheat-sheets, the authors found no difference between the two groups (Gharib, Phillips, & Mathew, 2012).

 

Open-Book versus Closed-Book Exams 

 

The closed-book exam is an established approach to assessment in higher-education. It is  both widely accepted by educators and frequently used (Theophilides & Koutselini, 2000) and basically tests how well a student uses the knowledge they can recall with no additional material available for use on the exam.   On the other hand, open-book exams allow students to consult textbooks, notes, and other course–related material during the exam.  Some educators may consider open-book tests less conventional, but they have gained popularity across the entire spectrum of education including primary, secondary, and higher education (Baillie & Toohey, 1997; Eilertsen & Valdermo, 2000).  Impediments to widespread adoption of open-book exams in studies include indications that students spend considerable time looking for answer instead of formulating their answers, and that open-book exams result in a reduction of preparation time in studying (e.g., Boniface, 1985; Rakes, 2008; Theophilides & Koutselini, 2000). Also, Agarwal, Karpicke, Kang, Roediger, and McDermott, (2008) reported mixed findings relative to long-term or delayed retention of material.  This behavior is complicated by changes in students’ study-behavior based on their expectancy of an open-book exam and its impairment on long-term retention (Agarwal & Roediger, 2011).  While our study did not address the issue of student expectancy, it is an important issue to address in future studies given its impact on student performance in other studies.

 

The literature suggests that open-book exams may need new instructional techniques that address different cognitive processes and knowledge levels.  For example, open-book exams might need designs that give students every opportunity to demonstrate their knowledge level and what they can accomplish in the time allotted.  Feller, (1994) recognized that teachers will have to pay more attention to teaching the higher-level skills which includes conceptualization, problem solving, and reasoning.  This is not a new dilemma by any means.  Some of the earliest writings on the subject highlight issues with open-book exams including that they will likely reduce study by allowing students with a false sense of security that will allow them to “slide through” with minimum study (Kalish, 1958).  Also, the advent of the various forms of personal computers, search engines, and other trappings of an information-rich classroom environment, have created changes in pedagogy.  Improvements to this aspect of education represents another educational research opportunity including “open-book, open-web” (OBOW) testing protocols (Williams & Wong, 2009).

 

In general, the relevant literature varies greatly in its orientation and can be categorized by measurement of student performance, assessment of student learning, and identification of various behavioral effects on students such as exam preparation and test anxiety.  This study attempted to address each of these aspects of the open-book versus closed-book exam debate.

 

Measurement of Student Performance

 

As previously referenced, Kalish’s (1958) early investigations into the potential impact of open-book exams addressed the contention that the opportunity to look up material at its source should provide greater accuracy of response than depending upon memory.  While this position was not specifically validated, the fact that open-book exams measured different abilities was verified and this encouraged future study (Kalish, 1958).

 

Differences in student performance was also noted when the exam format changed.  For example, students who took open-book exams the entire semester experienced significantly lower grades on closed-book final exams relative to those who took closed-book exams the entire semester (Moore & Jenson, 2007). In a more recent study, student judgements of comprehension were higher when students benefited from being able to use the open-book format (Ackerman & Leiser, 2014).

 

Using examinations that were specifically designed to test critical thinking and higher-order skills, Ioannidou (1997) compared results of students taking open-book versus closed-book exams.  She concluded that there was no significant difference in the scores of students taking open-book versus closed-book exams, and found that students that expect an open-book test might have less study motivation (Ioannidou, 1997). Other studies directly assert that student performance is actually worse on open-book exams (Boniface, 1985).

 

Assessment of Student Learning

 

While assessing learning is important, many feel that tests can do more.  Exams can enhance learning while also improving long-term retention (Roediger & Karpicke, 2006b).  The question becomes not if tests are beneficial, but test implementation.   For example, Agarwal, Karpicke, Kang, Roediger, and McDermott (2008) found that open-book testing recorded better initial performance, but the benefit did not continue.  Others have stated that closed-book final exams do not adequately measure deep conceptual understanding. Williams’ (2006) position is that closed-book final exams encourage “cramming" and “data dumps” and suggest that closed-book invigilated exams have become anachronisms.

 

This phenomenon is described in the education literature as deep versus surface learning (Entwhistle, 1997).  In general, we feel deep learning is best for contemporary students, and open-book testing has been identified as an excellent means to stimulate deep learning.  However, even recognizing this deep learning versus surface learning perspective, researchers in the field of medical education found opposite results.   Heijne-Penninga, Kuks, Hofman, and Cohen-Schotanus (2008) determined that closed-book tests stimulated deep learning more than open-book exams partially because students had more motivation to study for closed-book exams.  In a related study,  Heijne-Penninga, Kuks, Schonrock-Adema, Snijders, and Cohen-Schotanus (2008) suggested that by breaking the vast amount of medical information into core knowledge and backup knowledge, open-book testing complements closed-book testing and  would be useful for assessment programs.

 

Another educational philosophy referred to as Constructionist Learning, contends knowledge is created by the students’ learning activities, not necessarily transmitted by direct instruction.   Constructionists argue that learning will occur only when the learner is actively engaged (Williams, 2006) and we feel that open-book testing enhances engagement.  Constructivism focuses on knowledge construction, not knowledge reproduction (Herrington & Standen, 2000).  We also feel this position supports open-book testing.

 

Williams and Wong (2009) argue that open-book exams are more authentic and more constructively aligned with stated learning outcomes.  Their position is that closed-book exams are anachronisms given the needs of a knowledge economy and the incompatibility with constructivist learning theory (Williams & Wong, 2009).  We also feel open-book exams compliment this educational philosophy.

 

Eilertsen and Valermo (2000) viewed open-book tests as a means to encourage thinking at higher cognitive levels and promote study and teaching methods.  One of their preliminary findings was that open-book exams stimulate learning and noted that the test itself could be an arena for learning (Eilertsen & Valermo, 2000).

 

The field of managerial accounting recognized the changing role of the practitioner that requires a new skill set and approach to problem-solving (Siegel & Sorenson, 1999).  Educators in the accounting community can potentially benefit from considering findings from Albrecht and Sack (2000) that a rule-based memorization for certifying exams is inefficient and do not prepare students for the business world. In a landmark study performed by Albrecht & Sack (2000), one participating accounting educator stated:  

 

An accounting student needs to know that there are technical rules and regulations.  He or she doesn’t need to be able to tell me what FAS 124 is.  I don’t even know what FAS 124 is, but if I need to know it, I know where to get it. (p. 37)”.

 

As higher education embraces on-line pedagogy, an additional question of how the internet supports learning and how teachers best assess learning looms. Some feel that learning is fundamentally a social process and as our culture and technology evolve, so must higher education.  Preparing students to answer fact-based multiple-choice questions by rote memory is not adequately preparing them for future careers. The key is to develop instructional approaches that foster innovation, creativity, and independent thinking (Bruckman, 2002).

 

Behavioral Effects on Students

 

There is considerable discussion as to whether students’ grades are strongly associated with “good” academic behavior.  Educator should engage in activities that promote good academic behaviors, but Moore and Jenson (2007) found indications of the opposite occur. They found that compared with student facing a closed-book exam, students with a scheduled open-book exam were less likely to attend class and help sessions, or submit extra credit assignments (Moore & Jenson, 2007).  The results also indicated that students preparing for a closed-book exam tended to postpone their study until the end of the semester and focused on the memorization of material in the textbook (Moore & Jenson, 2007).

 

Theophilides and Koutselini (2000) found that students studying for open-book exams tended to review various sources and integrated the information they reviewed.  Further, during the open-book exam, students worked creatively and “probed deeply” into the material (Theophilides & Koutselini, 2000).  Phillips found that open-book exams improved study skills by constructing tests with contextual clues that helped students effectively identify correct answers in the text (Phillips, 2006).

 

Open-book exams also compliment a learner-centered approach to education.  For example, the reduction in the level of anxiety of an open-book exam, whether warranted or not, may be a result of more comprehensive exam preparation and more consistent learning environment with students avoiding “cramming” (Theophilides & Dionysiou, 1996; Theophilides & Koutselini, 2000). 

 

However, we also recognize additional complex behavioral issues and possible negative impacts of open-book exams that other authors have identified.  For example, the use of open-book exams may require professors to ask questions on cognitive levels beyond recall including conceptualization, problem solving, and reasoning (Feller, 1994).  It might also add an additional burden on the instructor since creating effective and valid open-book exams requires a professor to expertly create an open-book exam. Anecdotally, students may engage in a race to see how quickly they can find answers to an open-book exam, as opposed to a guessing game of what questions they will face on a closed-book exam that should be committed to rote memory. The perceptions of harder questions and second-guessing the instructor might create anxiety for some students.  In any case, these behaviors do not create optimal learning environments, but when recognized, can be mitigated.

 

Our study hopes to add value to the rich open-book versus closed-book exam debate in an effort to decrease the gap between the knowledge that our students obtain in courses, the skills they develop when taking our exams, and how they will eventually perform in an operational setting.

 

The Study

 

Our study investigated whether an open-book versus closed-book testing protocol significantly impacted students’ performance on major exams and their attitudes regarding the textbook and the course.  Specifically, we hypothesized:

 

H1: Students taking open-book pre-exam quizzes will perform better on open-book exams than students taking closed-book pre-exam quizzes.

 

H2: Students taking open-book pre-exam quizzes will see clearer links between the textbook and course material and believe they learned more in the course than students taking closed-book pre-exam quizzes.

 

Methodology

 

Students in our undergraduate introductory managerial accounting course experienced a semester-long testing protocol incorporating either open-book or closed-book pre-exam quizzes to prepare for three major open-book exams during the semester.  The final exam was also an open-book exam.  This offering is a required course for all undergraduate Management Majors at our AACSB-International Accredited business program and represented one of the largest sample populations available.  Second, as mentioned earlier, the accounting educational community has aggressively embraced alternative assessment studies such as ours and suggest improving introductory accounting courses with pedagogy that emphasized increased student involvement in the learning process (AECC, 1990; 1992).  Third, the course had a robust set of learning objectives which could be utilized in future studies to measure students’ achievement of learning objectives between testing protocols.

 

The total sample size consisted of 235 students across ten separate sections of the course taught by four instructors.  One-half of each of the four instructors’ sections of students prepared for each of the three open-book major exams with two open-book pre-exam quizzes, while the other sections of students prepared for these major exams with closed-book pre-exam quizzes.  Each instructor ensured that students’ grades would have no impact by the testing protocol they experienced, and that there was no advantage for being a student in either protocol.  The students in the two testing protocols took similar pre-exam quizzes and major exams (i.e., similar conceptual questions with different numbers) and the same final exam.

 

Upon completion of the course, all students completed a survey of questions investigating their attitudes toward open-book testing and its relationship to the course, its textbook and learning.  The survey instrument was the same for all instructors.  Students responded to each statement using a 5-point Likert-scale where 1 was strongly disagree and 5 was strongly agree.  The statements examined whether students saw clear links between the materials covered in the textbook readings and class lectures, if the exams were closely related to the textbook, if students recommended the open-book testing approach, and if students’ instructors provided suggestions on how to effectively use the textbook.  Additional questions surveyed students as to whether they spent more time working problems or exercises in the textbook, or more time preparing the textbook for use as reference during the exam to prepare for the open-book exams, and whether they felt they learned more or less using the open-book testing approach.  Finally, the survey asked if they had a false sense of security in preparing for the open-book exams.

 

Results

 

We used PASW Statistics 18 to analyze our data (See Table 1).  Our findings indicated that students who took open-book pre-exam quizzes did not perform significantly better on any of the three open-book major exams than students who took closed-book pre-exam quizzes.  However, students who took open-book pre-exam quizzes did perform significantly better on the open-book final exam than students who took closed-book pre-exam quizzes.  As might be expected, students’ performance on the open-book final exam was significantly impacted by their performance on the three open-book major exams.  Thus, these findings only partially supported our first hypothesis.

 

In examining the survey data, we did find significant differences between students in the open-book versus closed-book pre-exam quiz sections.  Specifically, students in the open-book sections more strongly agreed that they saw clear links between materials covered in textbook readings and class lectures (M = 4.32 for open, 4.12 for closed) and the exams were closely related to textbook (M = 4.18 for open, 3.99 for closed) than students in the closed-book sections.  The open-book students also more strongly agreed that their instructor provided suggestions on how to effectively use the textbook (M = 4.04 for open, 3.82 for closed).  Although there was not a statistically significant difference between open-book and closed-book students and their recommendation of the open-book testing method, 82.6 percent of the students strongly agreed or agreed with the statement “I recommend the open textbook method for other classes (M = 4.26)”.

 

Although not statistically significant across our open-book versus closed-book testing protocol, 51.3 percent of students said they spent more time working problems/exercises in the textbook to prepare for the open-book exams, whereas 48.7 percent of student said they spent more time preparing the textbook for use as reference during the exam.  Eighty-eight percent of students felt they learned more using the open-book testing approach with no statistically significant difference between students in the open-book versus closed-book sections.  We did find a statistically significant difference between students in the open-book and closed-book sections and their having a false sense of security in preparing for open-book exams.  Specifically, although 70.5 percent of all students indicated they did not have a false sense of security in preparing for the open-book exams, students in the open-book sections had less of a false sense of security, as the mean for students in the open-book sections was 0.23 and 0.36 for students in the closed-book sections (where 0 was ‘no’ and 1 was ‘yes’).  Thus, our students did not possess overconfidence in preparing for the open-book exams.

 

Table 1. Means, Standard Deviations, and Correlations.

 

                          

                                NOTE: N = 235.

                                * p < .10. ** p < .05. *** p < .001.

 

 

Findings and Discussion

 

Although we found only partial support for our hypothesis that open-book pre-exam quizzes would significantly increase students’ performance on open-book major exams, we believe our study was successful in demonstrating usefulness of this testing protocol.  Specifically, students like the idea of open-book exams, but not necessarily for the reasons educators might think.  Anecdotally, students indicated they learned more through the open-book testing approach than they do through the conventional closed-book approach regardless of their grades on the assessments, because they were able to focus on mastering concepts to solve the accounting problems rather than memorize technical aspects they could look up in the textbook.  Further, they also indicated they did not have a false sense of security going into the open-book exams, as they knew they would have to be able to work through the problems and apply concepts rather than simply report facts from the textbook, as well as having a time-constraint. These findings can provide a foundation for suggested open-book exam “best practices.”  The key to successful implementation and increased student learning rests with how well the open-book testing protocol and process are described to the students.  Also, best practices might include effort to ensure instructors are as unified as possible in their desire to investigate new approaches to enhancing student learning.  

 

Conclusion and Suggestions for Future Research

 

In our quest to prepare our students for careers in the “open-book world” they will encounter after graduation, we believe open-book exam approach is useful for enhancing student learning while effectively preparing our students for real-world operational decision making. Even though our study did not answer the proposition of whether open-book testing encourages life-long learning in students by enlightening them that they do not need to “know” all the answers, we still feel that referring to source material for guidance is an attribute. There were indications that open-book testing encouraged active student engagement in their learning, and in many cases expanded their confidence in being able to work through difficult concepts.  This as well as several other areas warrant further investigation, and we offer several suggestions for future study. 

 

First and foremost, it would be beneficial to conduct a follow-on study in which the open-book versus closed-book treatments are carried throughout the entire course with all the instructors.  Additionally, even with over 250 students, in our opinion, we did not have enough subjects to warrant the multiple experimental treatments; open-book and closed-book pre-exam quizzes crossed with open-book and closed-book exams.  Specifically, we recommend having one-fourth of the students in a large population course experience the following treatments: open-book pre-exam quizzes and open-book exams, closed-book pre-exam quizzes and open-book exams, open-book pre-exam quizzes and closed-book exams, and closed-book pre-exam quizzes and closed-book exams.  We believe this approach would accommodate a more robust study and may produce significant findings.

 

It would also be very interesting to conduct longitudinal studies of students at various levels of academic experience and development.  These longitudinal studies offer the promise of understanding open-book preparation and performance differences for students with different levels of experience and academic maturity.  Well-constructed longitudinal studies could also follow specific students throughout their academic careers to evaluate possible effects the open-book methodology has on student development and performance. These longitudinal studies can also give greater insight into possible negative impacts of open-book exams on student learning and performance.  For example, mixed findings have been reported regarding long-term or delayed retention of material covered in open-book exam approaches (Agarwal, Karpicke, Kang, Roediger, & McDermott, 2008).  Longitudinal studies are important as these long-term effects of the methodology are more carefully studied and reported.  

 

Many other potential research opportunities in the study of open-book exams exist. Educators agree that open-book exams reduce anxiety, and students appear to consistently prefer open-book exams despite the acknowledge understanding that they generally require the exhibition of higher-order thinking skills (Brightwell, Daniel, & Stewart, 2004).  However, as discussed earlier, questions might actually be harder on an open-book exam because they are testing at a higher level of learning.  Consequently, we suggest pre-test post-test treatments of attitudes associated with this phenomenon.  Also, although open-book exams do not necessarily lead to higher achievement in terms of test scores, they do seem to reduce unnecessary rote memorization of facts which allows students to prepare for exams more constructively (Theophilides & Dionysiou, 1996).  We did not specifically address this phenomenon, but recognized it during our study and suggest future investigation. 

 

Open-book exams may also need new instructional techniques that address different cognitive processes and knowledge levels. In other words, perhaps studies should be conducted on the manner in which a student approaches a problem, based on the open-book versus closed-book experience.  Related to this idea, open-book exams may be more difficult to construct (Eilertsen & Valdermo, 2000), and faculty may spend relatively too much time preparing and grading open-book exams while other faculty may also be insufficiently trained or experienced in open-book exam construction (Vanderburgh, 2005). We suggest additional research be oriented toward the instructor, not just the student. 

 

There are also numerous other hybrid options of open-book versus closed-book exam protocols; including an open hand-written note-card approach that show promise in not only high levels of learning but enjoyment of the course (Block, 2012).   Research, such as the one conducted by Gharib, Phillips, & Mathew (2012) studying “cheat-sheets” versus open-book or closed-book exams, warrant further investigation.

 

Finally, as discussed early, no open-book exam study would be complete without properly addressing advancements that are occurring in the on-line education environment. We feel it should not be the ubiquity of the internet that drives the exam approach; it should be a question of if students learn better via open-book exams that also allow access to the internet.   We feel that perhaps the richest potential area of study may be to investigate the learning effects of not only open-book exams, but open-computer exams as well.  To increase the real-world feel of an exam and to better replicate a future professional work environment, open-book open-computer exams, such as those suggested by Williams and Wong (2009), could be implemented and their impact on learning enhancement studied.

 

Future research oriented toward addressing these potential research topics will contribute to the ongoing open-book versus closed-book exam debate.  These educational process improvement efforts will help address demands that the educational community enhance student learning and success, as well as future professional performance. 

 

We are hopeful that educators in other disciplines find our study useful for their course development.  At a minimum, we hope the readers of this study gained insight into a testing protocol they may have dismissed in the past, but now may consider adding to their assurance of learning repertoire in their quest to enhance their students’ learning and future professional success.

 

References

 

AACSB International Accreditation Coordinating Committee. (2007). AACSB assurance of learning standards: An interpretation. An AACSB White Paper, 20.

Accounting Education Change Commission. (1990). Position statement number one: Objectives of education for accountants. Issues in Accounting Education, 5, 307-312. 

Accounting Education Change Commission. (1992). The first course in accounting: Position statement No. 2.  Issues in Accounting Education, 7(2),249-251.

Ackerman R., & Leiser D. (2104). The effect of concrete supplements on metacognitive regulation during learning and open-book test taking. British Journal of Educational Psychology, 84(2):329-348.

Agarwal, P. K., Karpicke, J. D., Kang, S. H. K., Roediger, H. L. III, & McDermott, K. B. (2008). Examining the testing effect with open-and closed-book tests. Applied Cognitive Psychology, 22(7), 861-876.

Agarwal, P. K., & Roediger, H.L. III. (2011). Expectancy of an open-book test decreases performance on delayed closed-book test. Memory, 19(8), 836-852.

Albrecht, W. S., & Sack, R. J. (2000). Accounting education: Charting the course through a perilous future (Vol.).  Sarasota, FL: American Accounting Association.

Allitizer, R. L., & Clausen, T. S. (2008). Computer-mediated exams: Student perceptions and performance. Advances in Accounting, Finance and Economics,1(1), 1-7.

Anakwe, B. (2008). Comparison of student performance in paper-based versus computer-based testing. Journal of Education for Business, 84(1) 13-17. 

Anderson-Fletcher, E. (2005). Going from zero to sixty in twelve months: Implementing assessment at the Bauer College of Business. In K. Martel & T. Calderon (Eds.), Assessment of student learning in business schools: Best practices each step of the way (pp. 64-83). Tallahassee, FL: Association for Institutional Research, AASCB International.

Apostolou, B., Watson, S. F., Hassell, J. M., & Webber, S. A. (2001). Accounting education literature review (1997-1999). Journal of Accounting Education, 19(1), 1-61.

Baillie, C., & Toohey, S. (1997). The “power test:” Its impact on student learning in a materials science course for engineering students. Assessment and Evaluation in Higher Education, 22(1), 33-49.

Barr, R. B., & Tagg, J. (1995). From teaching to learning - A new paradigm for undergraduate education. Change: The magazine of higher learning, 27(6), 12-26.

Block, R. M. (2012). A discussion of the effects of open-book and closed-book exams on student achievement in an introductory statistics course. PRIMUS, 22(3), 228-238.

Boniface, D. (1985). Candidates’ use of notes and textbooks during an open-book examination. Educational Research, 27(3), 201-209.

Brightwell, R., Daniel, J. H., & Stewart, A. (2004). Evaluation: Is an open book examination easier?  Bioscience Education E-Journal, 3. Retrieved February 11, 2015, from

      http://www.bioscience.heacademy.ac.uk/journal/vol3/Beej-3-3.htm

Bruckman, A. (2002). The future of e-learning communities. Communications of the ACM, 45(4), 60-63.

Collett, N. J. (2000). Innovation or renovation: Effective project design for accounting and MBA students. Accounting Education, 9(1), 67-92.

Damast, A. (2007, May 3). Duke works to stem Fuqua fallout. BusinessWeek. Retrieved February 11, 2015, from http://www.bloomberg.com/bw/stories/2007-05-03/duke-works-to-stem-fuqua-falloutbusinessweek-business-news-stock-market-and-financial-advice 

Entwistle, N. (1997). Contrasting perspectives in learning. In: F. Marton, D. Hounselland, & N. Entwistle (Eds.). The Experience of Learning: Implications for teaching and studying in higher education, 3rd (Internet) edition (pp. 106-125). Edinburgh: University of Edinburgh, Centre for Teaching, Learning and Assessment. Retrieved February 11, 2015, from

      http://www.docs.hss.ed.ac.uk/iad/Learning_teaching/Academic_teaching/Resources/
Experience_of_learning/EoLChapter1.pdf

Eilertsen, T. V., & Valdermo, O. (2000). Open-book assessment: A contribution to improved learning? Studies in Educational Evaluation, 26(2), 91-103.

Feller, M. (1994). Open-book testing and education for the future. Studies in Educational Evaluation, 20(2), 235-238.

Gharib, A., Phillips, W., & Mathew, N. (2012). Cheat sheet or open-book? A comparison of the effects of exam types on performance, retention and anxiety. Psychology Research, 2(8), 469-478. Retrieved February 11, 2015 from 

      http://files.eric.ed.gov/fulltext/ED537423.pdf

Heijne-Penninga, M., Kuks, J. B. M., Hofman, W. H. A., & Cohen-Schotanus, J. (2008). Influence of open-and closed-book tests on medical students’ learning approaches. Medical Education, 42(10), 967-974.

Heijne-Penninga, M., Kuks, J.B.M., Schonrock-Adema, J., Snijders, T.A.B., & Cohen-Schotanus, J. (2008). Open-book tests to complement assessment-programmes: Analysis of open and closed-book tests. Advances in Health Sciences Education, 13(3), 263-273.

Herrington, J., & Standen, P. (2000). Moving from an instructivist to a constructivist multimedia learning environment. Journal of Educational Multimedia and Hypermedia, 9(3), 195-205.

Ioannidou, M. K. (1997). Testing and life-long learning: Open-book and closed-book examination in a university course. Studies in Educational Evaluation, 23(2), 131-139.

Kalish, R. A. (1958). An experimental evaluation of the open book examination. Journal of Educational Psychology, 49(4), 200-204.

Moore, R., & Jensen, P. A. (2007). Do open-book exams impede long-term learning in introductory biology courses?.  Journal of College Science Teaching, 36(7), 46-49.

Paisey, C., & Paisey, N. J. (2004). An analysis of accounting education research in accounting education: an international journal - 1992-2001. Accounting Education, 13(1), 69-99.

Phillips, B. J., & Phillips, F. (2007). Sink or skim: Textbook reading behaviors of introductory accounting students. Issues in Accounting Education, 22(1), 21-44.

Phillips, G. (2006). Using open-book tests to strengthen the study skills of community-college biology students. Journal of Adolescent and Adult Literacy, 49(7), 574-582.

Pollitz, J. H., Christie, A., & Middleton, C. (2009). Management of library course reserves and the textbook affordability crisis. Journal of Access Services, 6(4), 459-484.

Rakes, G. C. (2008). Open book testing in online learning environments. Journal of Interactive Online Learning, 7(1), 1-9.

Ramaley, J. A., & Leakes, A. (2002). Greater expectations: a new vision of learning as the nation goes to college. Association of American Colleges and Universities. 

Roediger, H. L., & Karpicke, J. D. (2006a).  The power of testing memory: Basic research and implications of educational practice. Perspectives on Psychological Science, 1(3), 181-210. 

Roediger, H. L., & Karpicke, J. D. (2006b) Test-enhanced learning taking memory tests improves long-term retention. Psychological Science, 17(3), 249-255.

Siegel, G., &  Sorensen, J. E. (1999). Counting more, counting less: Transformation in the management accounting profession - The 1999 practice analysis of management accounting. Montvale, NJ: Institute of Management Accountants.

Sosbe, T. (2005). Testing by the book. Certification Magazine, 7(6), p.4.

Supiano, B. (2009). The billion dollar question: Which students benefit? The Chronicle of Higher Education, 55(26), A22-23.

Theophilides C., & Dionysiou, O. (1996). The major functions of the open-book examination at the university level: A factor analytic study. Studies in Educational Evaluation, 22(2), 157-170.

Theophilides, C., & Koutselini, M. (2000). Study behavior in the closed-book and the open-book examination: A comparative analysis. Educational Research and Evaluation, 6(4), 379-393.

US Government Accountability Office (2005).  College textbooks enhanced offerings appear to drive recent price increases. (Publication No. GAO-05-806). Retrieved February 11, 2015, from http://www.gao.gov/assets/250/247332.pdf

Vanderburgh, P. M. (2005). Open-book tests and student-authored exam questions as useful tools to increase critical thinking. Advances in Physiology Education, 29(3), 183-184.

Yang, Y., & Cornelious, L. F. (2005). Preparing instructors for quality online instruction. Online Journal of Distance Learning and Administration, 8 (1). Retrieved February 11, 2015, from http://www.westga.edu/%7Edistance/ojdla/spring81/spring81.htm

Weisberg, M. (2011). Student attitudes and behaviors toward digital textbooks. Publishing Research Quarterly, 27(2), 188-196.

 

Williams, J. B. (2006) The place of the closed book, invigilated final examination in a knowledge economy.  Educational Media International, 43(2), 107-119.

Williams, J. B., & Wong, A. (2009). The efficacy of final examinations: A comparative study of closed-book, invigilated exams and open-book, open-web exams.  British Journal of Education Technology, 40(2), 227-236.

 

APPENDIX I

 

Textbook Survey

 

Instructions:

In an attempt to enhance learning, we are very interested in your opinion about how we present course material.  You are to read each statement and indicate your own personal feelings about the use of textbooks in this course by marking how much you agree with each of the following statements.  Using the scale below, fill in the appropriate bubble on your General Answer Sheet.

   

   A                        B                    C                    D                 E

Strongly          Disagree         Neutral           Agree          Strongly

Disagree                                                                               Agree

 

1)  I use textbooks to prep for exams in other classes in the same way I am using my textbook to prep for exams in my accounting class.

 

2)  My instructor provided suggestions on how to effectively use my text book.

 

3)  I use the textbook publisher’s website to supplement the textbook.

 

4)  I see clear links between materials covered in textbook readings and class lectures.

 

5)  Exams were closely related to the textbook.

 

6)  After receiving my score from a graded test, I reviewed the tested material again in the textbook.

 

7)  I recommend the open textbook method for other classes.

 

8)  I plan to sell my accounting textbook at the end of the semester.

 

9)  The price I paid for my accounting textbook was fair.

 

10)  The unremitting and nonsensical use of mildly amusing clipart in class lectures made it difficult to focus on learning objectives from the textbook.

 

Please provide short answers (not just “yes” or “no” responses) to the following questions in the area provided.

 

11)  What was your strategy for using your textbook in your accounting class?

 

 

 

 

12)  Did you spend more time working on problems/exercises in the text or more time preparing the text for use as a reference during the exam? 

 

 

 

 

13)  Do you feel you learned more or less using the open textbook approach?

 

 

 

 

14)  Did you have a false sense of security in preparing for the open-textbook exams? Please discuss why or why not. 

 

 

 

15)  Please provide any other comments or observations you may have on the use of textbooks in this course.


[1]Corresponding author's email: steve.green@usafa.edu

Divider