educators, an excellent way to incite a debate is to discuss
the strengths and weaknesses of any particular testing
protocol. In an era of unprecedented change to the
educational landscape, including curricula innovations, new
approaches to teaching and assessment, and an emphasis on
process improvement have created an environment that is
often referred to as learning-centered (Ramaley & Leskes,
2002). Additionally, the proliferation of distance
education and the abundance of credible on-line degree and
certification programs have highlighted student interest and
dynamic educational environment there has also been an
increase in concerns about testing protocols, addressing
learning outcomes, and assessment of student performance
(Yang & Cornelious, 2005). These realities coupled with the
inexorable transition from traditional pencil-and-paper
exams to computer-mediated exams have authors investigating
many different aspects of various new testing protocols
including; test anxiety, preparation, and how students feel
about various exam modes (Alltizer & Clausen, 2008). With
alternative educational approaches, there are often concerns
about cheating and plagiarism (Damast, 2007) and discussion
on how to address them (Williams, 2006) continue as well.
These pressures for transformation have also motivated
educators to investigate a wide variety of improvement
opportunities including testing, assessment, and assurance
of learning. We feel examining open-book exams, and making
the findings available, will encourage educators to
investigate and identify opportunities for educational
Testing, Assessment and Assurance of Learning
to assessment are as varied as the educators that use them,
and the academic disciplines they represent. The act of
“grading” was long-viewed as the best, or at least adequate,
means of assessing students’ learning. As researchers
consistently determined that tests and quizzes measure
retention as well as knowledge (Roediger & Karpicke, 2006a),
and as the educational landscape continued to change,
performance measurement with alternative forms of valid
assessment became the holy grail for educators.
business schools, a major change occured with the passage of
the Association to Advance Collegiate Schools of Business
International’s (Association to Advance Collegiate Schools
of Business International [AACSB International], 2007)
standards and requirements for a more structured approach to
assurance of learning and the outcomes assessment process
(Anderson-Fletcher, 2005). These new AACSB standards
forced a re-examination of educational processes at business
schools and encouraged teachers to continuously improve the
quality of education. At our institution, we are keenly
interested in investigating improvements to our educational
processes and encourage sharing results with the academic
investigation of the relatively unorthodox assessment
approach of using open-book exams is an excellent example of
an attempt to enhance learning by continuously improving our
educational processes in the spirit of the AACSB standards.
Specifically, our study investigated whether consistent
open-book testing would improve student performance on major
exams and ultimately better prepare our students for
real-world operational decision-making environments they
will encounter. It also examined whether open-book testing
would improve students’ overall satisfaction with courses,
justification of required textbooks, and enhanced motivation
general education perspective, there is an on-going movement
to help students become “intentional learners” who are
capable of adapting to new environments, integrate disparate
knowledge, and experience continuous learning throughout
their lives (Ramaley & Leskes, 2002). For the last two
decades, practically all stakeholders in the educational
process have been demanding that the educational community
constantly search for improvements to student learning and
success (Barr & Tagg, 1995). Given the exponential growth
in readily available knowledge, the assertion that
technology is making information increasingly easy to access
is also significant. This changing technology has had an
important influence on pedagogy as students’ behavior
transform and adapt to contemporary realities such as
digital textbooks or e-textbooks (Weisberg, 2011).
university, the United States Air Force Academy (USAFA), we
feel the use of open-book exams may align better with our
stated learning objectives and the technology that our
students access. More importantly, we feel open-book exams
will be more representative of the professional setting our
students will encounter upon graduation. Investigation of
the benefits of open-book exams will address in part the
persistent call for relevance and “real world” application (Collett,
2000) in higher education. Our study addresses and
compliments many academia-wide initiatives on this topic.
is that traditional time-proven pedagogy, instructional
activities, and assessment techniques might not be optimal
approaches for all disciplines. Specifically we feel this
may be true for our study involving the relatively
structured discipline of managerial accounting. In a
contemporary dynamic workplace, we feel decision-making is
essentially an “open-book” activity where managers do not
rely upon memorized information to act effectively. There
are many professional bodies that agree with this premise.
For example, in 2005 the National Association of
Communication Systems Engineers (NACSE) call closed-book
testing “archaic” and “not reflective of the real world” and
subsequently changed its certification examinations and
training to open-book testing (Sosbe, 2005, p. 4).
the same growing void that we recognized in education and
practice has also been identified in the accounting
community (Albrecht & Sack, 2000), and has been extensively
documented (Apostolou, Watson, Hassell, & Webber, 2001),
researched (Paisey & Paisey, 2004; Phillips & Phillips,
2007), and promulgated (Accounting Education Change
Commission [AECC], 1990, 1992). Stakeholders, such as
students and employers, have exasperated the so-called
capabilities-gap, by demanding what they want from higher
education and the realities of what universities can
provide. As a result, formal outcome assessment of
accounting programs has become increasingly significant as
accreditation bodies require evidence of assurance of
learning (AACSB, 2007).
we feel that the type of learning the undergraduates at our
institution experience can improve if we attempt to
replicate the modern dynamic workplace they will encounter
after graduation. Therefore we contend that open-book exams
would be a closer representation to what graduates would
encounter “on the job” including for example being a pilot,
program manager, or an accountant. Feller (1994) felt
closed-book exams test what students can memorize while
open-book exams better represent real-life situations where
considerably more resources are available. Granted, a pilot
needs to memorize certain emergency procedures. However,
since each emergency a pilot might encounter is by
definition unique, they must be able to assess the situation
and adapt appropriately. Even if our graduates do not fly,
to improve learning, we assert that consistent open-book
quizzing and examination protocols will better prepare
students for the real-world operational decision-making they
testing also addresses the discontent associated with
textbook purchase for courses. Students get frustrated when
they pay large sums of money for textbooks that are either
sparingly used during a course, or find that success in the
course is not dependent on their use of the textbook.
Students spend billions of dollars each year on textbooks
with legitimate complaints of too frequent revisions and
needless bundling. There are reports that textbook prices
tripled from 1986-2004 (US Government Accountability Office
[GAO], 2005). Economic realities have undoubtedly forced
some students to choose courses based on whether a textbook
is required or even based on the cost of the textbook.
Authors are addressing this textbook crisis by studying
alternatives to required textbooks such as library reserves
(Pollitz, Christie, & Middleton, 2009). Finally, the
federal government has tried to alleviate some of the
textbook cost burden by increasing direct aid and suggesting
that textbook costs be tax deductible for eligible filers
(Supiano, 2009). We argued that there are few better ways
to illustrate the value of an expensive textbook than to
allow students to reference it during open-book exams, and
optimally retain it for future use.
believe there may be valuable insights into open-book
testing protocols that many educators may have dismissed in
the past. We feel this is especially true in education
communities that culminate the learning experience with
closed-book computer-based certification tests such at the
Certified Public Accounting (CPA) and Certified Management
Accounting (CMA) exams. However, even as educators embraced
computer-based exams in their courses as improvements to
their assessment portfolios, research indicated no
significant difference in student performance on
computer-based exams versus traditional paper-based tests (Anakwe,
2008). We feel by including open-book exams into their
assurance of learning repertoire, whether they are
computer-based or traditional paper-based, educators will
likely enhance student learning while addressing the needs
of future employers.
improvement is an integral part of the mission of our
institution, USAFA. We are very circumspect as to how we
select and how we approach our improvement efforts. We take
great strides to ascertain that any study that directly
involves students receives particular scrutiny. The design
of this experiment ensured that our learner-focused
institutional goals and objectives were not compromised, our
assurance of learning and assessment processes were
enhanced, and our students received equitable treatment
regardless of the testing methodology. To accomplish this
we offered all of the major exams, including the final exam,
for every student in the course, in an open-book format. We
used the pre-exam quizzes solely as the testing vehicle and
alternated open-book and closed-book versions depending
across sections of the course. Also, the pre-exam quizzes
represented only 10% of their course grade ensuring that
motivation for accomplishing the pre-exam quizzes existed,
but the overall impact on the final grade was nominal.
Review of the Literature
assessment is not unique to any specific academic
discipline, the review of the literature on open-book versus
closed-book testing we chose was somewhat eclectic. We
relied upon a broader education-oriented body of knowledge
for relevant discussions of open-book and closed-book
testing protocols. However, we feel this approach enhanced
rather than diluted any finding or indications associated
with the study.
research efforts across several different academic
disciplines have studied, measured and reported on the
efficacy of open-book versus closed-book exams. A
cross-disciplinary review of the literature appears to be
inconclusive as to whether “better” learning occurs, varied
on whether “better” preparation occurs, and consequently is
diverse on which approach is superior. For example, in an
introductory biology course, Moore and Jensen purported that
open-book exams actually impede long-term learning (2007).
For an introductory statistics course, Block (2012)
discovered that in addition to a reduction in anxiety, the
use of open-book exams increased student enjoyment while
encouraging deeper student learning. Also, a new dimension
was investigated in several psychology courses with the
addition of “cheat-sheets,” or student produced notes that
are available during an exam. Results showed that students
performed slightly better on open-book exams versus
closed-book exams, but for students that predicted they
would do better with open-book versus cheat-sheets, the
authors found no difference between the two groups (Gharib,
Phillips, & Mathew, 2012).
Open-Book versus Closed-Book Exams
closed-book exam is an established approach to assessment in
higher-education. It is both widely accepted by educators
and frequently used (Theophilides & Koutselini, 2000) and
basically tests how well a student uses the knowledge they
can recall with no additional material available for use on
the exam. On the other hand, open-book exams allow
students to consult textbooks, notes, and other
course–related material during the exam. Some educators may
consider open-book tests less conventional, but they have
gained popularity across the entire spectrum of education
including primary, secondary, and higher education (Baillie
& Toohey, 1997; Eilertsen & Valdermo, 2000). Impediments to
widespread adoption of open-book exams in studies include
indications that students spend considerable time looking
for answer instead of formulating their answers, and that
open-book exams result in a reduction of preparation time in
studying (e.g., Boniface, 1985; Rakes, 2008; Theophilides &
Koutselini, 2000). Also, Agarwal, Karpicke, Kang, Roediger,
and McDermott, (2008) reported mixed findings relative to
long-term or delayed retention of material. This behavior
is complicated by changes in students’ study-behavior based
on their expectancy of an open-book exam and its impairment
on long-term retention (Agarwal & Roediger, 2011). While
our study did not address the issue of student expectancy,
it is an important issue to address in future studies given
its impact on student performance in other studies.
literature suggests that open-book exams may need new
instructional techniques that address different cognitive
processes and knowledge levels. For example, open-book
exams might need designs that give students every
opportunity to demonstrate their knowledge level and what
they can accomplish in the time allotted. Feller, (1994)
recognized that teachers will have to pay more attention to
teaching the higher-level skills which includes
conceptualization, problem solving, and reasoning. This is
not a new dilemma by any means. Some of the earliest
writings on the subject highlight issues with open-book
exams including that they will likely reduce study by
allowing students with a false sense of security that will
allow them to “slide through” with minimum study (Kalish,
1958). Also, the advent of the various forms of personal
computers, search engines, and other trappings of an
information-rich classroom environment, have created changes
in pedagogy. Improvements to this aspect of education
represents another educational research opportunity
including “open-book, open-web” (OBOW) testing protocols
(Williams & Wong, 2009).
the relevant literature varies greatly in its orientation
and can be categorized by measurement of student
performance, assessment of student learning, and
identification of various behavioral effects on students
such as exam preparation and test anxiety. This study
attempted to address each of these aspects of the open-book
versus closed-book exam debate.
Measurement of Student Performance
previously referenced, Kalish’s (1958) early investigations
into the potential impact of open-book exams addressed the
contention that the opportunity to look up material at its
source should provide greater accuracy of response than
depending upon memory. While this position was not
specifically validated, the fact that open-book exams
measured different abilities was verified and this
encouraged future study (Kalish, 1958).
in student performance was also noted when the exam format
changed. For example, students who took open-book exams the
entire semester experienced significantly lower grades on
closed-book final exams relative to those who took
closed-book exams the entire semester (Moore & Jenson,
2007). In a more recent study, student judgements of
comprehension were higher when students benefited from
being able to use the open-book format (Ackerman & Leiser,
examinations that were specifically designed to test
critical thinking and higher-order skills, Ioannidou (1997)
compared results of students taking open-book versus
closed-book exams. She concluded that there was no
significant difference in the scores of students taking
open-book versus closed-book exams, and found that students
that expect an open-book test might have less study
motivation (Ioannidou, 1997). Other studies directly assert
that student performance is actually worse on open-book
exams (Boniface, 1985).
Assessment of Student Learning
assessing learning is important, many feel that tests can do
more. Exams can enhance learning while also improving
long-term retention (Roediger & Karpicke, 2006b). The
question becomes not if tests are beneficial, but test
implementation. For example, Agarwal, Karpicke, Kang,
Roediger, and McDermott (2008) found that open-book testing
recorded better initial performance, but the benefit did not
continue. Others have stated that closed-book final exams
do not adequately measure deep conceptual understanding.
Williams’ (2006) position is that closed-book final exams
encourage “cramming" and “data dumps” and suggest that
closed-book invigilated exams have become anachronisms.
phenomenon is described in the education literature as deep
versus surface learning (Entwhistle, 1997). In general, we
feel deep learning is best for contemporary students, and
open-book testing has been identified as an excellent means
to stimulate deep learning. However, even recognizing this
deep learning versus surface learning perspective,
researchers in the field of medical education found opposite
results. Heijne-Penninga, Kuks, Hofman, and Cohen-Schotanus
(2008) determined that closed-book tests stimulated deep
learning more than open-book exams partially because
students had more motivation to study for closed-book
exams. In a related study, Heijne-Penninga, Kuks,
Schonrock-Adema, Snijders, and Cohen-Schotanus (2008)
suggested that by breaking the vast amount of medical
information into core knowledge and backup knowledge,
open-book testing complements closed-book testing and would
be useful for assessment programs.
educational philosophy referred to as Constructionist
Learning, contends knowledge is created by the students’
learning activities, not necessarily transmitted by direct
instruction. Constructionists argue that learning will
occur only when the learner is actively engaged (Williams,
2006) and we feel that open-book testing enhances
engagement. Constructivism focuses on knowledge
construction, not knowledge reproduction (Herrington &
Standen, 2000). We also feel this position supports
Wong (2009) argue that open-book exams are more authentic
and more constructively aligned with stated learning
outcomes. Their position is that closed-book exams are
anachronisms given the needs of a knowledge economy and the
incompatibility with constructivist learning theory
(Williams & Wong, 2009). We also feel open-book exams
compliment this educational philosophy.
and Valermo (2000) viewed open-book tests as a means to
encourage thinking at higher cognitive levels and promote
study and teaching methods. One of their preliminary
findings was that open-book exams stimulate learning and
noted that the test itself could be an arena for learning
(Eilertsen & Valermo, 2000).
The field of
managerial accounting recognized the changing role of the
practitioner that requires a new skill set and approach to
problem-solving (Siegel & Sorenson, 1999). Educators in the
accounting community can potentially benefit from
considering findings from Albrecht and Sack (2000) that a
rule-based memorization for certifying exams is inefficient
and do not prepare students for the business world. In a
landmark study performed by Albrecht & Sack (2000), one
participating accounting educator stated:
accounting student needs to know that there are technical
rules and regulations. He or she doesn’t need to be able to
tell me what FAS 124 is. I don’t even know what FAS 124 is,
but if I need to know it, I know where to get it. (p. 37)”.
education embraces on-line pedagogy, an additional question
of how the internet supports learning and how teachers best
assess learning looms. Some feel that learning is
fundamentally a social process and as our culture and
technology evolve, so must higher education. Preparing
students to answer fact-based multiple-choice questions by
rote memory is not adequately preparing them for future
careers. The key is to develop instructional approaches that
foster innovation, creativity, and independent thinking
Behavioral Effects on Students
considerable discussion as to whether students’ grades are
strongly associated with “good” academic behavior. Educator
should engage in activities that promote good academic
behaviors, but Moore and Jenson (2007) found indications of
the opposite occur. They found that compared with student
facing a closed-book exam, students with a scheduled
open-book exam were less likely to attend class and help
sessions, or submit extra credit assignments (Moore &
Jenson, 2007). The results also indicated that students
preparing for a closed-book exam tended to postpone their
study until the end of the semester and focused on the
memorization of material in the textbook (Moore & Jenson,
and Koutselini (2000) found that students studying for
open-book exams tended to review various sources and
integrated the information they reviewed. Further, during
the open-book exam, students worked creatively and “probed
deeply” into the material (Theophilides & Koutselini,
2000). Phillips found that open-book exams improved study
skills by constructing tests with contextual clues that
helped students effectively identify correct answers in the
text (Phillips, 2006).
exams also compliment a learner-centered approach to
education. For example, the reduction in the level of
anxiety of an open-book exam, whether warranted or not, may
be a result of more comprehensive exam preparation and more
consistent learning environment with students avoiding
“cramming” (Theophilides & Dionysiou, 1996; Theophilides &
also recognize additional complex behavioral issues and
possible negative impacts of open-book exams that other
authors have identified. For example, the use of open-book
exams may require professors to ask questions on cognitive
levels beyond recall including conceptualization, problem
solving, and reasoning (Feller, 1994). It might also add an
additional burden on the instructor since creating effective
and valid open-book exams requires a professor to expertly
create an open-book exam. Anecdotally, students may engage
in a race to see how quickly they can find answers to an
open-book exam, as opposed to a guessing game of what
questions they will face on a closed-book exam that should
be committed to rote memory. The perceptions of harder
questions and second-guessing the instructor might create
anxiety for some students. In any case, these behaviors do
not create optimal learning environments, but when
recognized, can be mitigated.
hopes to add value to the rich open-book versus closed-book
exam debate in an effort to decrease the gap between the
knowledge that our students obtain in courses, the skills
they develop when taking our exams, and how they will
eventually perform in an operational setting.
investigated whether an open-book versus closed-book testing
protocol significantly impacted students’ performance on
major exams and their attitudes regarding the textbook and
the course. Specifically, we hypothesized:
H1: Students taking open-book pre-exam quizzes will perform
better on open-book exams than students taking closed-book
H2: Students taking open-book pre-exam quizzes will see
clearer links between the textbook and course material and
believe they learned more in the course than students taking
closed-book pre-exam quizzes.
our undergraduate introductory managerial accounting course
experienced a semester-long testing protocol incorporating
either open-book or closed-book pre-exam quizzes to prepare
for three major open-book exams during the semester. The
final exam was also an open-book exam. This offering is a
required course for all undergraduate Management Majors at
our AACSB-International Accredited business program and
represented one of the largest sample populations
available. Second, as mentioned earlier, the accounting
educational community has aggressively embraced alternative
assessment studies such as ours and suggest improving
introductory accounting courses with pedagogy that
emphasized increased student involvement in the learning
process (AECC, 1990; 1992). Third, the course had a robust
set of learning objectives which could be utilized in future
studies to measure students’ achievement of learning
objectives between testing protocols.
sample size consisted of 235 students across ten separate
sections of the course taught by four instructors. One-half
of each of the four instructors’ sections of students
prepared for each of the three open-book major exams with
two open-book pre-exam quizzes, while the other sections of
students prepared for these major exams with closed-book
pre-exam quizzes. Each instructor ensured that students’
grades would have no impact by the testing protocol they
experienced, and that there was no advantage for being a
student in either protocol. The students in the two testing
protocols took similar pre-exam quizzes and major exams
(i.e., similar conceptual questions with different numbers)
and the same final exam.
completion of the course, all students completed a survey of
questions investigating their attitudes toward open-book
testing and its relationship to the course, its textbook and
learning. The survey instrument was the same for all
instructors. Students responded to each statement using a
5-point Likert-scale where 1 was strongly disagree and 5 was
strongly agree. The statements examined whether students
saw clear links between the materials covered in the
textbook readings and class lectures, if the exams were
closely related to the textbook, if students recommended the
open-book testing approach, and if students’ instructors
provided suggestions on how to effectively use the
textbook. Additional questions surveyed students as to
whether they spent more time working problems or exercises
in the textbook, or more time preparing the textbook for use
as reference during the exam to prepare for the open-book
exams, and whether they felt they learned more or less using
the open-book testing approach. Finally, the survey asked
if they had a false sense of security in preparing for the
We used PASW
Statistics 18 to analyze our data (See Table 1). Our
findings indicated that students who took open-book pre-exam
quizzes did not perform significantly better on any of the
three open-book major exams than students who took
closed-book pre-exam quizzes. However, students who took
open-book pre-exam quizzes did perform significantly better
on the open-book final exam than students who took
closed-book pre-exam quizzes. As might be expected,
students’ performance on the open-book final exam was
significantly impacted by their performance on the three
open-book major exams. Thus, these findings only partially
supported our first hypothesis.
the survey data, we did find significant differences between
students in the open-book versus closed-book pre-exam quiz
sections. Specifically, students in the open-book sections
more strongly agreed that they saw clear links between
materials covered in textbook readings and class lectures (M
= 4.32 for open, 4.12 for closed) and the exams were closely
related to textbook (M = 4.18 for open, 3.99 for closed)
than students in the closed-book sections. The open-book
students also more strongly agreed that their instructor
provided suggestions on how to effectively use the textbook
(M = 4.04 for open, 3.82 for closed). Although there was
not a statistically significant difference between open-book
and closed-book students and their recommendation of the
open-book testing method, 82.6 percent of the students
strongly agreed or agreed with the statement “I recommend
the open textbook method for other classes (M = 4.26)”.
statistically significant across our open-book versus
closed-book testing protocol, 51.3 percent of students said
they spent more time working problems/exercises in the
textbook to prepare for the open-book exams, whereas 48.7
percent of student said they spent more time preparing the
textbook for use as reference during the exam. Eighty-eight
percent of students felt they learned more using the
open-book testing approach with no statistically significant
difference between students in the open-book versus
closed-book sections. We did find a statistically
significant difference between students in the open-book and
closed-book sections and their having a false sense of
security in preparing for open-book exams. Specifically,
although 70.5 percent of all students indicated they did not
have a false sense of security in preparing for the
open-book exams, students in the open-book sections had less
of a false sense of security, as the mean for students in
the open-book sections was 0.23 and 0.36 for students in the
closed-book sections (where 0 was ‘no’ and 1 was ‘yes’).
Thus, our students did not possess overconfidence in
preparing for the open-book exams.
Table 1. Means, Standard Deviations,
NOTE: N = 235.
* p < .10. ** p < .05. *** p < .001.
Findings and Discussion
found only partial support for our hypothesis that open-book
pre-exam quizzes would significantly increase students’
performance on open-book major exams, we believe our study
was successful in demonstrating usefulness of this testing
protocol. Specifically, students like the idea of open-book
exams, but not necessarily for the reasons educators might
think. Anecdotally, students indicated they learned more
through the open-book testing approach than they do through
the conventional closed-book approach regardless of their
grades on the assessments, because they were able to focus
on mastering concepts to solve the accounting problems
rather than memorize technical aspects they could look up in
the textbook. Further, they also indicated they did not
have a false sense of security going into the open-book
exams, as they knew they would have to be able to work
through the problems and apply concepts rather than simply
report facts from the textbook, as well as having a
time-constraint. These findings can provide a foundation for
suggested open-book exam “best practices.” The key to
successful implementation and increased student learning
rests with how well the open-book testing protocol and
process are described to the students. Also, best practices
might include effort to ensure instructors are as unified as
possible in their desire to investigate new approaches to
enhancing student learning.
Conclusion and Suggestions
for Future Research
In our quest
to prepare our students for careers in the “open-book world”
they will encounter after graduation, we believe open-book
exam approach is useful for enhancing student learning while
effectively preparing our students for real-world
operational decision making. Even though our study did not
answer the proposition of whether open-book testing
encourages life-long learning in students by enlightening
them that they do not need to “know” all the answers, we
still feel that referring to source material for guidance is
an attribute. There were indications that open-book testing
encouraged active student engagement in their learning, and
in many cases expanded their confidence in being able to
work through difficult concepts. This as well as several
other areas warrant further investigation, and we offer
several suggestions for future study.
foremost, it would be beneficial to conduct a follow-on
study in which the open-book versus closed-book treatments
are carried throughout the entire course with all the
instructors. Additionally, even with over 250 students, in
our opinion, we did not have enough subjects to warrant the
multiple experimental treatments; open-book and closed-book
pre-exam quizzes crossed with open-book and closed-book
exams. Specifically, we recommend having one-fourth of the
students in a large population course experience the
following treatments: open-book pre-exam quizzes and
open-book exams, closed-book pre-exam quizzes and open-book
exams, open-book pre-exam quizzes and closed-book exams, and
closed-book pre-exam quizzes and closed-book exams. We
believe this approach would accommodate a more robust study
and may produce significant findings.
also be very interesting to conduct longitudinal studies of
students at various levels of academic experience and
development. These longitudinal studies offer the promise
of understanding open-book preparation and performance
differences for students with different levels of experience
and academic maturity. Well-constructed longitudinal
studies could also follow specific students throughout their
academic careers to evaluate possible effects the open-book
methodology has on student development and performance.
These longitudinal studies can also give greater insight
into possible negative impacts of open-book exams on student
learning and performance. For example, mixed findings have
been reported regarding long-term or delayed retention of
material covered in open-book exam approaches (Agarwal,
Karpicke, Kang, Roediger, & McDermott, 2008). Longitudinal
studies are important as these long-term effects of the
methodology are more carefully studied and reported.
potential research opportunities in the study of open-book
exams exist. Educators agree that open-book exams reduce
anxiety, and students appear to consistently prefer
open-book exams despite the acknowledge understanding that
they generally require the exhibition of higher-order
thinking skills (Brightwell, Daniel, & Stewart, 2004).
However, as discussed earlier, questions might actually be
harder on an open-book exam because they are testing at a
higher level of learning. Consequently, we suggest pre-test
post-test treatments of attitudes associated with this
phenomenon. Also, although open-book exams do not
necessarily lead to higher achievement in terms of test
scores, they do seem to reduce unnecessary rote memorization
of facts which allows students to prepare for exams more
constructively (Theophilides & Dionysiou, 1996). We did not
specifically address this phenomenon, but recognized it
during our study and suggest future investigation.
exams may also need new instructional techniques that
address different cognitive processes and knowledge levels.
In other words, perhaps studies should be conducted on the
manner in which a student approaches a problem, based on the
open-book versus closed-book experience. Related to this
idea, open-book exams may be more difficult to construct (Eilertsen
& Valdermo, 2000), and faculty may spend relatively too much
time preparing and grading open-book exams while other
faculty may also be insufficiently trained or experienced in
open-book exam construction (Vanderburgh, 2005). We suggest
additional research be oriented toward the instructor, not
just the student.
also numerous other hybrid options of open-book versus
closed-book exam protocols; including an open hand-written
note-card approach that show promise in not only high levels
of learning but enjoyment of the course (Block, 2012).
Research, such as the one conducted by Gharib, Phillips, &
Mathew (2012) studying “cheat-sheets” versus open-book or
closed-book exams, warrant further investigation.
discussed early, no open-book exam study would be complete
without properly addressing advancements that are occurring
in the on-line education environment. We feel it should not
be the ubiquity of the internet that drives the exam
approach; it should be a question of if students learn
better via open-book exams that also allow access to the
internet. We feel that perhaps the richest potential area
of study may be to investigate the learning effects of not
only open-book exams, but open-computer exams as well. To
increase the real-world feel of an exam and to better
replicate a future professional work environment, open-book
open-computer exams, such as those suggested by Williams and
Wong (2009), could be implemented and their impact on
learning enhancement studied.
research oriented toward addressing these potential research
topics will contribute to the ongoing open-book versus
closed-book exam debate. These educational process
improvement efforts will help address demands that the
educational community enhance student learning and success,
as well as future professional performance.
hopeful that educators in other disciplines find our study
useful for their course development. At a minimum, we hope
the readers of this study gained insight into a testing
protocol they may have dismissed in the past, but now may
consider adding to their assurance of learning repertoire in
their quest to enhance their students’ learning and future
AACSB International Accreditation Coordinating Committee. (2007). AACSB
assurance of learning standards: An interpretation. An
AACSB White Paper, 20.
Education Change Commission. (1990). Position statement
number one: Objectives of education for accountants.
Issues in Accounting Education, 5, 307-312.
Accounting Education Change Commission. (1992). The first course in
accounting: Position statement No. 2. Issues in
Accounting Education, 7(2),249-251.
Ackerman R., & Leiser D. (2104). The effect of concrete supplements on
metacognitive regulation during learning and open-book test
taking. British Journal of Educational Psychology,
Agarwal, P. K., Karpicke, J. D., Kang, S. H. K., Roediger, H. L. III, &
McDermott, K. B. (2008). Examining the testing effect with
open-and closed-book tests. Applied Cognitive Psychology,
Agarwal, P. K., & Roediger, H.L. III. (2011). Expectancy of an open-book
test decreases performance on delayed closed-book test.
Memory, 19(8), 836-852.
Albrecht, W. S., & Sack, R. J. (2000). Accounting education: Charting
the course through a perilous future (Vol.). Sarasota, FL:
American Accounting Association.
Allitizer, R. L., & Clausen, T. S. (2008). Computer-mediated exams:
Student perceptions and performance. Advances in
Accounting, Finance and Economics,1(1), 1-7.
Anakwe, B. (2008). Comparison of student performance in paper-based
versus computer-based testing. Journal of Education for
Business, 84(1) 13-17.
Anderson-Fletcher, E. (2005). Going from zero to sixty in twelve months:
Implementing assessment at the Bauer College of Business. In
K. Martel & T. Calderon (Eds.), Assessment of student
learning in business schools: Best practices each step of
the way (pp. 64-83). Tallahassee, FL: Association for
Institutional Research, AASCB International.
Apostolou, B., Watson, S. F., Hassell, J. M., & Webber, S. A. (2001).
Accounting education literature review (1997-1999).
Journal of Accounting Education, 19(1), 1-61.
Baillie, C., & Toohey, S. (1997). The “power test:” Its impact on
student learning in a materials science course for
engineering students. Assessment and Evaluation in Higher
Education, 22(1), 33-49.
Barr, R. B., & Tagg, J. (1995). From teaching to learning - A new
paradigm for undergraduate education. Change: The
magazine of higher learning, 27(6), 12-26.
Block, R. M. (2012). A discussion of the effects of open-book and
closed-book exams on student achievement in an introductory
statistics course. PRIMUS, 22(3), 228-238.
Boniface, D. (1985). Candidates’ use of notes and textbooks during an
open-book examination. Educational Research, 27(3),
Brightwell, R., Daniel, J. H., & Stewart, A. (2004). Evaluation: Is an
open book examination easier? Bioscience Education
E-Journal, 3. Retrieved February 11, 2015, from
Bruckman, A. (2002). The future of e-learning communities.
Communications of the ACM, 45(4), 60-63.
Collett, N. J. (2000). Innovation or renovation: Effective project
design for accounting and MBA students. Accounting
Education, 9(1), 67-92.
Damast, A. (2007, May 3). Duke works to stem Fuqua fallout.
BusinessWeek. Retrieved February 11, 2015, from
Entwistle, N. (1997). Contrasting perspectives in learning. In: F.
Marton, D. Hounselland, & N. Entwistle (Eds.). The
Experience of Learning: Implications for teaching and
studying in higher education, 3rd (Internet) edition
(pp. 106-125). Edinburgh: University of Edinburgh,
Centre for Teaching, Learning and Assessment. Retrieved
February 11, 2015, from
Eilertsen, T. V., & Valdermo, O. (2000). Open-book assessment: A
contribution to improved learning? Studies in Educational
Evaluation, 26(2), 91-103.
Feller, M. (1994). Open-book testing and education for the future.
Studies in Educational Evaluation, 20(2),
Gharib, A., Phillips, W., & Mathew, N. (2012). Cheat sheet or open-book?
A comparison of the effects of exam types on performance,
retention and anxiety. Psychology Research, 2(8),
469-478. Retrieved February 11, 2015 from
Heijne-Penninga, M., Kuks, J. B. M., Hofman, W. H. A., & Cohen-Schotanus, J.
(2008). Influence of open-and closed-book tests on medical
students’ learning approaches. Medical Education,
Heijne-Penninga, M., Kuks, J.B.M., Schonrock-Adema, J., Snijders,
T.A.B., & Cohen-Schotanus, J. (2008). Open-book tests to
complement assessment-programmes: Analysis of open and
closed-book tests. Advances in Health Sciences Education,
Herrington, J., & Standen, P. (2000). Moving from an instructivist to a
constructivist multimedia learning environment. Journal
of Educational Multimedia and Hypermedia, 9(3),
Ioannidou, M. K. (1997). Testing and life-long learning: Open-book and
closed-book examination in a university course. Studies
in Educational Evaluation, 23(2), 131-139.
Kalish, R. A. (1958). An experimental evaluation of the open book
examination. Journal of Educational Psychology, 49(4),
Moore, R., & Jensen, P. A. (2007). Do open-book exams impede long-term
learning in introductory biology courses?. Journal of
College Science Teaching, 36(7), 46-49.
Paisey, C., & Paisey, N. J. (2004). An analysis of accounting education
research in accounting education: an international journal -
Phillips, B. J., & Phillips, F. (2007). Sink or skim: Textbook reading
behaviors of introductory accounting students. Issues in
Accounting Education, 22(1), 21-44.
Phillips, G. (2006). Using open-book tests to strengthen the study
skills of community-college biology students. Journal of
Adolescent and Adult Literacy, 49(7), 574-582.
Pollitz, J. H., Christie, A., & Middleton, C. (2009). Management of
library course reserves and the textbook affordability
crisis. Journal of Access Services, 6(4), 459-484.
Rakes, G. C. (2008). Open book testing in online learning environments.
Journal of Interactive Online Learning, 7(1),
Ramaley, J. A., & Leakes, A. (2002). Greater expectations: a new
vision of learning as the nation goes to college.
Association of American Colleges and Universities.
Roediger, H. L., & Karpicke, J. D. (2006a). The power of testing
memory: Basic research and implications of educational
practice. Perspectives on Psychological Science, 1(3),
Roediger, H. L., & Karpicke, J. D. (2006b) Test-enhanced learning taking
memory tests improves long-term retention. Psychological
Science, 17(3), 249-255.
Siegel, G., & Sorensen, J. E. (1999). Counting more, counting less:
Transformation in the management accounting profession - The
1999 practice analysis of management accounting.
Montvale, NJ: Institute of Management Accountants.
Sosbe, T. (2005). Testing by the book. Certification Magazine,
Supiano, B. (2009). The billion dollar question: Which students benefit?
The Chronicle of Higher Education, 55(26), A22-23.
Theophilides C., & Dionysiou, O. (1996). The major functions of the
open-book examination at the university level: A factor
analytic study. Studies in Educational Evaluation,
Theophilides, C., & Koutselini, M. (2000). Study behavior in the
closed-book and the open-book examination: A comparative
analysis. Educational Research and Evaluation, 6(4),
US Government Accountability Office (2005). College textbooks
enhanced offerings appear to drive recent price increases.
(Publication No. GAO-05-806). Retrieved February 11, 2015,
Vanderburgh, P. M. (2005). Open-book tests and student-authored exam
questions as useful tools to increase critical thinking.
Advances in Physiology Education, 29(3), 183-184.
Yang, Y., & Cornelious, L. F. (2005). Preparing instructors for quality
online instruction. Online Journal of Distance Learning
and Administration, 8 (1). Retrieved February 11,
2015, from http://www.westga.edu/%7Edistance/ojdla/spring81/spring81.htm
Weisberg, M. (2011). Student attitudes and behaviors toward digital
textbooks. Publishing Research Quarterly, 27(2),
Williams, J. B. (2006) The place of the closed book, invigilated final
examination in a knowledge economy. Educational Media
International, 43(2), 107-119.
Williams, J. B., & Wong, A. (2009). The efficacy of final examinations:
A comparative study of closed-book, invigilated exams and
open-book, open-web exams. British Journal of Education
Technology, 40(2), 227-236.
attempt to enhance learning, we are very interested in your
opinion about how we present course material. You are to
read each statement and indicate your own personal feelings
about the use of textbooks in this course by marking how
much you agree with each of the following statements. Using
the scale below, fill in the appropriate bubble on your
General Answer Sheet.
C D E
Strongly Disagree Neutral Agree
1) I use
textbooks to prep for exams in other classes in the same way
I am using my textbook to prep for exams in my accounting
instructor provided suggestions on how to effectively use my
3) I use
the textbook publisher’s website to supplement the textbook.
4) I see
clear links between materials covered in textbook readings
and class lectures.
were closely related to the textbook.
receiving my score from a graded test, I reviewed the tested
material again in the textbook.
recommend the open textbook method for other classes.
8) I plan
to sell my accounting textbook at the end of the semester.
price I paid for my accounting textbook was fair.
unremitting and nonsensical use of mildly amusing clipart in
class lectures made it difficult to focus on learning
objectives from the textbook.
provide short answers (not just “yes” or “no” responses) to
the following questions in the area provided.
was your strategy for using your textbook in your accounting
12) Did you
spend more time working on problems/exercises in the text or
more time preparing the text for use as a reference during
13) Do you
feel you learned more or less using the open textbook
14) Did you
have a false sense of security in preparing for the
open-textbook exams? Please discuss why or why not.
provide any other comments or observations you may have on
the use of textbooks in this course.