At some point in their educations, students must learn copious
amounts of information. To do this, they use a variety of well-known
strategies such as study groups, note-taking services, and videotapes
of lectures. In fall 2004, a group of first-year dental students at the
University of Michigan (U-M) School of Dentistry asked to have all
dental school lectures videotaped and recordings made available on a
Web site. The students’ doubted their ability to accurately
summarize in their notes the quantity of information presented in
lectures. The students thought that reviewing a video recording of each
lecture would help them better retain the biomedical information
presented.
The Office of Dental Informatics is responsible for the development,
implementation, and evaluation of learning-technology activities in the
school. This includes faculty development, course Web pages, research
with and about learning technology, and the formative and summative
evaluation of learning technology projects. The office is also
responsible for the acquisition, installation, and operation of
classroom and computer lab equipment.
In response to the students’ request, the Dental Informatics
group applied formative evaluation strategies to determine the ultimate
solution. The group determined that podcasting (see the sidebar) audio
recordings of lectures provided a better technology solution for the
students’ needs than the originally requested video recordings.
Formative Evaluation and
Instructional Design
Instructional design and formative evaluation strategies are
commonly used in developing instructional products, especially for
computer-based instruction and other learning technologies. Formative
evaluation provides information to help monitor and improve product
development to ensure that it meets its intended goals.1
Instructional design consists of several stages (see Figure 1). In
each stage, designers use formative evaluation techniques to obtain
feedback concerning the product from clients, subject matter experts,
colleagues, and learners.2 Feedback gathered via formative
evaluation in one stage of the instructional design process is used in
subsequent stages to help improve the product. Formative evaluation
information is collected in four ways: self-reporting, observation,
tests, and records.3 Self-reporting, the most commonly used
method, refers to users’ directly relaying their experiences with
or opinions of the product, generally by means of a questionnaire,
survey, or interview with an evaluator. Observing users’ behavior
and interaction with the products is another popular method for
collecting information. When trying to determine whether cognitive or
behavioral skills have been affected, tests are generally used.
Occasionally records and documents, such as server access log files,
are used to gather information on the frequency of downloads—data
that can give an idea of how much a product is being used.
Click image for larger view.
In the Define stage of instructional design, developers begin
defining the scope of the learning activity, identifying learner
characteristics, establishing constraints, and collecting resources.
During this stage, the learning technology team members, who have
limited knowledge of the subject matter and the intended audience,
commonly make many of the design suggestions. They gather information
about the target audience’s prior knowledge, interests, and
experiences with the subject matter through interviews and focus
groups. This information guides design decisions.
The Design stage consists of creating a prototype, flowcharts, and
storyboards. Feedback is gathered by assessing the user
audience’s attention, comprehension, information retention,
personal involvement, and user-computer comfort.4 These data
guide revisions during production of the product, potentially
eliminating costly and time-consuming changes at a later stage if the
product does not meet user needs.
During the Development stage, the product is created, tested, and revised until the client is satisfied.5
Formative evaluations occasionally are used outside of the design,
development, and implementation stages for a product. Often it is
difficult to find funding, resources, and support for the development
of a product without results showing that it will effectively
accomplish its intended goal.6 The results of a formative
evaluation can serve as an indicator of a product’s success with
its intended audience, which in turn can tip the scale toward securing
resources needed for further development.
Instructors often hesitate to integrate new products or technology
into their courses without evidence that it will benefit student
learning. Information retrieved from a formative evaluation can help
them determine whether a product should be implemented.
Once the product has been deployed in its intended setting,
formative evaluation can serve several different purposes. Developers
can use feedback to make small improvements that were not anticipated
during the initial development, for example. Feedback can also help
steer future iterations of the product. If the product will be used in
a different environment or with a different audience, information
gathered during this part of the evaluation can guide reconfiguration
of the product.
Formative Evaluation: A Case Study
Formative evaluation strategies are used routinely when developing
computer-based instructional programs such as patient simulations or
tutorials. When students requested that lectures be videotaped and made
available on a Web site, they didn’t get an immediate decision to
grant or deny the request. A research university such as the U-M values
inquiry. This culture made it natural to apply strategies to
systematically determine whether videotaping was the best solution. We
conducted a series of three pilot studies, using formative evaluation
and instructional design techniques to guide the process. Flagg7
described four types of formative evaluation measures, two of which we
used in this project—self-reporting and records. No tests were
administered, nor were observations of students conducted.
Students initiated and supported the project, so were equal partners
from the beginning in the formative evaluation process. An Advisory
Group consisting of five dental students and one representative from
Dental Informatics was formed to direct the project. The staff member
supported the project’s technical activities.
The Advisory Group decided to conduct a pilot in single course, with
the goal of determining whether video recordings would be the most
beneficial media format. The group chose six questions to answer before
making a final recommendation:
From the beginning we expected that one pilot could not answer every
question. At the same time, we knew we needed answers to all the
questions to reach a complete solution.
The results of the first pilot would determine the project’s
direction and yield additional questions. The first pilot focused on
answering question 1, the second pilot answered question 2, and the
third and final pilot answered question 3. We examined questions 4, 5,
and 6 across all three pilots.
Certain constraints placed on the project contained costs and
ensured student involvement. First, students were responsible for
obtaining instructor permission to record lectures. Second, students
had to provide their own playback devices. The school provided
technical support. A university grant from the provost’s office
funded the few additional expenses.
Pilot 1—Media Format
The focus of this pilot was to answer the question, What is the best media format for lecture review?
Pilot 1 Methods. The Advisory Group selected
part of a microbiology course for the pilot because of the difficulty
of the content and the dependence on diagrams and other visuals during
the presentation. Faculty permission was obtained to record the
lectures and post the electronic presentation files on the course Web
site.
The microbiology course met three times weekly for a total of 3.5
hours per week. The Advisory Group discussed possible media formats and
chose three types: (1) video, (2) audio synced with the images from a
PowerPoint presentation, and (3) audio only. The Dental Informatics
staff member attended and recorded each lecture, using a digital video
(DV) camera.
Each resulting DV file was exported as a video file, audio synced
with PowerPoint slide images, and saved as audio only. Two days after
the lecture, these files were posted on the course Web site, created
using CTools. The U-M’s course management software, CTools was
developed using the open source content management system Sakai.
Because of the university-imposed file size limit, only links to the
files were posted in CTools; the School of Dentistry stored the media
files on its QuickTime streaming server. It took the technical lead on
the project approximately 3.5 hours to record, complete the
postproduction process, and post the files for each hour of lecture.
The self-reporting measures for learning needs were a 12-question survey administered to the entire class (N
= 105) one week after the pilot concluded and a focus group of six
students immediately following the survey. For records, we looked at
server logs to gauge the frequency with which students used each of the
three media types. Finally, records were kept of the time spent by the
technical staff on the project in order to calculate cost.
Pilot 1 Results. This pilot focused on
determining the best media format for lectures (question 1). The three
formative evaluation measures—student survey, focus group, and
server logs—showed that students preferred the audio-only format.
Of a possible 105 participants, 30 downloaded media directly from the
Web server. Server logs revealed that 20 percent of downloads were
video, 14 percent audio synced with PowerPoint, and 66 percent audio
only. The average time from posting date to download was 16.2 hours.
The nature of electronic files makes it impossible to determine usage
among those who obtained the media in other ways (from a friend burning
a CD, for example, direct file transfer from another user, or media
used in groups).
Of the 105 students in the class, 70 (66.67 percent) completed the
survey. Due to the formative nature of this project, we report only the
percentage of students responding to a question. In Table 1 the second
column reports the percentage of survey respondents who selected that
answer to the question, and the third column shows responses from
students who reported using the lecture download system.
Click image for larger view.
The results from the students who used some form of media are
clear and confirmed in the server log records. Responding users both
preferred (66.1 percent) and used (66.1 percent) audio-only over both
the requested format of video and PowerPoint synced with audio. They
primarily reviewed lectures, although a small percentage (9.1 percent)
used the online lecture as a replacement for attending class. Most of
the students reported using the files to study at home, but some also
used them to take advantage of down time when working out at the gym
(8.8 percent) or during their commute (8.8 percent) to school. Of the
students who used the media, some downloaded their media of choice on a
regular basis as soon as it was available (25.9 percent). More students
tended to download the files close to the relevant exam (44.4 percent).
In addition to an analysis of the server logs and the survey, the
focus group provided more in-depth comments. Four of the focus group
participants downloaded and used media files; two did not. The focus
group discussion probed in more detail the questions asked in the
survey. Comments from the group corroborated findings from other
sources, specifically that students preferred audio as the media choice.
The focus group also helped us further explore some of the survey
findings. For example, in the survey an overwhelming number of users
(87.3 percent) reported the CTools Web site easy to use. When we
explored the same question in the focus group, however, they revealed
that downloading is a cumbersome method of acquiring files from
numerous course Web sites. They suggested a number of possible
improvements (an issue addressed in the third pilot). Focus group
participants also reported that the most popular playback devices were
personal computers and iPods.
Results of the survey, consultation with the Advisory Group, and
responses from the students in the focus group helped us address which
courses, if recorded, would most benefit students (question 4). Survey
options focused on information-dense course content with heavy reliance
on visuals. The course selected by the most students (48.3 percent),
histology, involves great detail and a large number of diagrams. This
meant that they could concentrate on what was being said during the
lecture about the visual materials instead of trying to be
stenographers capturing all the information being presented. The audio
recording served as a safety net that enabled students to listen to the
lecture repeatedly for information they didn’t record in their
notes during class. They identified the next course for recording,
biochemistry (45 percent), based on the faculty lecturer’s fast
speaking rate. The fast-paced delivery made it difficult to record all
the important information in their notes, so the ability to review the
lecture was extremely important.
To determine the support costs in terms of staff time and workflow
(question 5), the technical staff member documented the time spent
recording, processing, and posting the files. Recording time for one
hour of class plus 2.5 hours for processing time (from DV to the three
formats) was 3.5 hours total staff time. Additional time costs such as
equipment set-up were not tracked.
Question 6 asked whether the number of students participating
warrants the cost of the project. The dental class has approximately
105 students. Participation in the first pilot was solely by word of
mouth. The server logs revealed that approximately 30 students (29
percent) participated voluntarily in the pilot, which we considered
successful. Also, the four students in the focus group who had used the
recordings were adamant about how helpful they found the recordings.
Pilot 1 Conclusions and Decision. The
results of the first pilot clearly indicated students’ preference
for the mobility of the audio recordings rather than video. The
overwhelming support for audio combined with the low cost of producing
the audio files compared to video resulted in the decision to commit to
the project and to conduct subsequent pilots to answer the technical
questions about acquisition and dissemination.
The Design stage of instructional design relies on feedback to
eliminate expensive mistakes. Formative evaluation strategies used to
gather student feedback early in this project provided critical
information that redirected the focus of the content delivery. The
result was a satisfactory and cost-effective solution requiring less
technical support than a video solution.
At this stage we had identified a workable, though incomplete,
solution. Having found students’ preferred media choice for
reviewing lecture content, we could have stopped our formative
evaluation at this point, adopted the chosen system, and attempted to
scale its implementation school-wide. However, questions remained about
how best to capture and disseminate these audio recordings.
Pilot 2—Acquisition Solutions
For clarity, we describe pilots 2 and 3 as two separate pilots
although they ran in parallel. After pilot 1 revealed students’
strong preference for audio, the logistics of acquiring high-quality
recordings of lectures at a reasonable price became the focus of pilot
2. This pilot included two courses, for a total of six class hours each
week.
Pilot 2 Methods. From the beginning, we
attempted to contain costs. Because the iPod would be a low-cost
solution, we explored it first as an audio capture device. Students
reported using iPods to record lectures, and a few students placed
iPods with supplementary microphones on their desks in the front row of
the lecture halls. This method produced unsatisfactory audio quality
and was highly dependent on lecturer position. Because of this, the
Office of Dental Informatics used a Belkin Universal Microphone Adapter
to connect an iPod directly to the lecture hall’s amplified PA
system. Students immediately reported the resulting audio was of
extremely poor quality and almost useless. The inability to accurately
monitor audio levels for the iPod along with the iPod’s limited
recoding frequency (16-bit mono, 8 KHz equivalent to analog telephone
quality) resulted in extremely poor recordings.
Next we explored using a computer to capture the audio—an
Apple Powerbook G4. The analog audio signal from the classroom’s
PA systems was fed into a computer and captured using Apple’s
QuickTime Broadcaster. The Dental Informatics staff added metadata
(date, course name, instructor, and lecture title) to the completed
recording and posted the file to a Web site.
Using a computer for both media capture and processing streamlined
the workflow (see Figure 2). This reduced the time associated with
capturing, converting, and posting files. Additionally, eliminating
video as a delivery format removed the need to transfer media from DV
tape to computer.
Click image for larger view.
The most expensive component of any technology project is the
technical staff’s time. Thus, pilot 2 attempted to automate the
recording process. Once we finalized the computer platform as the
acquisition solution, we turned to automating the work performed by the
technical staff. AppleScript scripts were written to automate the
recording, file processing, and uploading workflow. We modified the
processing script to also create an Advanced Audio Codec (AAC) file as
an audio option for download. AAC is a subset of MPEG-4 and allows for
several advanced features like bookmarking and playback speed changes.
It is also the native format for the iPod, which many students reported
using.
Five students who volunteered to manage these tasks were trained in
the three-step process. At the beginning of a class lecture they
selected an icon from the computer’s desktop that ran a compiled
AppleScript application. This launched QuickTime Broadcaster and began
the recording using presets designed by the technical staff. The script
required student action at the end of the lecture by displaying an
“Add to iTunes” dialog box. Once the student clicked
“Add to iTunes,” the script requested metadata (lecture
title, lecturer’s name, and course) and added further
system-generated metadata (date and time). The file was then
transferred to a processing machine that automatically converted the
recording to both audio formats and uploaded it to the Web site. After
the class ended, the audio files were posted to the Web site.
The second pilot used two formative evaluation
strategies—self-reporting and records. The self-reporting
measures included a focus group and student e-mail notifications of
problems. For records, we examined server logs and kept records of the
time spent by technical staff on the project in order to calculate cost.
Pilot 2 Results. While the computer solution
was more expensive than using the iPod for audio capture, responses
from students and staff clearly indicated the superior audio quality
using a computer to capture the audio from the room’s public
address system. That answered question 2, about the best solution.
To determine costs in terms of staff time and workflow (question 5),
we added up time required for the steps involved. Reducing the number
of media formats lessened the recording, processing, and posting time
to two hours per class hour. Automating the process of converting and
posting files using Apple’s AppleScript technology also speeded
the process. Staff processing time dropped to 15 minutes a week (mostly
maintenance on processing machines), and files are available on the Web
within five minutes of a lecture’s conclusion.
Pilot 2 Conclusion and Decision. Now that
more information has been published about portable audio and iPods,
high-quality audio production is clearly the most critical component.8
While the tools used to create the audio files (see the sidebar) are
becoming easier to use, the process still requires professional
technical expertise.
The results of the second pilot clearly indicated that a low-cost
computer could easily capture high-quality audio recordings of
classroom lectures from the room PA system. In addition, automating the
recording, processing, and posting workflow greatly reduced staff time.
These two low-cost solutions meant that the school could afford to
sustain the project over time. The automated process combined with
student support allowed the project to scale up to the point that all
lecture halls now are equipped with lecture recording capabilities.
We could have stopped the evaluation here. While the solution did
not represent the eventual final choice of podcasting, it did meet the
students’ desired media format, allowed them to review lecture
content, and eliminated expensive staffing costs.
Stopping at this point would leave several student needs unexplored,
however. Specifically, students requested that we examine ways of
automatically notifying them of updated content and provide better
tools to navigate available files, along with enhanced features for
working with long audio recordings.
Pilot 3—Dissemination Solutions
The final pilot focused on improving the process by which students
identified and downloaded new recordings. Pilot 3 used the formative
evaluation strategies of self-reporting and records: feedback from the
focus group and Advisory Group, and review of the server logs.
Pilot 3 Methods. After the first pilot, the
Advisory Group selected a course in which the faculty member spoke
fast. The Advisory Group also requested recordings of the Integrated
Medical Systems (IMS) series because of the large amount of difficult
material it contained. Students felt that having another source to
review material would be especially beneficial in this class. IMS
lecture series met several days each week, for a total of 15
hours—a three-fold increase in hours recorded from the first
pilot.
Once lectures from multiple courses were available, a focus group
revealed that students would prefer to go to a single Web page to
access the lecture recordings instead of moving between numerous course
Web sites. Thus, access through the course management system, CTools,
was discarded in favor of a custom-built dynamic Web site with data
contained in a MySQL database. This allowed students to more easily
sort files by name, media format, and class.
Pilot 3 Results. We answered question 3, the
best way to disseminate the media, by consulting with the Advisory
Group and obtaining feedback from the focus group. The students
reported equal use of MP3 and AAC files, accessed both on personal
computers and iPods. Examination of server logs confirmed this, showing
equal downloads of MP3 and AAC files. In the focus group, students
reported using the audio book feature of the iPod to speed up or slow
down a lecture. Students specifically pointed to this single feature
that made the iPod more useful than other audio players.
The Advisory Group concluded that the custom Web site, while an
improvement over the CTools site, still was not the easiest method for
obtaining the files. Feedback from the focus group revealed that
students wanted a centralized Web site to more conveniently access the
audio files. While the course Web site served adequately for one
course’s lectures, the increasing number audio lectures made it
more difficult to access new content.
The focus group also suggested creating either a notification system
using e-mail or a subscription service for automatic notification when
files were posted. We added RSS (Real Simple Syndication) so that
students would not have to check for new files. Server logs
subsequently showed that 50 percent of files were downloaded via RSS.
Server logs served to measure student participation and thus help us
determine whether the number of students participating warranted the
cost of the project (question 6). The logs indicated no difference (30
students) at the end of the second pilot but an increase to 60 students
(of a possible 105) at the end of the third pilot. Students reported
that the convenience of obtaining files via RRS increased the
likelihood that they would download files. Server logs confirmed this,
revealing a much higher download volume compared to earlier in the
project.
Pilot 3 Conclusion and Discussion. The third
pilot gave solid answers to the project’s final two questions.
Because students used both MP3 and AAC formats equally, the decision
was made to provide both formats. Data also indicated that if lecture
recordings were convenient—from a central Web site and/or through
RSS—a significant number of students would use the functionality
voluntarily. As access became more convenient, student usage of the
lecture recordings increased from 28 percent to 57 percent of students
in the class using the service. The marked increase in users and usage
helped the school commit to the project.
Implementation Realities
The three pilots answered the six initial questions and solidified
the U-M School of Dentistry’s commitment to offer podcasts of
lectures as a teaching and learning service. The final step was to make
the service routinely available. Two very significant issues remained,
however: faculty support for podcasting, and the institution’s
ability to sustain podcasting into the future.
While the faculty involved in the three pilots were enthusiastic
about participating in a research and development project, having
lectures routinely recorded and distributed raises issues of
intellectual property. Conflicts between a lecturer’s
intellectual property rights and the need of students to acquire media
for review must be mediated in a way acceptable to both groups.
Once the school chose podcasting as a routine service, the
pilot’s makeshift authentication and authorization process needed
to be replaced by a very robust system. Through collaboration with the
U-M ITCS, the School of Dentistry adopted the U-M’s already
implemented authentication and authorization system, cosign. This
guaranteed that only U-M dental and dental hygiene students could
access the lecture podcasts and eliminated the potential intellectual
property issue, thus helping with faculty acceptance.
While the U-M School of Dentistry had proven that lecture podcasts
could be done for a reasonable cost, the developed system was
proprietary to the dental school. To guarantee that podcasting would be
sustained at a reasonable cost over an extended period of time and that
it would continue to evolve, it needed to be offered university-wide.
Ideally, the knowledge, processes, and services developed within the
School of Dentistry would be shared with the university community and
beyond. In partnership with Apple Computer, Inc., a follow-up pilot
investigated whether iTunes U could expand on the services provided by
the School of Dentistry. While still in progress, that pilot’s
initial results have proven positive, and the U-M and Apple Computer
are now investigating integrating iTunes U into Sakai. If that project
succeeds, then the School of Dentistry will use the Sakai podcasting
software, guaranteeing that podcasting lectures will occur routinely
for an extended period of time and at a very low cost.
Faculty and students are interested in exploring the expansion of
podcasting services as a separate project with its own analysis,
design, and development cycle. It is important to draw a distinction
between this deliberate project expansion9 and scope creep.
Scope creep refers to uncontrolled changes in a project’s goals
that cause the project to drift away from its original purpose. Our
purpose is to continue to research ways to aid student learning.
We will evaluate requests for new features with two intentions: that
all improvements assist student learning, and that all future
developments can be integrated into Sakai. For example, the processing
scripts are being written to easily support complementary media
components (PDFs, PowerPoint files, images) that can be associated with
a lecture’s audio file.
Lessons Learned
The unanticipated results of this project strongly reinforced two
lessons that can be applied to most learning technology projects: (1)
the importance of actively involving the client, and (2) the importance
of using proven instructional design and formative evaluation
techniques. The delivery mode of podcasting lectures (audio only) did
not match the students’ initial request. Indeed, stopping after
any of the pilots would have met some, but not all, of the
students’ needs as uncovered in our formative evaluation. An
interim solution also would not have encompassed both the user needs
(audio format, automatic download, easy browsing) and the institutional
needs (automatic recording and processing, full integration into
existing technologies) as podcasting did. By involving students in the
design process and applying proven formative evaluation methods, the
school avoided implementing a system that would have cost more and
might not have met student needs as well as podcasting lectures.
The students’ commitment through the entire project was
evidenced by their enthusiastic participation in focus groups, on the
Advisory Group, and on the survey. From the beginning, students knew
they were guiding the development of the project, and they sustained
their energy and commitment through completion. They now share in the
pride of seeing how their efforts benefit their classmates and the
classes of dental students following them. They also share in the
presentations, press releases, and writing about the project, as in
this article.
Technology is commonly implemented into teaching and learning
situations without using instructional design and formative evaluation
strategies or involving students. Faculty and administrators
(developers) usually make the decisions. Unfortunately, many novel and
innovative projects do not succeed or have disappointing results. Using
both the formative evaluation strategies suggested by Flagg10 and the instructional design process described by Alessi and Trollip11
helped us identify and adjust to unexpected circumstances and develop a
successful technical solution to a learning dilemma. Ultimately, use of
these strategies provided the critical data required to ensure
long-term and ongoing support.
of the contributions of numerous persons. We would like to especially
acknowledge the contributions of the University of Michigan School of
Dentistry Class of 2008 for working hard to improve their educational
experience and the education of the dental students who will follow
them. Drs. Dennis Lopatin and Dan Chiego deserve special recognition
for serving as the first two faculty volunteers to support this
student-initiated experiment.
of Dentistry both have a long tradition of valuing innovation in
teaching and learning. We thank the University of Michigan Office of
the Provost for its seed support of the project and the School of
Dentistry for its support in creating an environment where innovation
can thrive.
Sukwit and Sheri Schultz of Apple Computer, who had the early vision
and recognition that mobile devices such as iPods could be valuable
learning tools for students and that the School of Dentistry’s
project was an innovative use of the technology. Also from Apple
Computer, the iTunes engineers Eric Bailey and Sugam Jain deserve
thanks for their work on the iTunes project. Patrick McNeal from U-M
Information Technology Central Services is acknowledged for making the
U-M cosign authentication and authorization system work with this
project. We also thank Ruxandra Iacob for her help with the analysis
and Sharon Grayden for her editorial assistance.
for Implementation: Evaluating Educational Technology Applications and
Lessons,” American Journal of Evaluation, Vol. 25, No. 1, 2004, pp. 51–63.
Something in the Air: Podcasting in Education,” EDUCAUSE Review,
Vol. 40, No. 6, November/December 2005, pp. 33–46, <http://www.educause.edu/LibraryDetailPage/666?ID=ERM0561>.
