IWitness Evaluation Report

The Challenge of Embracing Change

A Tribute to Chesley Beaver

By Kim Simon

We are living during a time of transition, a period between generations, when many Holocaust survivors— our friends and loved ones— are passing on. Death after survival has become an unfortunate and regular presence. Yet as an organization, the Institute is animated by hopefulness, by glimpses of inspiration, and by dedicated people who are driven to affirm life in every day’s effort.

Among the most spirited of those people was Chesley Beaver, a friend and colleague who served as our evaluation consultant, helping us measure the change our work creates in the world.  Chesley embraced change with courage unlike anyone I’ve ever known. After his sudden death last year, we were and continue to be challenged to muster similar courage in his absence.

It was our fortune to work with Chesley almost continuously for three years, beginning in 2009 to evaluate the Institute’s first prototype of IWitness. Throughout our collaboration, I marveled at his ability to draw people toward him— individuals and groups of all ages, backgrounds, and interests. I remember how easily he related to students, scholars, educators, and supporters. People offered feedback and gave their time even when there was little time to give, all because of Chesley’s singular magnetism. He was passionately and intensely interested in people and ideas and even more so in how ideas could be applied.

In his evaluation reports, as seen in the example below, we did not just get mountains of data (which we did need, of course). Chesley made sure that there was always a practical application developed from the results. A terrific collaborator, he would usher us through the information and move us from passive reading to active understanding and application: to equip us with insight that would lead to improvements in our methodology and the IWitness platform.

Evaluation allows us to ascertain the value of an activity. It compels us to determine its meaning.  We have built a structure for evidence-based evaluation of the use of testimony in education. It is not enough to say that watching testimony has the potential to increase knowledge, change attitudes, and develop the capacity to engage in society responsibly. We need evidence. We have begun, therefore to define a model to measure attitudinal shifts, as well as lay the groundwork for long-term studies of the effect of testimony-based learning on student attitudes.  Measuring behavior and attitudes is difficult— a challenge for the entire emergent field of education evaluation—Chesley had the intellect, vision, and passion to partner with us to innovate and bring both practice and creativity to the challenge. 

A beloved friend and colleague, Chesley’s legacy and memory at the Institute is life-affirming: We will maintain and build upon what he helped us to begin—the inspiration, determination, and open-mindedness to search for the meaning and integrity of what we do.

IWitness Evaluation Report

By Chesley Beaver

For the Institute, evaluation is an evidence-based activity directed at collecting, analyzing, and communicating information on the workings and effectiveness of IWitness as an educational product.  The adoption of a social science approach reflects the Institute’s commitment to academic rigor and continued collaborations with scholars.  Throughout the testing process, USC Institutional Review Board provided the third-party oversight necessary for academic integration. 

Watching Testimony

System analytics, built into the IWitness application, track the amount of testimony watched by users, as well as the sources, i.e., individual survivors or witnesses. On average, each student tester watched roughly 90 minutes of testimony from four or more survivors/witnesses.  In total, the 75 student testers used IWitness to watch more than 102 hours of video testimony from 345 different survivors or witnesses— representing more than one-third of all the survivors/witnesses included in IWitness at the time of testing.[i] On average, each time a student played a portion of video testimony they viewed approximately six minutes and 11 seconds.

The depth (amount) and breadth of testimony (number of different survivors/witnesses) watched by students during the classroom tests were significantly higher than anticipated. Personal Strength to Survive, an activity on the topics of cultural and spiritual resistance, was consistently used in all classroom tests and system analytics reveal a less than 30 percent overlap in the survivors/witnesses accessed by students.  This suggests that students engaged with the activity and “resistance-related” search topics in unique and individual ways.  This user-driven approach to searching testimonies has consistently been noted by students as one of the features they like most when using IWitness. 

Emotional and Cognitive Factors 

Further insights on testimony come from two open-ended or unaided broad questions asked of students:  “What will you remember most about IWitness?” and “What did you like most about IWitness?”  Students overwhelmingly identify video testimony as the most memorable aspect of their experience on IWitness. 

Analysis of students’ comments reveals the emotional and personal connections expressed in response to watching testimony.  Students continually described the experience of watching video testimony by using words and phrases like “personal,” “feel their emotion,” “feel like I know the Holocaust survivor,” and “people I met.”  As one student noted when testing the IWitness prototype in April 2010, “… it goes past the computer screen and feels like she [survivor] is actually telling her story directly to me.”

Videos/testimonies were also what students liked most about IWitness.  The combination of these two elements—testimonies being the most memorable and well-liked aspect of IWitness—lays the foundation for significant cognitive gains in the educational use of video testimony.

The potential cognitive gains from using video testimony in the classroom are significant, and merit further investigation and validation with a larger student population.  Pretest and posttest measures of students’ perceived cognition, combined with comments from open-ended/unaided questions, show that IWitness enabled a deeper and multifaceted understanding of the Holocaust.  Noteworthy are the ways in which students recognized the integrity and validity of primary sources, as well as the understanding that comes from multiple perspectives on the same topic.  As a result, students who tested IWitness reported a 40 percent greater perceived knowledge about the Holocaust compared to the control group.

Opinions, Attitudes, and Stated Behavior

A series of questions in the pretest and posttest online surveys were designed to gauge students’ opinions about the subject matter and attitudes on the broader topics of racism, stereotyping, and social action.  These questions were first used as part of the Echoes and Reflections curriculum studies.[iii]  These findings should be used to provide directional insights and not statistical certainty.  Greater statistical validity will be provided, as many of these questions will continue to be asked of students at various intervals following their engagement with IWitness.  In addition, the Institute is exploring opportunities for longitudinal studies in order to understand the longer-term influence on attitudes and behaviors. 

Most notable from these findings is the increased interest students expressed in studying the Holocaust after using IWitness, and the increased perceived relevance the Holocaust has to their individual lives.  Nine out of 10 students rated IWitness better than other websites used for school.  Furthermore, 69 percent want to use IWitness outside the classroom for other school projects and/or personal research.  One interesting observation that merits further investigation is that for a small percentage of students (less than 15 percent) IWitness entrenched their negative opinions about the subject matter and deepened their lack of interest and perceived importance of the Holocaust.

Subject Matter Interest and Importance

Overall, after students tested IWitness, they were twice as interested in studying about the Holocaust as students in the control group, who were not exposed to IWitness.[iv]  Furthermore, the depth of interest is stronger with students who used IWitness – the number of students in the testing group who were “very interested” or “extremely interested” in studying the Holocaust was double that of their peers in the control group.  Students also rated the perceived importance of the Holocaust. While a majority of both groups found studying the Holocaust to be important, students testing IWitness reported stronger levels of agreement.

 

Subject Matter Relevance

 

Students who tested IWitness reported significant changes in how the Holocaust relates to their lives.  For a quarter of students, their experience testing IWitness resulted in the highest relevance rating possible— a 32 percent increase between pretest and posttest ratings for the same students.  Furthermore, when compared to the control group, students who tested IWitness were twice as likely to “strongly agree” that the effects of the Holocaust are still being felt today.

 

Social Action

Results from pretest and posttest student surveys show that students’ use of IWitness significantly increased two social-action measures— “speaking up” and “one person can make a difference.” After testing IWitness, three out of four students agreed that speaking up against stereotyping is important and that one person can make a difference. Most notable is the level of agreement expressed by students relative to these two social-action concepts. As the illustration below highlights, there was a 30 percent increase in the strongest level of agreement (“strongly agree”).  Furthermore, students who used IWitness are 35 were more likely to agree that, “It is important that everyone challenge racism.” Along with data on student interest and relevance, this finding indicates the potential IWitness has to engage students and connect them to the subject matter in a relevant and meaningful way.

Stated Behavior

The last question for students testing IWitness was projective—asking students to think if this experience will influence how they perceive and treat others from different backgrounds.  While the results require additional longitudinal measurements to distinguish intent from actual behavior, a vast majority stated that their behavior would change.  Of note, the 19 percent of students who said “no” or “I’m not sure” also had the lowest ratings for subject matter interest, relevance, and importance.

It is interesting to compare these findings to Echoes and Reflections, primarily a written curriculum with selected testimony clips.  Students using the Echoes and Reflections curriculum spent up to a semester covering multiple topics about the Holocaust. These Echoes and Reflections students were asked the same question, and 63 percent stated that their behavior would change as a result.  However, as noted by evaluators in these previous studies, longer exposure to the complex historical topic (Holocaust) could lead to doubt about perceived knowledge regarding the details.  These comparisons, along with other findings, prompt further investigation with greater numbers of students.

 

This article first appeared in the Summer 2013 issue of PastFoward, the Institute's digest.