College of the Canyons -- Title III

The Institute of Teaching and Learning

Instructional Innovation Classroom

Research Project Report

Name: = Rebecca Eikey

Email address: Rebecca.Eikey@canyons.edu

Phone Ext or Daytime Phone: 5314

Department: = Chemistry

Full-Time/Adjunct: full-time

Classroom Research Project Proposal Title

Evaluation of Calibrated Peer Review Program as a Tool to Facilitate the Development of Writing Skills in Chemistry

In the space below, present your research question: that is, what did you want to discover from your classroom projec= t?

I wanted to understand if and how the Calibrated Peer Review (CPR) assignments affected students’ perception on their understanding of the material, technical writing skills, and reviewing abilities.

In the space below, present a summary of your research project (approximately 500-1000 words), including a general description of the scope and nature of your project, the design or the outl= ine that you followed. Include objectives, activities, timelines, data collecti= on procedures, evaluation methodology, and/or benchmarks that provided progress measures for your project.

My research project focused on the use of Calibrated= Peer Review (CPR) assignments in a general chemistry course. The Calibrated Peer= Review program is an online program and is divided into three stages. The first st= age is the text entry stage, where students review source material and compose = an essay of required length using a list of suggested guiding questions. The second stage consists of reviewing three “calibrated” essays. T= hese essays are reviewed according to a set of given questions and then rated us= ing a scale of 1-10 (1 =3D very poor; 10 =3D excellent). This stage is meant to= help prepare students in the act of reviewing.&= nbsp; The reviewing consists of evaluating correctness of content and effectiveness of communication, including grammatical structure. Their performance in this stage affects how much their reviews of their peers will weigh in the scoring process. The third stage consists of the peer review process, where the students review three peer essays anonymously, and the self-review process. Their score in the assignment is based on four parts: = the weighted average peer-score of their text/essay; their performance in the c= alibration stage; their peer reviewing score (as compared to their peers’ review= s of the same essays); their self-review score (as compared to their peers reviews). There were three CPR assignments given during the course of the semester. Each assignment was worth 40 points of t= heir grade (or 120 points total and 10% of their grade in the course). Each assignment was run over a two-week period, where the first week was given f= or the composing of the essay, and the second week was given for all reviewing components. The program is designed so that work can be saved and worked on= at one’s own pace.

It has been shown that the use of CPR assignments en= hances content learning and improves writing and reviewing skills. I wanted to test the impact of learning in my courses by aligning exam questions to content = in the CPR assignments. As explained above, through the use of the CPR program, the student is exposed to and evaluates six different essays on the same to= pic. This is in addition to the essay that the student wrote his/herself. Theref= ore, one would expect an enhanced understanding of the content material. The fir= st CPR assignment focused on the topic of reaction stoichiometry. This particular assignment starts = as a small-scale reaction performed during the laboratory meeting. The percentag= e of students who completed the assignment was 95%. For the exam question that was dir= ectly related to the CPR topic, the average score was 9 out of 15 (std dev 5) or a 60%. The first exam had 10 questions and was worth 100 points. = The high score was 83; low 25; average 58 (std dev 1= 8). The second CPR assignment was on basic atomic structure of an atom, using copper as the example element. The percentage of students who completed this assignment was also 95%. There were two exam questions related to the conte= nt of this second assignment. The average was 7 out of 12 (std dev 3) or 58% on question 1; and the average was 8 out of 9 (std dev 2) or = 89% on question 2. The second exam had 9 questions and was also worth 100 point= s. The high score was 99; low 44; average 68 (std d= ev 22). The final CPR assignment was on explaining the molecular structure of a molecule using bonding theories. The percentage of students who completed t= he final assignment was 100%. This assignment began a little sooner than the corresponding lecture material due to scheduling. There were two exam quest= ions relating directly to the content in the CPR assignment. The average on the first question was 12 out of 18 (std dev 5) or 6= 7%; the average on the second question was 8 out of 15 (std dev 4) or 53%. The third exam had 11 questions and was also worth 100 points. The high score w= as 100; low 35; average 62 (std dev 22). It appears= that the average on the questions related to the CPR content is consistent with = the overall average on the exam. In both cases, these averages are lowered than= the desired 75%. The exception is of one of the questions from exam 2, which ha= d a much higher average (89%) compared to overall average (68%) for the exam. A= few questions arise: 1. Assuming CPR enhances content understanding, how much lower would these scores have been in the absence of the assignments? 2. Are the exam questions written in the appropriate fashi= on so as to facilitate a correlation with the CPR assignments? 3. On an indivi= dual basis, how does the students’ exam scores correlate to their CPR essays – it is correct to expect a low exam sc= ore to correlate with an unsophisticated essay? In any event, clearly looking o= nly at average exam scores of 20 students doesn’t allow for one to draw a meaningful conclusion as to whether or not CPR assignments enhance understanding of the content.

The twenty students who participated in the CPR assignments were also asked to complete a survey at the end of the semester. All 20 students completed the survey. The following open-ended questions we= re asked:

1. What three things did you like best about the use of the Calibrated Peer Review= (CPR) assignments?

2. What three things did you like least about the use of the CPR assignments?

3. How has the use of the CPR assignments impacted your understan= ding of the material?

4. How has the use of the CPR assignments impacted your writing abilities?

5. How has the use of the CPR assignments impacted your reviewing abilities?

In response to the first question, three themes emerged as to = what the students liked best about the CPR assignments. One theme is on the desi= gn of program itself. For example, students liked that the program was easy to use, that they could resubmit the work and that all of this could be done at one’s own pace. A second theme that emerged deals with reviewing the essays. First, a couple of students identified in this question that the program helped to improve their reviewing skills. Secondly, the students li= ked that they were able to compare their understanding of the material to their peers’ through the essays. For example, one student explained that “being able to grade other text entries” helped to “furth= er understand the material as well as being able to gauge how much we already knew.” The third theme that emerged from this question deals with con= tent understanding. Thirteen students expressed an increase in content understan= ding through the use of the assignments in response to this question. Students identified that the need for additional research on the topic was useful and that they needed to have a better understanding of the material in order to communicate effectively through essay form. Finally, they identified that t= he repetitive nature of the program helped with content understanding. For example: “I believe the = use of the Calibrated Peer Review assignments provided us with additional practice= and allowed us to gain a better understanding for specific aspects of chemistry= that may not have been emphasized in class. The task of writing short essays with specific guidelines allows us to not only focus on facts and text material,= but also challenges us to fully understand the subject.”

In response to t= he second question, the majority of students (15) focused on two related aspec= ts: the word limit and the guiding questions. The students felt it was difficul= t to fully answer all the guiding questions within the specified word limit. In technical writing, one must write in a concise fashion. Therefore, it was a challenge for the students to learn how to write concisely for these given topics.

The final three survey questions were answered as follows. Seventeen students (85%) identified that the use of the CPR assignments enhanced their content understanding: “I think it made me understand = more because I was able to write about what I learned about, so if I understood the material, I could w= rite about it.” Ten students (50%) identified an improvement in their writ= ing abilities. These students identified that the CPR assignments forced them t= o be concise in their writing. As one student stated: “In my science class= es, I am now less wordy and ‘flowery’ in my writing style.” Interestingly, three students (15%) identified that being forced to write concisely had a negative impact on their writing ability. The rest of the students (7 or 35%) did not believe the assignments had any impact on their writing abilities. Finally, sixteen students (80%) believed that the assignments helped with their reviewing abilities. They cited that they are more aware and read more carefully when reviewing other essays. Other stude= nts identified that the calibration stage was beneficial for the peer and self-reviewing stage: “by showing me what I need to look for in the example responses, I was able to evaluate others … better.” Six students identified that the assignments either had no impact on their reviewing abilities or the impact was limited to the assignment itself and could not be translated to an overall improvement.

In conclusion, the use of the CPR assignments does help enhance content understanding in chemistry (85% of students identify in this manner= ), although this is not readily apparent when looking only at exam performance. Secondly, regard= ing the effect on students’ writing abilities, the largest impact was on teac= hing students to be concise, to the point, and direct when it comes to writing a= bout chemistry. Finally, the majority of students (80%) identified a positive im= pact or improvement on their reviewing abilities.

Briefly describe any modifications to your original research proposal.

The only modifications were on the content areas sel= ected for the second and last CPR assignments (as explained in the mid-semester report) .

How many students (approximately) participat= ed in the research project?

What did you (as an instructor) learn from y= our research, and in what way(s) will your future students benefit from this research?

I learned that the CPR assignments do help enhance instruction. I will continue to use the CPR program in enhance instruction = in my courses.