The objectives of my revised research proposal (originally submitted for consideration in fall 2005) are as follows: (1) to develop a series of programmed, online laboratory tutorials for students in Anthropology 101L: Physical Anthropology Laboratory (1 Unit, hereafter referred to as “Anthro 101L”)  through the COC  “Blackboard” online platform; (2) to test and evaluate the effectiveness of the tutorials with statistical data collected from Blackboard and feedback obtained from student questionnaires; (3) to adapt each tutorial to the stated objectives, instructions, contents and procedures used for in-class laboratory assignments; and (4) to devise practical methods to motivate students to actively utilize computer-assisted materials for in-class lab preparation (See the Revised Proposal submitted November 2005).

This (revised) proposal is framed under an assumption that key anthropological theory, topical concepts, terminology and background information from the co-requisite basic, together with the essential measurement, analytical and descriptive tools needed for specific lab exercises, must be appropriately integrated and incorporated into individual tutorials at every stage of the design process; further, that student usage of the tutorials is itself a learning process that must be continually monitored, evaluated and, when necessary, occasion additional revisions to individual tutorials.

This is deemed to be a critical need for many Anthro 101L students, who currently may take the co-requisite Anthropology 101 course (“Anthro 101) from any instructor, either concurrently with the lab, or have completed Anthro 101 up to one year prior to taking Anthro 101L. Following the initial design of fifteen new tutorials in early 2005 (and revising parts of each during the summer and fall 2005 semesters per the principles outlined above), the first series of seven tutorials were presented to spring 2006 lab students in two of the three sections of Anthropology 101L: Physical Anthropology Laboratory. Students accessed each tutorial via Blackboard and used them as a sole resource in preparing for in-class laboratory exercises. They evaluated each tutorial via written questionnaires (Questionnaire B) and, following preliminary data summary and analysis, two slightly different versions of a revised Questionnaire C were prepared for use in the remaining labs. One version of Questionnaire C was administered to the original group of two lab sections (designated “experimental”). A second version was used to elicit comparable data from students in the third lab section (the “control” group), who used a written lab manual/workbook without access to online tutorials during the semester. 

Table A shows a partial data summary taken from the supplemental Questionnaire A administered to lab students at the first lab period of the spring semester (See “Evaluation Methodology” in Revised Proposal, page 4):

TABLE A

 Table A shows (see Column I) that of 89 originally-registered students, 35 (approximately 39%) had previously completed Anthro 101. A set of 30 fixed-choice, matching and open-ended definition quiz questions on basic lecture course content was administered to students in the “previously completed” group in all lab sections. Quiz results (see Column VII) show average correct answers in each section (M 12:00 pm, 3:00 pm and 7:00 pm) were 21.5, 16.7 and 14.1 out of a possible 30, respectively. “Previously completed” students in all lab sections appear not to have (1) initially learned or (2) remembered in detail much of the content from the lecture course. That some of this knowledge was initially learned but not remembered by some students, may be indicated by the self-reported final grades for 27 of these students in Column VIII of 13 “A’s,” 9 “B’s,” 5 “C’s,” 0 “D’s” and 0 “F’s.”  The relatively high number of “A” and “B” grades may indicate that, if many did well initially in the lecture course, many also forgot what they had learned by quiz time. The number of unreported grades (8 out of 35) of all students quizzed also indicates that perhaps significant learning for at least some students never occurred. Either way, data from Questionnaire A, taken together with the continuing stream of anecdotal requests, complaints and comments from students regarding their ongoing need for more background and contextual information for individual labs, shows that both the major assumption of this study and its practical goals are valid:

Online tutorials should be designed in recognition of identified problems in prerequisite and co-requisite course articulation and variability by capitalizing on the relevance, quality, accessibility, and diversity of learning resources available on the world wide web; individual tutorials should effectively compensate for any perceived deficiencies in previous student preparation and/or learning and retention; tutorials should provide up-to-date and well-integrated background, and lab-specific information in a manner appropriate to the different cognitive modes and learning styles of individual students; and tutorials should be used in a manner that maximizes the possibilities for collaborative student  success in a laboratory learning environment.  (See “Hybrid Model for a Laboratory Course” in Revised Proposal, page 6)

An initial series of fifteen (15) online tutorials were prepared during summer 2005 and made available to lab students via Blackboard on a “non-required” basis in fall 2005 as a supplement to the regular lab handouts and workbook (see page 2 of the Revised Proposal for full description of the process). These tutorials were re-designed during the semester (with the aid of student input), with the view of using them during subsequent semesters as “stand alone” resources along with the written lab handouts (also revised). The goal was to replace the currently-used lab manual (which may students complained (quite rightly) was both too complicated and too expensive. Consequently, as revised online tutorials were implemented (usually one per week) and loaded on to Blackboard for the experimental (EXP) group, no additional supplemental or background resources were available to EXP students during the semester--except for my usual procedural briefing, usually given at the start of each lab session, and the written reference and procedural information in the lab handout. Control group students (CON), however, continued to use the manual/workbook (with the same briefing and written handout) throughout the semester without access to Blackboard or the online tutorials.

All course documents (especially the course syllabus) were amended, and new versions of each were prepared for the EXP and CON sections that clearly reflected the different expectations and procedures for each group. An in-lab introduction on accessing and using Blackboard was provided to both EXP sections during the first class section on February 6, 2006 and a full explanation of the research project, its objectives and procedures, was provided to both the EXP and CON groups. For the EXP students, special emphasis was placed on the need for diligent, continuing student evaluation for each tutorial throughout the semester. Questionnaire A was also administered to all 89 originally-registered students during this first class period.     

Each week during the semester, EXP students were expected (and encouraged) to access the appropriate online tutorial, which listed the objectives and necessary background and practical skills websites for review and practice and featured an organized set of web contents for each of the individual lab stations. Some websites (in whole or in part) were used several times; others provided very small, focused “pieces” of information and practicum, intended to guide the student step-by-step through the lab preparation. Explanatory text connected the individual site links into a narrative that guided students through the entire exercise. Locating, evaluating and adapting sites that appealed to the different learning styles and cognitive modes of online learners were major objectives in designing each tutorial. Students regularly provided feedback as to “what worked” versus “what didn’t work.”

Each week, individuals completed a “Tutorial Quiz” based upon their online study and personal interaction with the tutorial contents. Parts of the “quiz” were simply calculated to see if the tutorial had been accessed in whole or in part. Others allowed for analysis and recording of critical information from the tutorial for later application in the lab. Students were directed to turn in quizzes immediately upon entering the lab (to discourage sharing of information and to make overt copying more difficult). Quizzes were promptly graded and usually returned within ten minutes for use in the lab. Questionnaire B (the tutorial evaluation form) was detached and retained by the instructor. It was reviewed on-the-spot for correct student I.D information, completeness, and the presence of open-ended student comments (exemplars of the “Lab Tutorial A Quiz” and the Questionnaire B for Lab A are attached to the hard copy version of this report). Questionnaires with incomplete or ambiguous responses were immediately returned to individual students for completion and/or correction. Each completed questionnaire earned five points of academic credit—a decision based on the fact student input was a critical part of the tutorial use-process throughout the course. Because of the openness and enthusiasm and consistency demonstrated by the students throughout this process, and the many useful comments received, I am reasonably confident that this methodology was sound, and that the information collected is both valid and reliable.      

Questionnaire B results for Labs A through G are summarized in Table B below. Eight statements (Q1 through Q8) were listed sequentially. After completing work on the tutorial, the student was requested to circle a response of either “Yes,” “Mostly” or “No” to each statement. Two additional questions (Q9 and Q10) asked for the number of minutes spent reviewing tutorial contents (Q9) and completing the tutorial quiz and evaluation form (Q10). Two additional requests to list three of the “most and least useful” websites by name were included, together with a final open-ended question: “Do you have suggestions on how to improve this online tutorial and make it more useful to students?” and both categories yielded additional, valuable information. Statements and questions in Questionnaire B were presented in the forms appearing in the heading of Table B below. Note also that Lab B totals do not include data from the 7 PM section due to the instructor’s illness on Feb 20, 2006. Calculated percentages were based on total actual number of “Yes,” “Mostly” and “No” responses returned by students.

TABLE B


Data from Labs A through H summarized in Table B shows that the initial set of tutorials were fairly well received by the majority of lab students, as evidenced in their responses within the categories of access and navigation ease (Q1 and Q2), overall content (Q3 and Q4), usefulness for actual lab preparation (Q5 through Q7) and recommended use by others (Q8). “No” responses varied from a low of 4% in Lab E to a high of 18% in Lab C, while overall “Yes” and “Mostly” totaled 91% of all responses on all seven labs. The results for the “average time spent” questions, however, varied widely. The calculated variances between averages from lab to lab suggest several possible explanations, but current data from Table B in these two categories is not considered reliable. The 116 open-ended written responses received on Questionnaire B, however, taken together with more ad hoc and anecdotal student information, provided a basis for revising Questionnaire C for use in obtaining data from Labs H through M in the second half of the course.

The revised Questionnaire C does not include a “Tutorial Quiz,” since it was felt that the majority of EXP group students were now accessing Blackboard on a regular basis and had developed reasonable diligence in using and applying the tutorials to their lab work. The revised Questionnaire C, however, did allow for first-time data to be collected from CON group students as well (in slightly-modified format) who had been exclusively using the lab manual/workbook. It also allowed for a systematic comparison of remaining lab data from both CON and EXP groups. Both the CON and EXP versions of Questionnaire C have been included in the Appendix to the hard copy version of this report. Data from all CON and EXP students completing Labs H through M appears in Table C below. Note also that the possible range of responses to each statement was increased to five from the initial three to clarify the inherent ambiguity of the “Mostly” response in Questionnaire B:” “Strongly Agree,” “Agree,” “No Opinion,” “Disagree” and “Strongly Disagree.”

TABLE C

One trend apparent in the comparison between students using both the CON and EXP groups in Labs I through M, appears to be that the tutorial-using EXP group spends more time in outside lab preparation (approximately 51 minutes per lab). The manual-reading CON group, however, take only 40% of this average (approximately 31 minutes) to prepare for their in-class labs. For both groups, this represents considerably less than the conventional three hours usually recommended for outside class/lab preparation. It should be noted that care was taken throughout the semester (especially during the latter half) to ensure that student responses on Questionnaire C--especially in the “timekeeping” categories--were as reliable as possible. This was accomplished by spot-checking all individual questionnaires as they were submitted, and making immediate inquiries about apparently missing or incomplete data, or data that did not look right. Students in both groups proved to be consistent and co-operative in providing good data throughout the semester and as a result, this researcher is confident that the information obtained (except where otherwise indicated) is both valid and reliable. Based on many ad hoc discussions with individual students, one plausible explanation for the apparent differences in outside lab preparation between the two groups may be that, because online tutorials cannot be directly accessed during class (i.e. no computers available in the lab), the tutorial-dependent EXP students had no choice but to complete lab preparations outside class. The manual-users, however, had the option of both outside preparation, plus the additional advantage of being able to refer to the manual in the lab during the actual lab exercise. Notwithstanding the ever-present constant (albeit gentle) grumblings about “how much work” is involved for a one-unit course, however, neither group reported spending even one full hour each week in outside lab preparation. Indeed, providing online resources that will enable students to learn effectively and efficiently and continue to motivate greater efforts in regular outside preparation, remains a major challenge in tutorial design. 

A second trend in the Table C data above shows that for Labs I through M, members of the EXP group either “Strongly Agree” or “Agree” with all the statements over 75% of the time, and “Disagree” or “Strongly Disagree” only 6% (with 19%  having “No Opinion”). In the CON group, the pattern of results is similar: 66% “Strongly Agree” or “Agree,” and 13% “Disagree” or “Strongly Disagree.” But does a 19% “No Opinion,” showing from the EXP group (coupled with the 75% “Strongly Agree” or “Agree” showing among them) mean that the tutorial users are “more satisfied with” or “better at” using tutorials than written manuals during their lab preparations? The tutorial users appear slightly more satisfied using tutorials (presumably having used conventional written texts and manuals throughout their college courses), but manual users (who share this common resource background) showed only slightly less satisfaction in their use of the more conventional written materials. Referring back to Table A, however, we note that only 40% of all students in the EXP group (27 of 68 students) reported taking even one previous online class of any type during their college careers—a total of only 30 students out of 89 overall (an “un-whopping” 34% of all lab students enrolled!) “pre-adapted” to the unique advantages of online learning. What is heartening (and significant), however, is how quickly the EXP tutorial users adapted to the use of relatively new and unfamiliar (and previously unannounced) online resources. In doing so, they may have slightly outdone their manual-reading CON counterparts, who were already well-adapted to the exclusive use of written resource materials in their lab preparations and may account for their relatively greater number of “Disagree” and “Strongly Disagree” responses (84, or 13%) among the CON students, and the relatively fewer numbers of “Strongly Agree” and “Agree” responses (458, or 66%) reported for Labs I through M on Questionnaire C.

Overall, the pattern of responses on Questionnaire C shows that students in both CON and EXP groups benefited about equally from the online and manual/workbook lab-prep resources provided to each, respectively. Each group felt reasonably well-prepared for the in-class labs, although many continued to complain about the lack of “background” and “explanation” for particular exercises. Yet the students in both groups remained positive about the quality and utility of the materials they used, despite spending relatively little outside time studying and preparing for lab exercises. A final question remains: How well did each of these two groups actually perform in the lab course?

TABLE D
Table D above shows the distribution of final lab grades for students in both the CON and EXP sections. This represents the percentage (40%) portion of the course grade determined by performance in the lab exercises and “homework” (online tutorials and lab manual preparation), as opposed to the portion from lab exams and quizzes (60%). The overall results are summarized below:

CON group students totaled 89% “A’s” and “B’s” for their work in the lab (53% and 37%, respectively), while EXP students earned “A” and “B” grades which equaled 67% (52% plus 15%) of all lab grades awarded within their group. This similarity in distribution is also repeated at the other end of the curve, where the manual-reading CON students received 11% “D’s” and no “F’s” and the tutorial-using EXP’s shared 16% (10% and 6%) of the lower-end grades received within their respective groups. The relative percentage of “A’s,” D’s,” and “F’s” in both groups was virtually the same, with the “B’s” showing a divergence of 37% among the CON’s and only 15% among the EXP’s. Since no attempt was made in this research to formally link individual student performance using either manuals or tutorials, with final point-totals and course grades, we can only conclude that students using either written or online resources have a roughly equal chance of performing at similar levels in Anthro 101L. Further, specific levels of performance among individual students probably have very little to do with either their previous completion of the Anthro 101 lecture course. A second, more tentative conclusion (not directly tested) is that prior online coursework is a poor predictor of student success in this particular course context. This research did indicate, however, that new relationships between conventional “hands-on” laboratory pedagogy coupled with unique possibilities offered by new online learning modalities, promise new and exciting ways to enrich the laboratory experience for students and provide new opportunities for innovative research and course improvement by their instructors.    

The first modification to the revised proposal was necessary because of a flaw in the Blackboard programming (see Progress Report 1, January 2006). This caused an inability to gather usage data or compile statistics for any of the “Content” files—specifically, the online tutorials. As a result, the only usage data available during the first six weeks was initial student logons tracked by how quickly (or slowly) individual email addresses were entered into the system. Students without entered addresses were regularly asked if they were using the Blackboard tutorials and, if not, to begin using them immediately. Blackboard “Statistics” provided additional indirect data by date and time of day in the “Non-Content” areas (e.g., “Announcements” and “Discussion Board” postings). Tallies confirmed that substantial numbers of students (nearly one-third) had not accessed Blackboard even once during the first two weeks of the course. Continued, daily checks showed that most students did not begin actively using Blackboard until the fourth week of the course. Despite the fact that the “Content” statistics report only individual “hits” (and not the exact time of day or duration of access for each), this data would have provided only limited insight into how students would have actually used the tutorials. However, the inability of Blackboard to provide information in this category is a major flaw and was an impediment to this research project.

The second modification was needed because the need for a separate evaluation of both CON and EXP groups using comparable data had always been deemed critical. In writing the revised proposal, however, I had inadvertently neglected to specify exactly how such data could be obtained from both groups. After reviewing the Questionnaire B results during the first four weeks (in particular, the written “comments” provided by EXP students), I decided that a “mid-course” correction was indicated in the form of a revised Questionnaire B in two separate versions. The first would be used by the tutorial-using EXP students and the second by students in the manual-using CON group. Revised formats (collectively designated as Questionnaire C) are described above and were used to collect data from both control and experimental groups in Labs H through M inclusive, and were later summarized in Table C.

The third modification (actually a “modification-by-default”) was in regard to the tutorial-revision process. The original plan described in the revised proposal was to use individually-selected students, who would directly participate in the process of revising, editing and testing individual tutorials in exchange for HITE-Project or extra credit. I was frankly unable to recruit any volunteers (no one was interested after four weeks of recruitment efforts); therefore, student input to the tutorial revision process was limited to data available from student questionnaires A through C and the ad hoc student comments (and complaints) that regularly came throughout the semester—some spontaneously, others in response to my questions.    

The first “lesson” I learned from this research was that student performance in Anthro 101L appears to have little or nothing to do with whether they had previously completed the Anthro 101 lecture course, or were taking it concurrently with the lab. Scores on the “content retention quiz” (see Questionnaire A) showed that only one-half to two-thirds of the “basics” from the lecture course were either not retained or not learned at all by students taking the quiz. This strongly suggests that all background materials supporting Anthro101L lab exercises need to be well-integrated into each lab exercise. The specific modality of these materials (whether online tutorials, written manual/workbook, CD-ROM (or a combination of these), appears far less important than how capable online tutorials (or other resources) are in providing remedial  “lecture” information to students. It is also extremely important that these online or other resources be tailored to the individual student requirements on an “as-needed” basis in particular labs. Indeed, there appears to be a multi-functional role for “tutorials” to play in laboratory teaching and learning and for other, more conventional resources as well. This was well-demonstrated in the overall course performance of both the CON and EXP groups in Table D. These facts present additional challenges, as well: How to design and deliver a meaningful lab course, in which the majority of participating students have little or no science background and little motivation for success in science per se beyond the prospect of earning a valuable transfer unit in laboratory science. Based on the experience of doing this study, I believe that this can be accomplished—certainly with the aid of online resources, though probably with more conventional ones as well.
  
The second lesson learned is that perhaps the tutorial concept itself, in some instances, might be inverted; that is, instead of first requiring students to complete a full lab preparation (i.e. online tutorial), a more appropriate method might be to require a lab “follow-up” instead. This would be done by re-designing certain lab exercises to allow students to first interact in an experiment,  simulation-game or other class exercise, and then “respond” (as individuals or in small groups) through discussion board and web-based (tutorial) resources and finally submit a write-up on the process. This approach is already used for field trips: students first access the L.A. Zoo Website to obtain background information on the species they will observe; then they go to the zoo and conduct structured observations of primate behavior and record the information according to principles learned by watching film-clips in the lab; working in groups of two, they then compare notes and prepare a written report. A similar procedure is followed for an extra-credit field trip to the La Brea Tarpits. I plan to try out this “tutorial inversions” idea (tutorial perversions?) by re-designing several lab topics for fall 2006.
 
The third lesson is the importance of direct participation by students in the tutorial design, revision and implementation process as the primary “users” of online tutorials.  During this project, informal student input proved to be as important as the more structured questionnaire responses they provided. Students regularly answered questions on issues related to how well specific websites actually worked, how adequately the tutorials “steered” users through various alternatives, and how relevant tutorial contents were to what was going on in the lab. This study experience was a particularly valuable one in allowing me to elaborate specific questions in the above areas and obtain systematic answers (i.e. Questionnaires B and C). While it was disappointing not to find individuals willing to collaborate more intensively on this project, the idea of “student evaluators” remains viable for application in future labs. Regular student input would also be invaluable in helping to determine the “cost-benefit” relationships for new commercial products as they become available. There are many of these appearing every year, and there would truly be no substitute for hearing what students say about the quality and utility (and price!) of these books, manuals, CD-ROMS, “study guides,” etc. they may be required to purchase and use. Students need to “own” a part of the resource design and use process, and ways should be continually sought to actively involve them in the process of selecting, using and evaluating course materials.
 
The final lesson learned (or reinforced) is that most (lab) students are not as “computer- literate” as might be assumed based on the current popularity of online courses--or the fact that everyone seems to have a personal email address. Despite an almost universal access to personal computers, most students have not acquired basic specific internet search or research skills, possess online discussion board or “virtual classroom” experience (e.g. on Blackboard), or acquired any spreadsheet and word-processing expertise. These facts point to the need to develop lab resources that students will want to use regularly. This means that both the lab content and skills components (including computer usage skills) must be incorporated into the design process and then carefully monitored during actual use. One critical need for teaching this course in the future is the installation of computers in the lab to allow for immediate and continual reference of tutorial contents by students. Actual lab protocols, instructions, simulations would also be possible at each lab station. Another need is for software that will comprehensively track and record the results of specific student interactions with computer-based lab resources. This should help students (particularly those in non-science majors for whom this course was specifically designed) not only to complete individual lab assignments, but also provide more comprehensive opportunities to acquire scientific, research, computer  skills and knowledge beyond the immediate course objectives—tasks for which Blackboard is only partially suited. Finally, individually-accessible resources and built-in student progress tracking and assessment capabilities supporting a range of lab activities, would also begin to meet future needs for “make-up” labs, “open-labs” and more flexible, interdisciplinary learning in an expanding District learning environment.