Grading Frenzy and Comments
COMMENTS ADDED MAY 2 at the end of this entry.
It’s the end of the year, and many of us are engaged in grading papers, responding to last minute drafts, and holding conferences with students. The
The process we used is interesting, and fairly standard in writing assessment. We score each paper using rubrics (or metrics as they are also known) for a total of 50 points. The rubrics assess various criteria such as development, organization, grammar and punctuation, spelling, quality of documentation, and so on—and for each criterion the rubric describes achievement at different levels: we used exceeds, meets, or does not meet expectations. See a sample rubric at http://writingcenter.tamu.edu/content/view/239/67/. For each assignment and type of document the criteria used and the points awarded can be adjusted, depending on what the assessment is looking for and what the assignment is meant to teach.
Team one consisted of six graders, Team two of eight, and Team three of four. They were grouped according to expertise and background. All our graders have experience teaching or tutoring writing or in the content areas for which they graded. Each team spent time “norming,” that is, scoring a common paper and then discussing their results until they had basic agreement on the criteria used in the rubric. Once norming was done, scoring the sets for each class began.
Each paper was scored twice, for a possible total of 100 points. If the two graders differed by more than 7 points, we had a third grader break the tie and use the two closest scores. That didn’t happen too often, luckily. It took us from 9 am until 3 pm to norm and score about 175 papers, with time allotted for lunch and breaks. (And we all thank Leah Curtis, my assistant, for feeding and watering us.) We looked at graduate papers from Education and undergraduate papers from Chemistry, Engineering Ethics, Psychology, Agricultural Development, Biomedical Science, Marketing, Accounting, Finance, Information Science, and Management. Thanks to all you instructors who allowed us to score your students’ papers and to the
In the next few weeks we will interpret this data and see what it can tell us. We envision it as a snapshot of the quality of student writing in the disciplines at TAMU two years after the first writing-intensive courses were required. (Remember that in the first year, 2004-05, very few W courses were offered, so the second year is much more like a startup year),. We also envision it as form of feedback: W course instructors will be able to use the information to see what their students are learning well, and where they can improve instruction. I can use it to help them. And it may provide other interesting information. For example, for three sections of one class we were able to score drafts completed before a peer review and drafts completed after, which might tell us something about the value of peer review.
Already I can assure you, with only a cursory glance, that there is room for improvement. Naturally, students won’t leave a single W course writing perfect papers. We noticed some students struggling with how to present an argument—the temptation to just state the facts seems to be strong; we noticed that working quotes gracefully into a text was beyond many students, and that others found applying theoretical concepts to personal experience challenging. We suspect we might have seen some plagiarism.
We did not notice appalling illiteracy. Yes, we saw some comma splices, fragments, dangling modifiers, and misspellings, and more than we’d like. But of more concern is that students seem reluctant to grapple with ideas in any degree of specificity or depth.
By the end of the day our graders were getting good agreement (we’ll be running reliability statistics, too), but they were tired. I’d like to hear about the experience from their point of view, so I am inviting them to add to this blog. And I am inviting any TAMU faculty member who’d like to refine a rubric, participate in a similar session with their department or graduate students, or have papers from a class assessed by outside readers, to drop me an email (v-balester@tamu.edu). I promise you will learn something of value.
Comment from Alan Brillant, Sunday, April 25
Thank you for letting me participate. I was in team one.
My teammates said they were in it for the money, which is okay. I was ashamed to admit out loud that I was in it for the experience and the fun as well as the money (which seemed very generous).
Blue Baker lunch was unexpected and much appreciated. As was the good coffee and other goodies throughout the day. Very hospitable, thank you.
I totally enjoyed myself. Didn’t feel fatigued at all. The working environment was superb. Best of all, it was a lot of fun working with such bright and interesting people, two of whom I had been in touch with before and was delighted to be in touch with again.
My partner, Liz, who had been through graduate school and hated grading papers, warned me it would be the worst experience of my life and I would rather dig ditches or wash dishes than grade papers by the time the day was over. Not at all! I enjoyed myself, had a great time. It was one of the most enjoyable and pleasurable work experiences of my life! I was sorry we couldn’t go on or do it again!
Comment from Sarah Peters, Thursday, April 28
I also found the day enjoyable and I felt like I gained some important insight into student writing. One thing that stood out to me was that there was often a correlation between the specificity of the assignment provided by the instructor and the quality of the student papers. When the assignment was detailed and specific, even suggesting specific kinds of formatting to improve the ease of reading, the student papers were more focused, concise, and clear. Those papers fulfilled the assignments better because they had a better understanding of the purpose of the assignment.
When an assignment was short, consisting of only two or three sentences, the student papers were vague, disjointed, and unfocused. They didn’t really seem to know what they wanted to say. The kind of technical writing that my group read is well suited to detailed and specific assignments, and when the teacher provides such instructions in class, students will learn how to effectively communicate in other assignments that require them to make more of those choices on their own.
Comment from Kimberley Parsley, Tuesday, May 2
Like Allen and Sarah, I too enjoyed the day. I had not graded papers in a very long time but for some reason I found evaluating these papers to be far easier than any of the 104 or tech writing courses I taught as a graduate assistant. I wouldn’t say the papers were “better” in any way. In fact, I found the subject matter and the writing to be very similar to the “better” papers I received in my 104 classes. However, when I graded for my own classes, I found I was always struggling with my assessments. I worried I was being too hard or too easy on mistakes I found especially appalling. Namely, the student’s ability to argue an issue with clarity and decisiveness seems especially important.
Like Sarah, I believe the papers could have benefited from a clear understanding of the assignment. I think the reason I found the papers so much easier to grade than the papers from my own courses was the use of the rubric. It gave me a clear definition of what to look for in the papers and how high or low to score the papers. I feel I was able to make better judgments with the rubric as my tool. Likewise, I think the students are better able to write the kinds of papers we are looking for in our classes when they are given clear instruction. But to play devil’s advocate, from my own experience working as a tech writer at
