I’d like to point back to Kathleen Blake Yancey’s “Looking Back as We Look Forward” from our histories unit. I found this essay especially helpful, and I’m sure I’ll refer back to it throughout this course. There, Yancey discusses validity and reliability (two terms that I’ll openly admit to having a hard time wrapping my mind around) as a pendulum swinging and assessment as “an exercise in balancing the twin concepts of validity and reliability” (135). For Yancey, “Validity means measuring what you intend to measure, reliability that you can measure it consistently” (135). This sounds simple enough, but I think it also raises really difficult questions about just what it is we’re trying to measure with tests and assessments, and how we can achieve any sort of reliability on something as seemingly subjective as writing assessment.
Camp, in her essay “Changing the Model for the Direct Assessment of Writing,” also points to a tension between reliability and validity, pointing to psychometrics and multiple-choice writing tests as examples of high validity. But she also calls into question whether these tests are, in fact, valid. I think it comes back to Yancey’s point: if validity means measuring what you intend to measure, we must ask ourselves, just what is it we’re trying to measure with multiple-choice writing tests? Camp points to Cooper and others, who argue that “multiple-choice tests do not sample the full range of knowledge and skills involved in writing” (105).
I do appreciate Moss’s assertion, based on Haswell, that writing assessment should be open to public review, which “not only enhances the program that is the focus of the evaluation effort, but more importantly enhances the practice of evaluation itself” (158). This gets me to my own feeling about these issues: in my own experience, I don’t see how one can create an assessment measure that is not entirely contextual. Is this valid? I think so, because it examines what is being tested and adjusts the assessment to an individual context? Is it reliable? Now, there I’m not so sure…
Writing Assessment and Technology 