The following was used as the basis for the keynote address at the Learning from Practice Exchange of Teacher Knowledge and Research, Saskatoon, November 23-24, 2007. Janet McVittie is Assistant Professor in Curriculum Studies at the University of Saskatchewan and Shelly Loeffler teaches in the EcoQuest program in the Saskatoon Public School Division. Both were involved in “Assessing the Hard to Assess: Student Learning in a Middle Years Outdoor Environmental School” (McDowell Research Project #130).

Lessons Learned: The Effects of Assessment on the Love of Learning

Janet McVittie
Shelly Loeffler

Introduction

A mother received a phone call from her son’s teacher: her son, in grade 3, had scored 32% on a math exam. The purpose of the call seemed to be to prepare the mother for what she would inevitably learn when her son arrived home from school. Prepared she was, but also absolutely shocked: how could this be, when he had won the math award in grade 2? When he got home, she looked at his exam, and saw that he had 100% on the 32% of the exam that he had finished. Why had he abandoned his exam? He explained to his mother that he had to go outside otherwise the big boys would have got his fort. The mother phoned his teacher to explain the low mark was all about a fort. The answer was: The mark stands.

On what do we assess students? If we are assessing math learning, this was not an appropriate assessment. On the other hand, if we are assessing commitment to completing academic work, this was appropriate. Depending on a teacher’s beliefs, assessment might involve more than finding out what a student knows; regardless of what a teacher believes assessment is for, it also teaches students lessons – lessons about what is of most value in school. Unfortunately, the lesson of what is of most value in school is not well “scaffolded” and many students don’t know why they have scored poorly on an exam or why they have scored poorly at the end of the year. Certainly, at some point, somehow, we teachers must teach students to stick with the task at hand, not to abandon their school work to defend a play fort from the big boys. Or is this a “certainly”? Must we teach this lesson? And there are a number of other questions: should we teach commitment to task while assessing math? How is commitment to task best taught and learned? And how is it best assessed? We do not have enough conversations about this in schools, nor in pre-service training, nor in professional development.

Consider another story, again shared by a mother. Her daughter came home from kindergarten in mid November, thrilled because she had got a gold star that day. She had earned the gold star for “being good”. That was nice, her mother assured her. The next day, again a gold star, again a happy little girl. On the third day, the mother, suspecting why the teacher had suddenly decided to implement the gold stars asked if one particular boy had got a gold star. The little girl snorted her derision. “Has he ever got a gold star?” the mother asked. Another snort of derision. That little boy, in the daughter’s opinion, could never be good. The difficult answers were yet to come: “Why are you good?” asked the mother. “To get the gold star,” was the answer. “Were you good before the teacher started giving gold stars?” “Yes.” “Why were you good then?” “For the gold star.” What had been intrinsic for this little girl seemed to have become extrinsic. Not only was she now performing (being good) for the gold star, but she could not think of another reason for being good. And, another problem, the gold stars were not helping her classmate learn to be good.

Not only do grades carry a message about what school is for, as in the first story, but they change motivation for behaviour – not the behaviour, but the reason for the behaviour. A child who behaved virtuously because it was the right thing to do began to believe she should behave that way because of the extrinsic rewards. This conversation has been opened by Alfie Kohn (1994). He has pointed out the dangers of rewards and competitive systems – pointing to research demonstrating children choose the easiest route to an extrinsic reward, but left alone without extrinsic rewards will choose challenging, and more engaging, tasks. Kohn suggests to value work for the joy that comes from it, to value learning for its inherent worth, and I will add that we teachers can value children for the strengths each one brings to the learning situation. These rules, Kohn believes, are more likely to promote an intrinsic love of learning. We should end up with happy confident children, a goal of the Ecoquest program.

There has been much attention paid to assessment in the last little while. There is push for “standards”, and there are more and more large-scale assessments – national and international, as well as provincial. The national and international assessments interplay with the political climate calling for accountability, but as Andy Hargreaves has said – accountability is what we are left with when responsibility fails (2007). If teachers take responsibility for helping their students learn, and if students take responsibility to do the best they can, large-scale assessments can help us learn where we need more work. However, if the climate is about others imposing “accountability” on students and teachers for performance, large-scale assessments are to be feared. They can only point out our failures. Even those schools / teachers / students who perform best will learn there are aspects in which they don’t meet the grade.

The research evidence on assessment and evaluation is clear, and all points in the same direction. Assessment, done correctly, leads to significant improvement in student learning (Black & Wiliam, 1998a). Evaluation, especially high stakes evaluation, has negative effects on student learning. The term “Assessment FOR Learning” has come into vogue. The research on Assessment FOR Learning indicates: significant learning gains are possible for students of all ages and abilities, especially those traditionally considered low achievers (Stiggins & Chappuis, 2005). Gains due to assessment for learning strategies have been reported, with Black and Wiliam (1998a) reporting up to one grade level.

Definitions

Assessment. It has taken me a long time to get my head around this one. Assessment is both the data we collect and the thoughtful weighing of that data. As Alan Ryan (Saskatchewan Education 1991) explained it – when the cook tastes the soup, that is assessment. The cook (I will use the male pronoun, considering that it is Alan tasting his soup) will take a sip of the soup, savouring it as it touches his palette, considering it as it rolls over his tongue, and then will determine if he might want a slightly different blend of spice. He will decide what spices to add – in a process of self-assessment – and will adjust the spices and test the soup again. However, when that soup is offered to guests, then evaluation will occur. The guests will pass judgement, give value to, the soup. Once the guests taste the soup, the cook can no longer adjust that particular soup. This tells us the difference between assessment and evaluation. Assessment comprises the data and feedback loop; evaluation is the assigning of merit or worth to something – be it a thing or a process.

However, good guests might give formative feedback to the cook. They might offer suggestions for the next time the soup is made. Is this formative feedback, assessment or evaluation? It would be assessment, I believe, if there were no assigning of value to the tasted soup. And it would be formative assessment, since the guests are helping the cook to improve.

We can get into discussions of: if I give feedback to another, then I am, of course, assigning value. This is because I am selecting particular data over other data, and therefore deciding some things are more deserving of attention (or merit). Nonetheless, there is a difference between assessment and evaluation, even if there is a grey area between the two.

Recently, the terms Assessment OF Learning and Assessment FOR Learning have been differentiated. Assessment OF learning is when the teacher or an external agency collects the data, savours it, and assigns merit to the work. Period. Finished. (Rarely, however, does Assessment OF Learning stop with mere evaluation. More likely, something will come out of the test results. Students learn how they are performing, and can use the data to work harder, work harder on particular things, or to give up. Teachers can use the data to decide what concepts need more attention. This falls in the grey area between assessment and evaluation. However, with final grades, the grey area is much closer to the evaluation side than the formative-learning side.) Assessment FOR Learning means that the assessment WILL be used to modify the learning. To clarify Assessment FOR Learning, we must first consider the term Classroom Based Assessment which is contrasted with standardized tests or large scale assessments (these latter two being different – but this is not the subject of this talk). Classroom Based Assessment is teacher-designed assessment. The teacher will choose or design assessment materials, and will apply them to the students. In Saskatchewan, teachers do Classroom Based Assessment almost all the time. The teacher might or might not use CBA as Assessment FOR Learning. If the teacher uses the results of assessment to modify his or her teaching, then the teacher is using CBA as Assessment FOR Learning. The teacher might modify teaching immediately, or might modify his or her teaching for next year. As well, the teacher could use the results of standardized or large scale assessments as Assessment FOR Learning, if the teacher modifies his or her teaching because of results. Unfortunately, the results of the test are so slow in coming that modification might not take place for the current group of students.

Classroom Based Assessment is more likely to facilitate the teacher’s modification of teaching immediately, rather than externally designed, and thus slow to return information, tests. SK’s Assessment FOR Learning tests are created for the more distant changes – the teacher will modify teaching for next year’s group of students. Interestingly, SK’s Assessment FOR learning tests are designed with modification of teaching in mind – not with punishing schools, teachers, or students who perform poorly. How these tests are used ultimately, though, will be up to the particular school division. Dr. Brian Noonan is doing research on the necessity of having administrators on side in Assessment FOR Learning.

Formative feedback is a component of Assessment FOR Learning. Formative feedback is not so much for the teaching, however, as for student learning (not that the two can be separated). But formative feedback is immediately relevant to the student and specific for each student. The student gets specific, thoughtful, feedback with advice for improvement.

Formative assessment, according to Black and Wiliam (1998b) is “those activities undertaken by teachers - and by their students in assessing themselves – that provide information to be used as feedback to modify teaching and learning activities… [and] used to adapt the teaching to meet student needs”. Some researchers use the term “formative assessment” (first used by Scriven way back in the ’70’s) and some use the term Assessment FOR Learning. Both terms comprise both the teachers’ modifications of their teaching and the feedback given to the students.

Lastly, there is another term: authentic assessment. Authentic assessment can only be done when authentic tasks are being assessed. The characteristics of authentic tasks are

: when students are expected to “perform, produce, or otherwise demonstrate skills that represent real life learning demands” adding that contrived demands are not placed on the task (Villa, Thousand, Nevin & Liston, 2005. pg. 40).

: “(1) asking students to perform, create, produce, or do something; (2) tapping higher-level thinking and problem-solving skills; (3) using tasks that represent meaningful instructional activities; (4) involving real world applications; and (5) using human judgment to do the scoring” (Corcoran, Dershimer & Tichenor, 2004, pg. 213).

: open-ended, and ill-structured, representing the kinds of problems students will encounter when they are adults outside-of-school context, and as having more than one possible approach and more than one right answer, as well as requiring a student to represent his or her knowledge and skills (DeCastro-Ambrosetti & Cho, 2005).

: “involves students in tasks that are worthwhile, significant, and meaningful” (pg. 499). She added that these activities should encourage students to take academic risks, as well as applying knowledge in new situations. (Surtamm, 2004, pg. 499).

My definition for an authentic task, coming from the literature is: a task assigned to one or a group of students which has some value to the students beyond an assessment hoop. The task involves students making thoughtful and critical decisions about a significant, meaningful, and worthwhile activity. Choices are about what exactly to do, and / or how to do it, and / or how to represent their results. Students have choices which might be about task design or task presentation or assessment. One aspect of authentic tasks: students know they are still learning, even when the task is completed.

Authentic assessment is the appropriate assessment of these out-of-school tasks. How would these tasks be assessed if they were presented to an authentic audience? The criteria for assessment are appropriate for the task – not criteria pre-designed for report cards.

It is no wonder that teachers have difficulty learning more about Assessment FOR Learning. There are so many terms, each with its own flavour to savour!

To summarize: Assessment FOR Learning will lead teachers to modify their teaching, and this might benefit this year’s group of students or next year’s group of students. The results are more likely to be felt immediately if Classroom Based Assessment is used instead of large scale or standardized, externally imposed tests. In SK, our provincially designed tests are about Assessment FOR Learning – for teachers to modify their teaching for next year’s crop of students (although this will depend on the particular school division).

Teachers might provide formative feedback on student assignments. Formative feedback has immediate value for students, especially if they are encouraged (or required) to redo the assignment, using the feedback. There is merit in students just reading over the feedback and using it on the next assignment, though.

Prior Research

If formative feedback is associated with marks, students will spend their time arguing the mark rather than reading and profiting from the feedback. I have met a few students who will argue marks even when they are told they may take the feedback and redo the assignment. The reason I have met so few, I believe, is because I have the privilege of teaching in the College of Education, and our students care about learning.

There has been some research conducted on marks, but it is difficult to learn much about their effect, since students who are not assigned marks often self-select into those programs, and there are so few mark-less programs. Marks have been lauded because they are rewards (incentives) for students to work hard; because they show precisely how one student compares to another; and because students need to get used to being marked for the next level of schooling. In other words, a claim is often made that “marks are just a part of our society and so students need to learn how to adapt.”

Nonetheless, there is one study: students in a university course were divided into three groups. One group got formative feedback, one got formative feedback and grades, and one got grades only. The group that ended up performing the best was the group that got only formative feedback. More studies are required. What do students think of number grades? What about letter grades? What happens to the quality of their work when number and letter grades are removed and they get only formative feedback? Do some students respond better to formative feedback than others? What if high school students have some courses that have no grades and other courses that do have grades – how do they perform in the two kinds of courses?

How is student progress going to be reported to students? To their parents? To their next year’s teachers? What grade levels does a mark-less course work best for? Why do we have grade levels?

Let us further complicate the issue in an examination of what students believe they are being assessed on. There have been many studies indicating that if students believe that learning goals, as opposed to performance goals are being assessed, they are more likely to take risks, and more likely to improve their learning. Learning goals mean that it is the process that counts, as opposed to the product (performance goals). Do students believe they are being marked on their academic work, or are they aware that many teachers are marking them on their attitudes? Trevor Gambell and Darryl Hunter have done research on high school ELA teachers’ grading practices and found that attendance, homework checks, effort, and attitude were commonly assigned a value.

If students believe that they will receive higher grades for “effort”, they are more likely to expend the necessary effort. However, students must be taught what effort is – all too often, effort is seen as putting in time with books in front of one’s face. Not enough. If students have had formative feedback identifying skills they should work on and processes for working on those skills, then their effort shows in the quality of the product; students can and will make the time to access appropriate resources.

An emphasis on learning goals and on effort instead of on product results in students working harder and creating better products. Interesting.

The research on student learning has indicated hands-down that students learn better when assessment contributes directly to their learning. Classroom based assessment allows teachers to more quickly respond to student learning, and formative feedback allows students to correct and improve their work immediately or on the next assignment. If marks are assigned, so far, the research evidence indicates student work deteriorates. Students perform better when learning goals (as opposed to performance goals) are valued, and when they know what effort is and that effort counts in their assessment.

Lastly, students are more likely to give the kind of product the teacher wants when the teacher describes that product to the student. This has led to teachers designing assignment criteria with students, helping students to understand the purpose of the assignment and what the assignment criteria mean. This has enhanced student performance. Interestingly, defining criteria helps teachers see what needs to be taught, as well as helping students learn what needs to be learned.

Assessment FOR Learning has been shown to affect learners in the following ways by:
(having) impacts on engagement, enthusiasm (Hayward, et al., 2004)
(improving) motivation (Shaffner, et al., 2000)
(improving) confidence (Stiggins & Chappuis, 2005)
(improving) intrinsic motivation, and attitudes towards school (Black & Wiliam, 1998b)

These positive effects are possible by making the assessment process more fair, meaningful, accurate, and valid – by involving students in the process. Interestingly, Shaffner el al. (2000) found a correlation between “students’ sense of fairness and their perceived sense of control over their own grades, and teachers’ self-reported level of competence in assessment” (p. 1).

Summary: Marks bad; feedback good. Value the soft things like process and effort over the hard things like product. Take the time to establish assessment criteria with students, and use these assessment criteria as a guide for what needs to be taught. All this can lead to significant – up to two grade levels – improvement in learning.

The bad news is:

Teachers have not been well prepared for assessing nor for evaluating students. It is not that they are any worse that standardized achievement tests, but they could be so much better than these tests. Andy Hargreaves (1989), in a study in England, found that teachers given the right to design their own curricula tended to use fairly traditional curricula. In Saskatchewan, where teachers have the right to assign grades to their students, teachers tend to use very conservative approaches. Hargreaves suggested that for teachers to do something different, they need leaders of some sort – probably from amongst their colleagues, and they need support from research evidence. We are quoting the research evidence to you. Do away with marks (probably), replacing them with clear criteria, emphasizing learning goals, and showing your students with formative feedback what kinds of effort each individual student has to put in.

Teachers cannot be blamed for poor assessment practices, when Colleges of Education are just now introducing courses for pre-service teachers in assessment and evaluation. As well, educational researchers are just now learning about the immense impact that appropriate assessment can have on student learning. As yet, the field is new, resulting in a confusing number of terms, making it not easy to learn about.

The good news is: teachers care that their students learn. Being Saskatchewan teachers, you can apply for support for research on assessment practices from the Dr. Stirling McDowell Foundation and take a few days a year of self designed professional development for conversations about assessment practices.

Assessing the Hard to Assess

We (Shelly, Janet, and two others) applied for Dr. Stirling McDowell Foundation research funds to investigate the attitudes and learning of students in Ecoquest. Ecoquest is an outdoor environmental education program for grade 8 students in Saskatoon Public School Division. The subject areas are completely integrated with one another and in the outdoors.

What is most interesting about the assessment aspect of the program is that Scott Thompson, program initiator, thoroughly researched the outdoor-environmental and integrative nature of the program before the program was began. He and Shelly, in initial discussions together, decided intuitively to do away with number grades. Their assessment practices were based on their many years of teaching experience – on what had become intuitive knowledge to them.

Consequently, the assessment aspect of Ecoquest was an interesting aspect to research. In Ecoquest, students are given clear descriptions early in the year on how to meet expectations for assignments. Students are assessed on learning goals – taking risks for learning – rather than on performance goals. Students are assessed for effort, but they are taught what effort looks like for each individual. For example, for a student who has trouble spelling, effort means getting his or her writing done early enough to have someone else proof-read it. Students are given both formal formative feedback on their assignments, and informal formative feedback as they begin and continue to work on their assignments. Assignments are, often, authentic tasks, such as analyzing an outdoor product and presenting the results to their peers early in the year. Their peers will make decisions on purchases, based on presentations. Students are expected to self-assess on their work. They have the teachers’ assessment in their hands, but not before their eyes, and they read over the comments the teachers have given them. Then students write their own comments and assign themselves a level of Not Yet Proficient, Proficient, and Exceeding Expectations. As well, students set their own goals in “vision and volition” statements, and self-assess their individual progress towards these goals.

The first part of the research was to ask students what they learned in the four dimensions – intellectual (academic), physical, social-emotional, spiritual. The results of the first set of interviews informed the teachers and the program was modified the next on this basis for the next year (Assessment FOR Learning).

Students did not often talk about specific details of what they learned academically – nor was this the purpose of the study. However they talked about the research skills they had learned, and the organizational skills they developed, both of which contributed to their ability to do very well in high school. For physical, they talked about learning the importance of daily physical activity and improving their skills in such things as biking and canoeing. For social-emotional, they talked about getting along with every one else in the class, learning to work with others, and learning about who they were. For spiritual, the interviewees seemed for the most part to have little to say. A few students mentioned getting to know themselves better, and others talked about solo time. That students did not talk about spiritual learning or growth does not mean it didn’t happen. Rather, it shows the word “spiritual” was not used.

For the most part, what students expressed as they talked about their learning, be it academic, physical, or socio-emotional, was change in attitudes. Their dispositions towards others, towards themselves, towards learning, had changed. They cared! They wanted to help others, they wanted to walk softly on Mother Earth, and they knew they were valuable and thus worth speaking up for. They had developed the confidence to say what they believed, to justify their opinions, to listen to the opinions of others. And they cared to take care of their own health – their minds, their bodies, their emotions, and their spirits.

Students were next asked about their assessment. Only two students said they would prefer to have number grades over the three letter assignations they could get (NYP or Not Yet Proficient – do it again, using the feedback given; P or Proficient – meets the criteria; and E or Exceptional or Exceeding Expectations). They liked the three levels, since that was an indication of approximately how well they were doing; as well, though, they liked not having number grades, because this did away with competition. Interestingly, two students mentioned having deliberately “dumbed” themselves down in grade 7 so they wouldn’t stand out. This shows that competition at this age is not necessarily a good thing. Middle years students, although desiring to be individual, generally have a stronger desire to be one with the group; taking away the number grades meant students could work harder. A different two students said they preferred number grades and defended the numbers for different reasons – one preferred the precision of numbers. The other, also believing that grades could be precisely assigned, said he would look at a peer’s work, if that peer did better than him, and find out how he might improve his own work. He implied there was only one right way to do an assignment.

Students believed that no grades and formative feedback went together, that one could not have formative feedback and grades, and that the lack of grades meant they would get formative feedback.

Students all commented on working harder academically in Ecoquest than they had ever worked before. They attributed this, for the most part, to wanting to do their best not to disappoint their teachers. However, most of them had respected their grade 7 teachers. Perhaps the reason they worked harder was because they were asked to do more. As well, with the assessment for learning, the students were given very clear instructions on how they could do better.

Students talked about learning and valuing assessing themselves. They were reminded by their teachers that for most of their lives they would have to self-assess, so they should learn how to do it. This is one reason the students gave for valuing self-assessment. One Ecoqust graduate talked about setting his own goals for high school assignments, rather than doing assignments for the teacher’s purposes. He could then assess how he had matched his own goals. Interestingly, he “earned” good grades on his high school assignments (does it always come down to this?)

This year, students begin self-assessment on their assignments earlier, and their teachers have withdrawn giving the P or E assessments to the students. Students are entitled to check what the teachers have assigned them in the way of P or E, but very few bother. In the first few years of the program, the teachers learned student self-assessment closely matched teacher assessment. Grade 8 is not high stakes time, so students are more likely to be honest. Students are capable of self-assessing when they have the tools in their hands.

Students now in the program are learning more about spirituality – and using the word, so these students would be able to talk about their spiritual development. They write their vision and volition statements, and they follow through on these, self-assessing their progress towards their own goals.

An interesting follow-up study would be to examine student development on their vision and volition goals. What are students’ personal goals and how do these change over the course of the year? How do students self-assess as they progress towards their goals?

Students in the program care about one another, about their learning, about their work, about their communities. They want to make the world a better place. These students have developed a sense of their own importance, and their ability. Some of this sense might come from the ways in which they are assessed. More likely, they are assessed in this manner because their teachers care that the students learn to the best of their ability, and become the best people they can become. In everything the two teachers do, they model they want their students to become happy, confident, and contributing people.

References

Black, P. & Wiliam, D. (1998a). Assessment and classroom learning. Assessment in education, 5(1), pp. 7-74.

Black, P. & Wiliam, D. (1998b). Inside the black box: Raising standards through classroom assessment. Retrieved from http://www.pdkintl.org/kappan/k_v80/k9810bla.htm.

Corcoran, C.A.; Dershimer, E.L. & Tichenor, M.S. (2004). A teacher’s guide to alternative assessment: Taking the first steps. The Clearing House, 77(5), pp. 213-216.

DeCastro-Ambrosetti, D. & Cho, G. (2005). Synergism in learning: A critical reflection of authentic assessment. The High School Journal, 89(1), pp. 57-60.

Hargreaves, A. (1989). Curriculum assessment and reform. Bristol, UK: Open University Press.

Hargreaves, A. (2007) Keynote address, Finding Our Way Conference, Saskatoon, SK.

Hayward, L., Priestley, M., & Young, M. (2004). Ruffling the calm of the ocean floor: merging practice, policy and research in assessment in Scotland. Oxford Review of Education, 30(3), 397-415.

Kohn, A. (1994). The Risks of Rewards. Retrieved from http://ericps.ed.uiuc.edu/eece/pubs/digests/1994/kohn94.html

Saskatchewan Education (1991) Student evaluation: A handbook for teachers. Regina, SK.

Shaffner, M., Burry-Stock, J. A., Cho, G., Boney, T., & Hamilton, G. (2000, April). What do kids think when their teachers grade? Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.

Stiggins, R., & Chappuis, J. (2005). Using student-involved classroom assessment to close achievement gaps. Theory Into Practice, 44(1), 11-18.

Suurtamm, C. (2004). Developing authentic assessment: Case studies of secondary school mathematics teachers’ experiences. Canadian Journal of Science, Mathematics and Technology Education, pp. 497-513.

Villa, R.A.; Thousand, J.S.; Nevin, A. & Liston, A. (2005). Successful inclusive practices in middle and secondary schools. American Secondary Education, 33(3), pg. 33-50.

[TOP]