Assessment conundrums

I am teaching a course for public school English teachers in Brasilia and one of the topics addressed is assessment. The aim of this part of the course is to improve teachers’ assessment literacy, allowing them to provide informed feedback on the assessment system used in their institution and develop assessment systems and tools that are in keeping with the most current assessment practices.

While going over the ELT assessment literature and discussing topics such as reliability, validity, washback, practicality, formative versus summative assessment, formal versus informal assessment, and alternative (or authentic) versus traditional assessment, we noticed some common misconceptions. The focus of this post, then, is to discuss these issues that seemed problematic to this group of teachers and that may be problematic to many others.

 

1- Validity and reliability

It is sometimes easy to confuse these terms. Validity addresses whether the assessment instrument accurately measures the construct it is intended to measure (e.g. communicative competence) and the content it was designed to assess, in a congruent way with how this content is addressed in the classroom. A composition test in a course that doesn’t teach composition skills is not a valid instrument.

I like to use the term alignment rather than validity when referring to classroom assessments. An assessment tool needs to be aligned with the learning objectives established for the course and the instructional strategies used in the classroom. If students practiced verb tenses, for example, by way of fill-in-the-blanks exercises only, the teacher can’t expect them to complete a dialogue with questions, using different verb tenses. The assessment items have to be in keeping with the instructional strategies.

Reliability has to do with consistency of students’ scores. In a program that uses the same test for all students, a reliable instrument is one in which the students would obtain the same result, no matter the teacher, the classroom, or the day of the week and time. If one teacher gives students 30 minutes to take the test, and the one next door gives them one hour, this affects the reliability of the instrument.

Reliability and validity need to be taken very seriously in standardized, high-stakes tests. Classroom assessments also need to be valid and reliable, but in a different way. For example, it would be impossible to reach the same level of interrater reliability for performance assessments in the classroom as an international proficiency exam of speaking or writing needs to have. However, some steps can be taken to have at least minimum reliability, such as the development of assessment rubrics and, if possible, the training of teachers to use the rubrics by way of calibration sessions.

Classroom assessments are usually achievement assessments, so a single instrument will probably not encompass all the elements in a communicative competence model like a proficiency test. It might not even encompass the four skills. It is important, though, that the assessment system include the four skills, with perhaps different instruments for each skill or a combination of two skills.

 

2- Formative or summative versus formal or informal assessment

It is common for teachers to think that all formative types of assessment are informal and the summative types are formal. Informal assessment is the assessment we conduct every day in our classrooms when we want to confirm if students have learned what we are teaching. We engage in informal assessment of our students on a daily basis, whether we are aware of it or not. Formative assessment is assessment that is done during the learning process and that is used to promote learning, rather than merely measure it. There is indeed an intersection between informal and formative assessment, but this doesn’t mean that formative assessment can only be informal. A piece of writing done in multiple drafts and assessed by way of rubrics, a project carried out in various stages, or even a graded exercise that students have the chance to retake are examples of formal yet formative types of assessment.

 

3- Performance or traditional versus formative or summative assessment

Traditional assessments usually contain selected-response items (e.g. multiple choice, matching, T or F, etc.) and limited constructed responses. Performance assessments are assessments of student production in real-life situations, such as an oral presentation, a piece of writing, or a project. While it is true that performance assessment lends itself more easily to formative uses, it doesn’t mean that all performance assessments are necessarily formative. For example, if a teacher has students do an oral presentation at the end of the course and grades students’ performance without giving them a chance for improvement, this is a summative type of assessment, though it is not traditional. On the other hand, if the teacher gives a multiple-choice quiz, readdresses the topics that students demonstrated difficulties in, and then reassesses them, this is an example of a traditional tool used formatively.

 

4- Performance assessment and rubrics

While many of the teachers I’m working with use performance types of assessment in their schools, they do not have a clear set of criteria to assess student work, so they assign grades based on their own, “idiosyncratic” criteria. In other words, they do not use an instrument that specifies what will be assessed and the level of performance expected. This is problematic in terms of reliability, for different teachers can value different aspects of performance, such as accuracy, fluency, or content.  The same student performance, thus, can be assessed very differently by different teachers if the same performance descriptors are not used.

 

5- Washback and feedback

Teachers also tend to confuse washback and feedback. Feedback is what we should obtain after we have administered an assessment tool. This feedback can come from students, such as how they felt about the assessment, if it was fair, if it measured the intended content, if the testing conditions were appropriate, etc. We can also obtain feedback from the test itself, by way of test analysis. Conversely, washback is related to how the assessment affects teaching. This is not something that can be easily seen with a single assessment at a specific time, but rather, it is the result of a whole assessment policy. For example, if teachers know that their students will be assessed by way of oral activities, they will be much more likely to focus on oral activities in their classroom than teachers whose students will be assessed only by way of written, multiple-choice tests. If the assessment criteria for these oral activities place much more emphasis on fluency than on accuracy, this will also affect how teachers deal with fluency versus accuracy in their classroom.

These were the issues brought up in our class discussions, based on the assigned readings for the course. Are there other assessment issues that you think are confusing, problematic or controversial?

 

Isabela Villas Boas

Isabela Villas Boas holds a Master's Degree in Teaching English as a Second Language from Arizona State University and a Ph.D. in Education from Universidade de Brasília. She has been at Casa Thomas Jefferson for 33 years, where she is currently the Corporate Academic Manager . Her main academic interests are second language writing, teacher development, ELT methodology, and assessment. She also supervises MA dissertations for the University of Birmingham. She has recently published the book “Teaching EFL Writing - A Practical Approach for Skills-Integrated Contexts.

17 Comments
  • Izabel
    Posted at 11:39h, 10 junho Responder

    Hi, teacher!
    Thank you very much for posting another explanation on the topics we`ve been studying!! When I think we have resources enough to support us, you come up with an even better résumé. This will help us a lot. I simply love taking this course and I`d like it to last `till the end of the year. “I wish I were you, when I grow up”. Thank you again and again and again…

    • Isabela Villas Boas
      Isabela Villas Boas
      Posted at 19:39h, 14 junho Responder

      Thank YOU, Izabel, for inspiring me to write these types of posts. We have been having great discussions in class and I also wish we could go on and on until the end of the year!

  • Jane Rocha
    Posted at 11:53h, 10 junho Responder

    Dear Isabela,

    Thank you very much for sharing such a great article about assessment concepts. I really benefitted from it by the way you defined them. It’s getting clearer in my head now! I also enjoyed reading it, as it helped me improve my vocabulary in English with the words such as conundrums, validity and reliability, performance descriptors, internal criterial, washback and feedback. And yes, dear teacher, I was one of your students who got confused with the concepts of “washback” and “feedback”. Thank you for posting this great help for us. May you be very successful and happy in your personal and professional projects. Have a great day you all out there.

    • Isabela Villas Boas
      Isabela Villas Boas
      Posted at 19:38h, 14 junho Responder

      Thank you, Jane. I was inspired by our fruitful conversations in class!

  • Denise Bussinguer
    Posted at 16:46h, 10 junho Responder

    Commenting your final question, “Are there other assessment issues that you think are confusing, problematic or controversial?”

    Well, I find it hard to correct compositions. Particularly, I like to sit down with the students and comment on their writings so they have the opportunity to improve it. Students are not used to write anymore, most of them. When they do it, they are afraid of making mistakes, they want to present a “perfect job” in terms of vocabulary and spelling, not necessarily a plausible communicative tool in the format of a writing composition. Students tend to Google translate what they have to write for their ENGLISH class, so we as teachers, have to make them aware of the need of THINKING in ENGLISH for improving fluency and writing itself. How should we do it? What’s the best approach to make the writing process and opportunity to improve fluency and thinking in English?
    I once studied, reflected upon it, and agreed with it that when students write, that’s when they have a chance to elaborate their “speech” , have time to reseach, check spelling and meanings on the dictionary, and use the thesaurus, for example. They also have the chance to revisit their writings. If we use the WRITING PROCESS as an opportunity to improve writing and make students involved and aware of taking the necessary STEPS in the writing process, they will be likely more aware of it and the chances for a better piece of writing will be greater. However, it takes time, doing it during a semester only. Peer editing and revising with the teacher before editing the final draft takes a long time, considering the number of students we have in each class. I mostly do it with the advanced ones, but there is not enough time to form the habit of writing; they have only 3 semesters at our public language school for that and I CAN, in fact, do it only once during the semester, and it happens after classes, during my planning. Ideally, it would be nice to have a writing course especifically designed for it. I have seen that practice makes it. So, we need time and lots of opportunities to make writing a habit. Writing improves fluency, I believe. What do you think?

    • SILVIO OLIMPIO BORGES DE ARAUJO
      Posted at 19:32h, 10 junho Responder

      June 10, 2015

      Dear Isabela,

      Thank you very much for posting such an interesting article about assessment concepts. It has improved a lot my vocabulary in the English language with the words such as conundruns, validity, performance descriptors, internal criteria, washback , feedback, etc.
      Best Regards. See you on Friday!

      • SILVIO OLIMPIO BORGES DE ARAUJO
        Posted at 19:36h, 10 junho Responder

        * conundrums

      • Isabela Villas Boas
        Isabela Villas Boas
        Posted at 19:35h, 14 junho Responder

        Glad you liked it, Silvio. You guys inspire me!

    • Isabela Villas Boas
      Isabela Villas Boas
      Posted at 19:38h, 14 junho Responder

      Hi, Denise. Teaching and assessing writing are really challenging. Here is the link to an article I wrote a while ago that I believe might help you, or rather, reiterate what you are already doing! https://www.academia.edu/8229004/Stepping_Stones_for_Successful_Writing

      • Denise Bussinguer
        Posted at 10:34h, 19 junho Responder

        Thanks, Isabela. I have just finished reading your article and it is great, a very comprehensive one! I am glad I am on the way and feel motivated to keep doing and improving what my students are doing. When we devote time to the writing process, especially after reading the genre type that will provoke critical thinking to generate writing within the same genre, we can gather greater fruits from our students responses. That’s what I have experienced. Instead of bored and unmotivated, they feel like writing! What I have seen is that once the students are not familiar with the writing process used in the American education system, for example, they have no clues where to start from, unless they are excellent writers in their own language, resulting in transference of knowledge from the mother tongue to the target one. However, because the writing process is so explicit for native speakers in North America, starting at a very young age, even our good Brazilian writers will benefit from its format and process. Also, if we read authentic materials with our students, by scaffolding it appropriately and using reading strategies to promote vocabulary acquisition and use the “same” or “similar” writer’s quality style to showcase our students, we will be generating higher order thinking skills in our classes. Eventually, students will feel confident to take risks to try writing in a positive way. That’s why I agree with you and researchers that writing is an essential skill to be taught and it should, if not must, be a follow up activity after reading (or listening) a text. Research has also shown that at the end, by learning a second language well, it will end up interfering positively and improving the mother tongue itself. That’s why I think it’s a path to be followed, due to its dual benefit to the foreign and the very own “native tongue”.
        I am so thankful I had the opportunity to teach ESL and apply many of the steps you mentioned in your article. I am still adjusting it to make it now benefit my EFL students. Thank you for your valuable contribution to the teaching & learning of a foreign language!

  • Valéria Ribeiro
    Posted at 19:47h, 10 junho Responder

    Isabella,
    As we may expect from you, the way you approached the subjects we have been discussing is brilliant!

    • Isabela Villas Boas
      Isabela Villas Boas
      Posted at 19:35h, 14 junho Responder

      Thank you, Valeria. I’m glad it was useful to you.

  • Eleuza Resende
    Posted at 11:07h, 12 junho Responder

    Dear Isabela,
    Thank you for this very interesting and useful article. I am going to use it on my talk in my school, as a support to clarify teachers doubts.

    • Isabela Villas Boas
      Isabela Villas Boas
      Posted at 19:34h, 14 junho Responder

      I’m glad you like it and that it will be useful to you, Eleuza!

  • Rosemary Pimentel Oliveira
    Posted at 17:05h, 15 junho Responder

    I´m so proud of being part of such a fabulous course. This article about the 7 key concepts was perfect. Sometimes I felt it was so difficult go on until the end of the course. I´m so glad that I haven´t given up. I´m considering the idea of continuing everything I can do and not being satisfied only with my pos- graduation. I only want to thank all of you!!

  • Nadja
    Posted at 07:22h, 17 junho Responder

    Teacher, you´re great!
    Thank you very much for this clear and objective text, It helped me a lot. Now, I know that washback is not a feedback and everything became so clear.
    Nadja

  • LILIAN MARA AMANCIO BATISTA
    Posted at 21:56h, 17 outubro Responder

    Hi everybody!

    Understanding the difference between the use of performance in a traditional way of evaluating and performance in a formative perspective is an important step for those who are looking for means to improve their practices in the teaching and learning process. Now, it is clear to me that it is not necessary to get rid of the traditional tools. They can be used effectively if we give the students the opportunities to be reassessed after being aware of their strengths and weaknesses. Therefore, assessing students formatively is not as dificult as it sounds.

Post A Comment