Assessment: The BasicsChapter 1

In this chapter, we will introduce the basic concepts related to assessment of students and how we can maximize the effect of assessment in giving a desired shape to learning.

While such an approach may have been appropriate to measure physical properties (such as weight, length, temperature, etc.), it certainly did not capture the essence of educational attainment. There are several qualities which we want our students to develop, but which are not amenable to precise measurement. Can you think of some of these? You may have rightly thought of communication, ethics, professionalism, etc., which are as important as other skills and competencies, but which cannot be precisely measured.

Let us also clarify some more terms that are often loosely used in the context of student assessment. “Test” and “tool” are two such 3terms. Conventionally, a “test” generally refers to a written instrument which is used to assess learning. Test can be paper/pencil-based or computer-based. On the other hand, a “tool” refers to an instrument used to observe skills or behavior to assess the extent of learning. Objective Structured Clinical Examination (OSCE) and mini-Clinical Evaluation Exercise (m-CEX) are examples of assessment tools.

Assessment in medical education is especially important because we are certifying students as fit to deal with human lives. The actions of doctors have the potential to make a difference between life and death. This makes it even more important to use the most appropriate tools to assess their learning. You will also appreciate that medical students are required to learn a number of practical skills, many of which can be lifesaving. Assessment is also a means to ensure that all students learn these skills.

Formative assessment should not be used for final certification. This implies that certain assessment opportunities must be designated as formative only, so that teachers have an opportunity to identify the deficiencies of the students and undertake remedial action. A corollary of this statement is that all assignments need not be graded, or that, all grades need not be considered during the calculation of final scores. From this perspective, all assessments are de facto summative; they become formative only when they are used to provide feedback to the students to make learning better. Formative assessment has been discussed in more detail in Chapter 16.

Summative assessment intends to test if the students have attained the objectives laid down for a specified unit of activity. It is also used for certification and registration purposes (e.g., giving a license to practice medicine). Did you notice that we said “attainment of listed objectives”? This implies that students must be informed well in advance, right at the beginning of the course, about what is expected from them when they complete the course, so that they can plan the path of their learning accordingly. Most institutions ignore this part, leaving it for students to make their own interpretations based on inputs from various sources (mainly from senior students). No wonder then, that we often end up frustrated with the way the students learn.

The contemporary trend is towards blurring the boundary between formative and summative assessment. Purely formative assessment without any consequences will not be taken seriously by anybody. On 5the other hand, purely summative assessment has no learning value or opportunity for improvement. There is no reason why the same assessment cannot be used to provide feedback, as well as to calculate final scores. We will discuss this aspect in more detail in Chapter 18.

We strongly believe that every teacher can play a significant role in improving student learning by the judicious use of assessment for learning. Every teacher may not be involved with setting high-stake question papers, but every teacher is involved with developing assessment locally to provide feedback to the students. Throughout this book, you will find a tilt toward the formative function of assessment.

Sometimes, assessment itself can be used as a learning task, in which case, it is called assessment as learning.

Results of CRT can only be a pass or a fail. Let us have an example. If the objective is that the student should be able to perform a cardio-pulmonary resuscitation, then he must perform all the essential steps to be declared as pass. The student cannot pass if he performs only 60% of the steps! CRT requires establishment of an absolute standard before starting the examination.

Again, there can be variations to this, and one of the commonly employed means is a two-stage approach, i.e. first use CRT to decide who should pass and then use NRT to rank order them. Traditionally in India we have been following this mixed approach. However, we do not seem to have established defensible standards of performance so far and often arbitrarily take 50% as the cut-off for pass/fail. This affects the validity of assessment. It is important to have defensible standards. Standard setting has been discussed in Chapter 23.6

There are several attributes that good assessment should possess. Rather than going into the plethora of attributes available in literature, we will restrict ourselves to the five most important attributes of good assessment as listed by Vleuten & Schuwirth (2005). These include:

The traditional view was that validity is of various types: content validity, criterion validity (this was further divided into predictive validity & concurrent validity), and construct validity (Crossley, Humphris & Jolly, 2002) (Fig. 1.1). This concept had the drawback of seeing assessment as being valid in one situation but not in another. With this approach, a test could cover all areas and have good content validity; but may not be valid when it comes to predicting future performance. Let us draw a parallel between validity and honesty as an attribute.

Validity is now seen as a unitary concept, which must be inferred from various evidences (Fig. 1.2). Let us come back to the “honesty” example. When would you say that someone is honest? One would have to look at a person's behavior at work, at home, in a situation when he finds something expensive lying by the roadside, or how he pays his taxes, and only then make an inference about his honesty. Validity is a matter of inference, based on available evidence.

Validity refers to the interpretations that we make out of assessment data. Implied within this is the fact that validity does not refer to the tool or results—rather, it refers to the interpretations we make from the results obtained by use of that tool. From this viewpoint, it is pertinent to remember that no test or tool is inherently valid or invalid.

Let us explain this further. Suppose we use a 200-question MCQs test to select the best students to get into postgraduate medical courses. We could interpret that the highest scorers of the test have the best content knowledge. To do this, we would have to gather evidence to check if all relevant portions of the syllabus had received adequate representation in the paper. We could also state that the students with the best aptitude have been selected for the course. For this we will need to present evidence that the test was designed to assess aptitude also. As you can see, validity is contextual. So here, it is not the tool (MCQ), which is valid or invalid, but the interpretations that we infer from the results of our assessment which matter.

Inferring validity requires empirical evidence. What are the different kinds of evidence that we can gather to determine if the interpretations we are making are appropriate and meaningful?

As Figure 1.2 shows, we need to gather evidence from different sources to support or refute the interpretations that we make from our assessment results. Depending on the situation, we might look for one or two types of evidence to interpret validity of an assessment. 8But ideally, we would need to look for evidence in the following four categories (Waugh & Gronlund, 2012):

To do this, one has to be fully aware of: why we are performing a particular assessment; the exact nature of construct being assessed; what we are expecting to obtain by conducting this exercise; what the assessment results are going to be used for; the exact criterion which are going to be used to make decisions on assessment results; and the consequences of using this assessment. Let's understand this in more detail.

Thus, sampling is a key issue in assessment. Always ask yourself if the test is representative of the whole domain that you are assessing. For this, look at the learning outcomes, prepare a plan (blueprint) and prepare items which correspond to these specifications. More on this will be dealt with in Chapter 6.

Now suppose we are introducing a new assessment method A and we want to see how it works in comparison to an existing assessment method B for the same purpose and in the same setting. To do this we can compare the results obtained from both tools in the same setting to see how effective method A is in comparison to the previous method B. Here we are concurrently judging the results of two methods to see if they are comparable. This is the concept of concurrent validity.

If we were to consider the construct ‘beauty’, we might use attributes such as physique, complexion, poise, confidence, and many such attributes to decide if one is beautiful. Similarly, in educational settings, subject knowledge, its application, data gathering, interpretation of data and many other things go into deciding the construct ‘clinical competence’. In medicine, educational attainment, intelligence, aptitude, problem-solving, professionalism, and ethics are some other examples of constructs.

All assessment in education aims at assessing a construct. It is the theoretical framework which specifies the hypothetical qualities that we seek to measure. For instance, we are not interested in knowing if students can enumerate five causes of hepatomegaly. But we are interested in knowing, if they can take a relevant history based on those causes. In this context, construct-related evidence becomes most important way to infer validity. Simply stated, results of assessment will be more valid, if they told us about the problem-solving ability of a student, rather than about his ability to list five causes of each of the symptoms shown by the patient. As a corollary, it can also be said that if the construct is not fully represented (e.g., testing only presentation skills, but not physical examination skills during a case presentation), validity is threatened. Messick (1989) calls this construct under-representation (CU).

While content and construct seem to be directly related to the course, the way a test is conducted can also influence its validity. A question may be included in an examination to test understanding of certain concepts, but if the papers are marked based on a student's handwriting, validity is threatened. If a test is conducted in a hot, humid, and noisy room, its validity becomes low, because then, one is 10also implicitly testing candidates’ ability to concentrate in the presence of distractions rather than their educational attainment. Notice here, that the construct that we were assessing has changed. Suppose an MCQ is framed in complicated language and if the students must spend more time in understanding the complex language of an MCQ, rather than on its content, then validity is threatened. Here besides content, one is testing vocabulary and reading comprehension. Similarly, leaked question papers, incorrect key, equipment failure, etc. can have a bearing on the validity. Messick (1989) calls this construct irrelevance variance (CIV).

Let us try to explain this concept in a different way. Let us say you conduct an essay type test and try to assess knowledge, skills, and professionalism from the same. We would expect that there would be low correlation between the scores on the three domains. On the other hand, if we conduct three different tests, say for example, essays, MCQs, and oral examination to assess knowledge, we would expect a high correlation between scores. If we were to get just the opposite results—i.e., high correlation in the first setting and low in the second, construct irrelevance variance would be said to exist. You can think of many common examples from your own settings, which induce CIV in our assessment. Too difficult or too complicated questions, use of complex language which is not understood by students, words which confuse the students and “teaching to the test” are some of the factors which will induce CIV. Designing OSCE stations which test only analytical skills will result in invalid interpretation about practical skills of a student by inducing CIV.

The contemporary concept of validity is that all validity is construct validity (Downing, 2003). It is the most important of all the evidences that we gather to determine validity.

These main concepts on validity have been summarized in Box 1.1.

We have for long, followed the dictum of “assessment drives learning,” which often results in extraneous factors distorting learning. A better way would be to let “learning drive assessment” so that validity is built into assessments. This concept of Programmatic Assessment has been discussed in detail in Chapter 18.

Similarly, the assessment tools should aim to test broad constructs rather than individual competencies like knowledge or skills. It is often better to use multiple tools to get different pieces of information on which a judgment of student attainment can be made. It is also important to select tools, which can test more than one competency at a time. There is no use of having one OSCE station to test history taking, another for skills, and yet another for professionalism. Each station should be able to test more than one competency. This not only provides an opportunity for wider sampling by having more competencies tested at each station but also builds validity.12

A commonly used definition of reliability is obtaining same results under similar conditions. The concept of obtaining same results under similar conditions might be true of a biochemical test. However, it is not completely true of an educational test. Let us say, during the final professional MBBS examination, we allot a long case to a student in a very conducive setting, where there is no noise or urgency, and the patient is very cooperative. But we know that in actual practice, this seldom happens. Similarly, no two patients with same diagnosis will have similar presentation. In the past, educationists have tried to make the examinations more and more controlled and standardized (e.g., OSCE and standardized patients), so that the results represent only student attainment and nothing else. We argue that it might be better to work in reverse— i.e., conduct examinations in settings as close to actual ones as possible so that reproducibility can be ensured. This is the concept of workplace based and authentic assessment.

We often tend to confuse between the terms ‘objectivity’ and ‘reliability’. Objectivity refers to reproducibility of the scores so that anyone marking the test would mark it the same way. There are certain problems in equating reliability with objectivity in this way. For example, if the key to an item is wrongly marked in a test, everyone would mark the test similarly and generate identical scores. But are we happy with this situation? No, because it leads to faulty interpretation of the scores. Let us add some more examples. Suppose at the end of final professional MBBS, we were to give the students a test paper containing only 10 MCQs. The results will be very objective, but they will not be a reliable measure of students’ knowledge. There is no doubt that objectivity is a useful attribute of any tool, but it is more important to have items (or questions) which are fairly representative of the universe of items which are possible in a subject area, and at the same time sufficient number of items so that the results are generalizable. In other words, in addition to objectivity we also need an appropriate and adequate sample to get reliable results. This example also shows how reliability evidence contributes to validity.

We will also like to argue that objectivity is not sine-qua-non of reliability. A subjective assessment can be very reliable if based on adequate content expertise. We all make predictions— subjective— about potential of our students and we rarely go wrong! The point that we are trying to make is that in educational testing there is always a degree of prediction involved. Will the student whom we have certified as being able to handle a case of mitral stenosis in the medical college be able to do so in practice? To us, reliability is 13therefore the degree of confidence that we can place in our results (try reading reliability as rely-ability).

A common reason for low reliability is the content specificity of the case. Many examiners will prefer to have a neurological case in the final examination in medicine. It is presumed that a student who can satisfactorily present this case can also present a patient with anemia or malnutrition. This could not be farther from the truth. Herein lies the importance of including a variety of cases in the examination to make them representative of what the student is going to see in real life. You will recall what we said earlier that a representative and adequate sampling is also important to build validity.

Viewing reliability of educational assessment differently from that of other tests has important implications. Let us suppose that we give a test of clinical skills to a final year student. If we look at reliability merely as reproducibility— or in other words, getting same results if the same case is given again to the student under same conditions— then we will try to focus on precision of scores. However, if we conceptualize reliability as confidence in our interpretation, then we will like to examine the student under different conditions (outpatients, inpatients, emergency, community settings, etc., and by different examiners) and on different patients so that we can generalize our results. We might even like to add feedback from peers, patients, and other teachers to infer about the competence of the student.

We often go by the idea that examiner variability can induce a lot of unreliability in the results. To some extent this may be true. While examiner training is one solution, it is equally useful to have multiple examiners. We have already discussed about need to include a variety of content in assessment. It may not be possible to use many assessment formats at one occasion, but this can happen when we carry out assessment on multiple occasions. The general agreement in educational assessment is that a single assessment, howsoever perfect, is flawed for making educational decisions. Therefore, it is important to collect information on several occasions using a variety of tools. The key dictum to build reliability (and thereby validity) for any assessment is to have multiple tests on multiple content areas by multiple examiners using multiple tools in multiple settings. The concept of Programmatic Assessment discussed in Chapter 18 largely follows this approach.

Validity and reliability of a test are very intricately related. To be valid, a test should be reliable. Reliability evidence contributes to validity. A judge cannot form a valid inference if the witness who is being examined is unreliable. Thus, reliability is a precondition for validity. But let us caution you that it is not the only condition. Please also be aware that generally there is a trade-off between validity and reliability: the stronger the bases for validity, the weaker the bases for reliability (and vice-versa) (Fendler, 2016).

An application-oriented perspective on validity and reliability of assessments has been discussed in Chapter 26.14

Portfolios, for example, can provide as much information as can be provided by rating scales. MCQs can provide as much information about knowledge as can be obtained by oral examinations. However, acceptability by students, raters, institutions and society, at large, can play a significant role in accepting or rejecting a tool. MCQs, despite all their problems, are accepted as a tool for selecting students for postgraduate courses, while methods like portfolios, which provide more valid and reliable results may not be. This is not to suggest that we should sacrifice good tools based on likes or dislikes, but to suggest that all stakeholders need to be involved in the decision-making process about use of assessment tools.

Linked to the concept of acceptability is also the issue of feasibility. While we may have developed very good tools for assessing communication skills of our students, resource crunch may not allow us to use this tool on a large scale.

Assessments do not happen in vacuum. They happen within the context of certain objectives. For each assessment, there is an expected use—it could be making pass/fail judgments, selecting students for an award or simply to provide feedback to teachers. Asking these three questions visually brings a lot of clarity in the process and helps in selecting appropriate tools.

This is not a mathematical formula but a notional one. This concept is especially important because it shows us how to compensate for deficiencies in assessment tools by their strengths. Results of some tools may be low on reliability but can still be useful if they are high on their educational impact. For example, results of MCQs have a high reliability, but little educational value. Results of mini-clinical evaluation exercise (mini-CEX), on the other hand, may be low on reliability, but have a higher educational value due to the feedback component. Still, both are equally useful to assess students. Similarly, if certain assessment has a negative value for any of the parameter, (e.g., if an assessment promotes unsound learning habits), then its utility may be zero or even negative.

The above five criteria contributing to utility of assessment were accepted by consensus in 2010 as the criteria for good assessment along with two additional criteria (Norcini et al., 2011). While we have retained the earlier nomenclature of “criteria,” there have been some modifications to it. Later at the 2018 Ottawa consensus meeting, the nomenclature was changed from “criteria” to “framework” for good assessment emphasizing the essential structure that these elements provide (Norcini et al., 2018). The alternative nomenclature of these seven elements was provided as: (1) Validity or coherence; (2) Reproducibility, Reliability, or Consistency; (3) Equivalence (the same assessment yields equivalent scores or decisions when administered across different institutions or cycles of testing); (4) Feasibility; (5) Educational Effect; (6) Catalytic effect (the assessment provides results and feedback in a fashion that motivates all stakeholders to create, enhance, and support education; it drives future learning forward and improves overall program quality); (7) Acceptability (Norcini et al., 2018). The same paper well summarizes the relationship between these elements of framework and purpose of assessment (formative or summative) rather well. Validity is essential for both the formative and summative purposes. While reliability and equivalence are more important for the summative assessments, the educational and catalytic effects are key to formative use. Feasibility and acceptability considerations are a must for both. Whatever nomenclature we may adopt, assessment can never be viewed in terms of a single criterion, framework or attribute.16

The other approach is to reduce stakes on individual assessments and take a collective decision based on multiple low stake assessments, spread throughout the course. This is the basis of programmatic assessment and will be discussed in Chapter 18.

As we go through the subsequent chapters, we will be discussing about assessment methods and assessment design in greater detail.