Principles of Assessment in Medical Education Tejinder Singh, Anshu
INDEX
ABCDEFHIKLMNOPQRSTUVW
360° Assessment (see Multisource feedback)
A
Admission procedures 248–251
Medical College Aptitude Test (MCAT) 249–251
multiple mini-interviews (MMI) 249, 251
National Eligibility-cum-Entrance Test (NEET) 252–255, 257
UK Clinical Aptitude Test (UKCAT) 249–250
AETCOM 69, 174–175, 290
Angoff method (see Standard setting)
Assessment tools
acute care assessment tool (ACAT) 126, 194–195
direct observation-based assessment of clinical skills 114–137
directly observed procedural skills (DOPS) 125–126, 185–186, 189, 225
ethics (see Ethics) 169–171
long case (see long case) 83–90
mini-clinical evaluation exercise (see Mini-clinical evaluation exercise) 117–125, 185–186, 188–189, 225
mini-peer assessment tool (mini-PAT) 128–129, 185–186, 193
multiple choice questions (see Multiple choice questions) 47–64
multiple mini-interview (MMI) 111, 143–147, 249–251
multisource feedback (360° assessment) 127–129, 170–171, 185–186, 192–193
objective structured clinical examination (see Objective Structured Clinical Examination) 91–113
objective structured long examination record (OSLER) 87, 129–130
online assessment (see Online resources for assessment) 300–301, 371–383
oral examination (see Oral examination) 138–148
patient management problem 301–303
portfolios (see Portfolios) 151–164
professionalism (see Assessment of Professionalism) 169–171
selection type questions (See Selection type questions) 39–46
structured oral examination 142–143, 144–146
team assessment of behavior (TAB) 129, 185–186, 193–194
viva voce (see also Oral examination) 138–148
workplace-based assessment (WPBA) (see also Workplace-based assessment tools) 184–196
written assessment (see Written assessment) 32
Assessment
assessment as learning 20, 26–28, 263
assessment for learning (see Assessment for learning) 20, 25, 28, 233–246, 263, 266
assessment of learning 20–25, 234, 263
assessment versus evaluation 2
attributes of good assessment 6
basic concepts 1–17
clinical competence 18–29, 84, 117, 120, 179, 275, 301, 353, 359
community-based (see Community-based assessment) 221–232
competency-based (see Competency-based assessment) 206–220
COLE framework 16
criterion-referenced versus norm-referenced 5
difference between assessment of learning and assessment for learning 234
end of training 288
ethics 165–177
expert judgment 28, 122, 262, 267, 357, 359
for selection (see Admission procedures) 247–260
formative (see Assessment for learning) 236
in-training 288
measurement versus assessment 2
objective 352–353, 357
objective versus subjective 359–361
online (see Online assessment) 296–307
professionalism (see Professionalism) 165–177
programmatic (see Programmatic assessment) 243–244, 261–277
purposes of 3
reducing assessment stress 15
subjective (See Subjective expert judgment) 353, 357
summative assessment limitations 234–235, 279–280
summative versus formative 3–6
test versus tool 3
triangulation of data 267
types 3
utility 15, 262
Vleuten's formula 15, 23–24, 262–263 (see also Utility of assessment)
workplace-based assessment (WPBA) (see Workplace-based assessment) 178–205
written 30–39
Assessment as learning 5, 20, 26, 263, 297
Assessment for learning 20, 25, 28, 233–246, 263, 266
attributes 237–240
cycle 236
effect size of feedback 235
faculty development for 242–245
methods 240–241
strengths 235
SWOT analysis 242
Assessment of learning 20–25, 234, 263
B
Bloom's levels 33
taxonomy 33
MCQ writing 56–58
Blueprinting 24
OSCE 101–102, 117
question paper setting 71–72
C
Checklists versus global ratings 358–359
Clinical competence 18
newble's model 31
COLE framework 16
Community-based assessment 221–232
4R model 223–226
clinical axis 223–224
evidence axis 223–224
personal axis 223–224
social axis 223–224
methods 224–226
direct observation of field skills 225
direct observation of professional skills 225, 230–231
directly observed procedural skills 225
family study assessment 225
logbook 225
mini-CEX 225
multisource feedback 225, 231
objective structured clinical examination 225
observation by community stakeholders 225
portfolios 225, 226
project assessment 225, 229–230
reflective writing 225, 228–229
assessment 228–229
rubrics 228–229
self-assessment of professional skills 225, 227
Paul Worley's framework 223–226
principles 222–223
Community-oriented medical education 221–223
Competency frameworks 19
Competency-based assessment 206–220
design 212–216
prerequisites 210
principles 210–212
Competency-based medical education (CBME) 206, 264
assessment, competency-based 206–220
Competency
core competencies 19
definition 206
dreyfus and dreyfus model 207
frameworks
ACGME competencies 19, 207
CanMEDs competencies 19, 207
five-star doctor 19
General Medical Council competencies 19, 207
indian Medical Graduate (IMG) 19
medical Council of India 19, 207
tomorrow's doctors 19, 207
ideal doctor 19
milestones 208
roles of Indian Medical Graduate 19
sub-competencies 207
Construct 9, 353
construct formulation 353
construct irrelevance variance 10, 353
construct underrepresentation 9, 353
Constructivism 267
Contrasting groups method (see Standard setting) 326–327
Cronbach's alpha (See Reliability)
D
Direct observation-based assessment of clinical skills 114–137
360° team assessment of behavior (TAB) 129
acute care assessment tool (ACAT) 126
directly observed procedural skills (DOPS) 125–126, 185–186, 189, 194
mini clinical evaluation exercise (mini-CEX) 117–125, 188–189, 225, 356
mini peer assessment tool (mini-PAT) 128–129
multisource feedback (360° assessment) 127–129
OSCE 90–113, 116, 131, 299, 356, 359
OSLER 129–130
professionalism mini evaluation exercise (PMEX) 127
tools (see Assessment tools) 115
Directly observed procedural skills (DOPS) 125–126, 185–186, 189, 225
Dreyfus and Dreyfus model 207–210, 213–216
E
Educational environment 239–240
Educational feedback (see Feedback to students)
Educational impact 14, 262
Educational system 236
Entrustable professional activities (EPA) 207, 213–219
designing EPAs 208, 213–217
EPA versus specific learning objectives (SLO) 210
stages of entrustment 213
Ethics
AETCOM 174–175
attributes 166
autonomy 166
beneficence 166
dignity 166
justice 166
non-maleficience 166
difference between professionalism and ethics 166
narratives 173–174
Evaluation 2
Evaluation of teaching (see Student ratings of teaching effectiveness) 342–351
Expert judgment (see Subjective expert judgment) 357, 359–361
F
Faculty development for better assessment 364–370
assessment for learning 242–245
for better assessment 364–370
formal approaches 367
informal approaches 367
model program for training 368–370
objective-structured clinical examination 100–101
transfer-oriented training 368
workplace-based assessment (WPBA) 199
Feasibility of assessment 14
Feedback (see also Feedback, educational; Feedback, from students; Feedback, to students) 25–26, 238–239
Feedback, educational 329–341
attributes 332–334
definition 329–330
descriptive 334–335
feedback loop 330–331
immediate feedback assessment technique 337
issues 338–339
models 335–336
feedback sandwich 335
PCP model 335
Pendleton model 335
reflective model 336
SET-GO model 336
STAR model 336
stop-start-continue model 336
opportunities 336–338
self-monitoring 337–338
strategies for improvement 339–340
types 332–333
benchmarking 332–333
correction 332–333
diagnosis 332–333
longitudinal development 332–333
reinforcement 332–333
Feedback, from students (see Student ratings of teaching effectiveness) 342–351
Feedback, to students (see Feedback, educational) 6, 15, 25, 26, 63, 86, 92, 103, 115, 117, 123, 130, 170, 180, 188, 211, 234, 237, 263, 266, 268, 282, 329–341
Feed forward 25–26
H
Hofstee method (see Standard setting) 327
I
Internal assessment 278–284
1997 MCI regulations 278–279
2019 MCI regulations 279
formative or summative 280–281
issues 286–288
principles 283
quarter model (see Quarter model) 281, 285–295
reliability 281–282
strengths 255–256, 278–284, 286
validity 282
Item analysis 308–318
item statistics 308–313
difficulty index 309–310
discrimination index 309–311
distractor efficiency 309, 311
facility value 309–310
point biserial correlation 311–313
test analysis 313–318
methods of estimating reliability 314–318
equivalent-forms reliability 314
internal consistency reliability 314–315
Cronbach's alpha 315–316
KR 20 formula 315
Kuder Richardson formula 315
split half method 315
standard error of measurement 317–318
parallel-forms reliability 314
test-retest reliability 314
reliability coefficient 284, 313–314
K
Knowledge
assessment of knowledge (see Written assessment) 30–46
free response type questions 30–39
multiple choice questions (see Multiple choice questions) 47–64
selection type questions (see Selection type questions) 39–46
type A (declarative) 31
type B (procedural) 31
Kolb's learning cycle 155, 330–331
Kuder Richardson formula (see Item analysis) 315
L
Logbook 185–187, 225
Long case 83–90
comparison with mini-CEX 131–133
comparison with OSCE 131–133
issues 84–85
OSLER 83, 260
process 83–84
strategies for improvement 85–89
M
Mentoring 26, 153, 244–245, 266, 271
Miller pyramid 20–23, 30–31, 92, 114–115
Mini-clinical evaluation exercise (mini-CEX) 117–125, 185–186, 188–189, 225
comparison with long case, 131–133
comparison with OSCE, 131–133
form 119–120
process 118–121
strengths 122
Mini-peer assessment tool (mini-PAT) 128–129, 185–186, 193
Modified essay questions (MEQ) 31, 35–36
Multiple choice questions (MCQs) 47–64
challenges of using MCQs 48–49
conducting MCQ tests 48
guidelines for writing MCQs 50–55
negative marking 60–61
optical mark reading scanners 59
scoring MCQs 58–60
standard setting 62
strengths of MCQs 48
structure of an MCQ 49
Multiple mini-interview (MMI) 111, 143–147, 249–251
Multisource feedback (360° assessment) 127–129, 170–171, 185–186, 192–193
N
Narratives 173–174
critical incident technique 173
portfolios 173–174
O
Objectification 357
Objective structured clinical examination (OSCE) 91–113
admission OSCE 111
blueprinting 101–102, 117
checklists versus global ratings 105, 116, 358–359
comparison with long case 131–133
comparison with mini-CEX 131–133
computer assisted OSCE (CA-OSCE) 110
examiner training 100–101
factors affecting utility 106–108
feasibility 106–107
group OSCE (GOSCE) 109
key features 93
modifications and innovations 109–111
multiple mini-interview (see Multiple mini-interview) 111
objectivity 107
reliability 108
remote OSCE (ReOSCE) 110
resources to conduct OSCE online 380–381
setup 97–103
simulated patients 100–101
standard setting (See Standard setting) 103–104
team OSCE (TOSCE) 109–110
telemedicine OSCE (TeleOSCE) 110
types of stations 93–97, 116
procedure stations 94, 95–97
question stations 94–95
rest station 97
validity 107–108
Objectivity 2, 108, 212, 282
reliability versus objectivity 352–363
Observable practice activities (OPA) 210
Online assessment 296–307
automation 298
cheating 305–306
consortia 306
designing 297–304
electronic patient management problem 301–303
implementation 304–307
methods 300–301
open-book exams 305
plagiarism 305
question formats 299–300
sharing resources 306
skill labs 306
take-home exams 305–306
triage 306–307
types of questions 300–301
Online resources for assessment 371–383
e-portfolios 379–380
for creating distributing and grading assessment 373
for high stakes examinations 381–382
for online collaboration 379
gamification apps 376–377
interactive tools for formative assessment 373–375
learning management systems 372–373
online security 381–382
proctor devices 381–382
quiz apps 376–377
to conduct online OSCE 380–381
to conduct online simulations 380–381
to create interactive videos 377–378
to create online polls 378–379
to create online surveys 378–379
to create rubrics 381
to enhance student engagement 373–375
Oral examination (viva voce) 138–148
cost-effectiveness 141–142
examiner training 147–148
flaws 139
halo effect 140
objectivity 139–140
reliability 140–141
strategies for improvement 142–148
strengths 139
structured oral examination 142–143, 144–146
validity 141
OSLER (see Direct observation-based assessment of clinical skills) 83, 129–130, 260
P
Patient management problem 301–303
Portfolios 151–164, 185–186
advantages 159–160
challenges 162–163
contents 152–154
definition 151–152
e-portfolios 379–380
for assessment 157–159, 173–174
for learning 152–157, 241
implementation 161–162
limitations 160–161
reflective writing 154–157
workplace-based assessment 185–186
Professionalism 165–177
AETCOM 174–175
altruism 166
assessment methods 169–171
principles 167–169
attributes 166
challenges 167–169
conscientious index 175
definition 165–167
difference between professionalism and ethics 166
multisource feedback 170–171
narratives 173–174
critical incident technique 173
portfolios 173–174
patient assessment 170, 171
peer assessment 170, 241
professional competence 166
professional identity formation 174–175
professionalism mini evaluation exercise (PMEX) 127, 172–173
self-assessment 169–170
supervisor ratings 170, 172
Professionalism mini-evaluation exercise (PMEX) 127, 172–173
Programmatic assessment 11, 26, 243–244, 261–277
CBME 264, 270
challenges 273–276
components 264–270
implementation 271–276
principles 264–267
rationale 262–264
traditional assessment versus programmatic assessment 268–270
triangulation of data 267
Q
Quarter Model 281, 285–295
format 289
implementation 289–293
Question banking 318–321
steps 319
uses 320–321
Question paper setting 65–82
blueprinting 71–72
determining weightage 68–70
item cards 73–76
limitations of conventional practices 66
moderation 77–78
steps for effective question paper setting 67–78
R
Reflections (see Reflective practice)
Reflective practice
for assessment for learning 26, 166, 173, 241, 266
models 155
reflective writing 154–157, 225, 228–229
rubrics 228–229
Reliability 12–13, 262, 354–356
equivalent-forms reliability 314
methods of estimating reliability 313–318
internal consistency reliability 314–315
Cronbach's alpha 315–316
KR 20 formula 315
Kuder Richardson formula 315
split-half method 315
Standard error of measurement 317–318
parallel-forms reliability 314
test-retest reliability 314
reliability coefficient 313–314
versus objectivity 352
S
Selection type questions 39–46
assertion-reason questions 41–42
computer-based objective forms 45
matching questions 43–44
key feature test 44
matching questions 42
multiple choice questions 40
multiple response questions 40
ranking questions 41
true-false questions 40
Self-monitoring 337–338
Self-directed learning 266
Short answer questions (SAQ) (see Written assessments) 31, 36
Simulated patients 100–101
Specific learning objectives 210
Standard error of measurement (see Item analysis) 317–318
Standard setting 24, 322–328
absolute standards 323–324
compensatory standards 324
conjunctive standards 324
criterion-referenced 323–324
effect on learning 324–325
MCQs 62
methods 325–328
for clinical skills 327–328
for knowledge tests 325–327
angoff method 326
contrasting groups method 326–327
hofstee method 327
relative method 325
need 323
norm-referenced 323–324
OSCE 103–104
relative standards 323–324
workplace-based assessment (WPBA) 184
Student ratings of teaching effectiveness 342–351
design of instrument 343–344
Dr Fox effect 347
generalizability 347
interpretation of data 345–346
logistics 344–345
misconceptions 342–343
misuses 342–343, 348
process 343–346
professional melancholia 346
purposes 348
reliability 346–347
validity 346–347
Subjective expert judgment 28, 122, 262, 267, 357, 359–361
T
Triage in medical education 306–307
U
Utility of assessment 15, 262
Vleuten's formula 15, 23–24, 262–263
V
Validity 6–11, 353–354
consequence-related evidence 8, 10, 262
construct-related evidence 8, 9–10, 353
content-related evidence 8
criterion-related evidence 8–9
factors which lower validity 11
Kane's arguments 354
key concepts 10
W
Web resources for assessment (see Online resources for assessment)
Workplace-based assessment (WPBA) 170–171, 178–205
difference from traditional assessment 180
faculty development 199
implementation steps 181–184
need 178–179
prerequisites to implementation 179–181
direct observation 180–181
feedback, 181
practice, 181
tasks 180
problem areas 199–202
quality parameters 197–199
role of assessors 196–197
role of trainee, 197
standard setting 184
strengths 182
tools 184–196
acute care assessment tool (ACAT) 194–195
assessment of performance, 194
case-based discussion (CbD) 185–186, 190–191
clinical encounter cards (CEC) 185–188
directly observed procedural skills (DOPS) 185–186, 189
discussion of correspondence (DOC) 185–186, 192
evaluation of clinical events (ECE) 185–186, 191–192
assessment tool (HAT) 195–196
LEADER case-based discussion (LEADER CbD) 195
logbook, 185–187
mini-clinical evaluation exercise 185–186, 188–189
mini-peer assessment tool (mini-PAT) 185–186, 193
multisource feedback (360° assessment) 185–186, 192–193, 225, 231
patient satisfaction questionnaire, 185–186
portfolio, 185–186, 225–226
procedure based assessment (PbA) 185–186, 189–190
safeguarding case-based discussion 196
sheffield assessment instrument for letters (SAIL) 185–186, 192
supervised learning events 194
team assessment of behaviour (TAB) 185–186, 193–194
weaknesses 182
Written assessment 31–38
closed-ended questions 32
context-poor questions 32
context-rich questions 32
essay questions 31, 34–35
modified essay questions (MEQ) 31, 35–36
multiple choice questions 47–64
open-ended questions 32
short answer questions (SAQ) 31, 36
best response type, 37
completion type, 37
open SAQ, 37–38
structured essay questions (SEQ) 31
×
Chapter Notes

Save Clear