Page 12: of Maritime Reporter Magazine (June 2021)

Training Tips for Ships

Tip #25

Assessment 101

Are Your Exams Valid?

ssessments are critical. But when we create them, to cover the knowledge or skill proportionately well, the num- what strategy do we use? Many of us simply rely ber of questions we ask should correlate with the complex- on intuition: what should our trainees know? Let’s ity of that knowledge / skill. Likewise, the dif? culty of the

Aask it! This may or may not work. But if we just questions should roughly match that which is required for real consider two simple, yet critical measures of assessment qual- world performance. And in scoring the exam, we should con- ity, we will produce better assessments, which means we will sider the weight of each question or section. Questions cover- have more accurate information about whether our of? cers ing competencies which reveal more about likely job perfor- and crew have the knowledge and skills required to perform mance should be weighted more heavily than those which are safely and effectively. What are these two core attributes of a less revealing. Questions covering critical knowledge or skills high-quality assessment? Validity and reliability. should be weighted such that the exam cannot be passed with-

When creating an exam, always have these two words in out acceptable performance on those - otherwise we risk hid- mind: Valid and Reliable. If we think of little else other than ing safety or performance risks. In the end, for a valid exam, validity and reliability when creating exams, and if we man- ensure that each and every question is there because it tells us age to get those right, we will likely be producing excellent something important about job performance (or whatever the assessments. Let’s look at validity ? rst in this edition of Train- goal), that it is presented at a dif? culty level that correlates ing Tips for Ships. Then in next month’s edition, we will look with the dif? culty of the task, and that it is weighted to be at reliability. correlated with its contribution to the performance of the task

A valid exam is one that tests what we actually want to know it supports and how critical that task is to performance and about the trainee. So, if we are writing a test in the hope that it safety. If you do these, you will be well on your way to writ- will reveal whether an employee will successfully ful? l a cer- ing a valid exam. tain role in the organization, then the result he or she achieves As a concluding thought, always keep in mind that except in on a valid test will correlate well with their job performance. the simplest of circumstances, exams can never provide full

That sounds simple, but there is some nuance to it. coverage of all knowledge, or the entire skill set identi? ed as

The validity of assessments naturally breaks down into two being important. Exams are never comprehensive, but instead parts. First, we must know what knowledge and skills are im- are more like auditing processes designed to reveal problems portant to the job at hand. Interestingly, we can determine this that require further evaluation. Thus, in choosing questions accurately by mining results of past exams and correlating the based on the important criteria of validity, avoid the temp- performance of individual questions to job performance as- tation to ask “everything” and instead focus on those valid sessments. This is the ? eld of learning analytics which was questions which will identify any areas requiring a deeper in- discussed in past editions of Training Tips for Ships. If that vestigation.

kind of analysis is beyond the current reach of your organiza- Next time, we will look at the second core attribute of a tion, then the best approach (until we get there) is to rely on high-quality exam: reliability. Until then, be healthy and sail our expert trainers. They will have good, intuitive ideas of safely.

what mariners need to know in order to perform well. And if

The Author the course is well designed to support job performance, then the learning objectives of the course will be an excellent re-

Goldberg source and all exam questions should be aligned with those

Murray Goldberg is CEO of Marine

Learning Systems which provides learning objectives.

software and services to optimize

The second part of achieving exam validity is to take the list knowledge, skills and behavior in of required knowledge and skills identi? ed above, and then maritime operators. formulate questions which appropriately test that knowledge

[email protected] and those skills. There are several parts to this. First, in order 12 Maritime Reporter & Engineering News • June 2021

MR #6 (1-17).indd 12 6/3/2021 2:10:14 PM