assessing skills at scale without fooling yourself

Somewhere in most large organizations there is a skills assessment that was written over a weekend by a single subject matter expert, contains 40 questions, and is used to make decisions about several thousand people. It produces a number. The number gets treated as a measurement, and decisions follow.

Assessment at scale is not hard because the technology is hard. It is hard because two concepts, validity and reliability, are almost always used loosely, and loose use produces confident decisions built on noise.

validity is a claim, not a property

The authoritative reference is the Standards for Educational and Psychological Testing, published jointly by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education. The 2014 edition makes a point that most internal assessment practice ignores.

Validity is not a property of a test. It is the degree to which evidence and theory support the interpretation of test scores for a proposed use. A test is not valid or invalid. A specific interpretation, for a specific purpose, with a specific population, is more or less well supported by evidence.

This changes what you have to produce. Validity cannot be bought, because it is an argument rather than a feature. The Standards name five sources of evidence you can build that argument from: test content, response processes, internal structure, relations to other variables, and consequences of testing. Validity is treated as a unitary concept, so these are not five types of validity. They are five kinds of support for one claim.

The Society for Industrial and Organizational Psychology publishes a workplace companion, the Principles for the Validation and Use of Personnel Selection Procedures, revised in 2018 to align with the 2014 Standards. It is the document to put in front of a vendor who says an assessment is validated without saying what it is validated for.

reliability sets the ceiling

Reliability is consistency of scores across replications: different items, different occasions, different raters. It comes in several forms and they are not interchangeable. Internal consistency asks whether items measure the same thing. Test-retest asks whether scores are stable over time. Inter-rater reliability asks whether two assessors agree.

An assessment can be highly reliable and completely invalid. It cannot be valid and unreliable, and there is arithmetic that shows why. The observed correlation between two measures cannot exceed the square root of the product of their reliabilities.

Work that through on a typical internal setup. Suppose your assessment has internal consistency of 0.60, which is common for a short home-built instrument. You validate it against supervisor performance ratings, whose reliability is often estimated at around 0.52. The highest correlation you could observe, even if the underlying relationship were perfect, is about 0.56. The ceiling was set before you collected a single data point.

Reliability is therefore not a technical nicety to be checked after launch. It is the budget inside which everything else has to fit.

the numbers that should recalibrate expectations

Most people working in talent carry validity expectations inherited from a 1998 research summary that has since been substantially revised. Sackett, Zhang, Berry and Lievens published a re-analysis in the Journal of Applied Psychology in 2022 showing that standard corrections for range restriction had systematically overcorrected, inflating published validity estimates by roughly 0.10 to 0.20 points.

Their revised operational validity estimates for predicting job performance are worth committing to memory.

Structured interviews: 0.42
Job knowledge tests: 0.40
Work sample tests: 0.33
Cognitive ability tests: 0.31
Assessment centers: 0.28
Situational judgment tests: 0.26
Conscientiousness measures: 0.19
Unstructured interviews: 0.19

Two things follow. The best available methods correlate with performance at around 0.4, which is useful and a long way from deterministic. Anyone reporting that an internal assessment correlates at 0.7 with performance has most likely contaminated the criterion, usually by having the same manager supply both the assessment and the performance rating.

The second point is that structure does the work. The same conversation, structured, predicts more than twice as well as the unstructured version. That finding transfers directly to internal skills assessment, where manager judgment usually resembles an unstructured interview with a rating scale attached.

five ways internal assessments fail

Across enterprise assessment programs, the same five failures repeat.

No content evidence. Nobody linked the items to a documented analysis of the job, so there is no answer to the question of why these skills and not others.

Too few items per construct. Reliability rises with test length. Using the Spearman-Brown relationship, a 10-item scale with reliability of 0.50 has to be four times longer to reach 0.80. Five-item subscales reported to two decimal places are the single most common error in internal assessment.

Single-rater judgment with no agreement check. If you have never computed inter-rater reliability, you do not know whether your talent grid measures capability or measures managers.

Purpose drift. An instrument validated as a learning diagnostic gets repurposed as a promotion gate. The scores did not change. The claim did, and the evidence supporting the old claim says nothing about the new one.

No consequence evidence. Selection rates by group are never examined. Under the United States Uniform Guidelines on Employee Selection Procedures, a selection rate for any group that falls below four-fifths of the highest group's rate is generally regarded as evidence of adverse impact. Whatever jurisdiction you operate in, that screen is a cheap early warning and takes minutes to compute.

a design and validation procedure

This is the minimum sequence for an assessment that will carry real decisions.

write the claim first

One sentence: scores on this assessment support decisions about X for population Y. If you cannot complete the sentence, the design is not ready to start. Every later step produces evidence for that one sentence.

run a job analysis with real experts

Recruit six to ten subject matter experts who currently do or supervise the work. Have them rate each candidate skill on criticality, frequency, and whether it is required at entry. Retain skills that at least 70% rate as critical. Document the panel and the ratings, because this is your content evidence and it will be the first thing challenged.

build a blueprint before writing items

Allocate items in proportion to criticality, and set a floor of eight to ten items for any construct you intend to report separately. If you want five subscores you need a long assessment or fewer subscores. Choose before writing, not after piloting.

pilot and cut the bad items

Pilot with at least 100 respondents where population size allows. Compute item difficulty, item-total correlations, and internal consistency. Drop items whose item-total correlation falls below 0.20. Target internal consistency of 0.80 or higher for decisions about individuals, and 0.70 for group-level workforce insight. Individual decisions demand the higher standard because the cost of an error lands on one person.

set the cut score with a defensible method

Do not pick 70% because it looks like a pass mark. Use a modified Angoff procedure. Each expert estimates the proportion of minimally competent candidates who would answer each item correctly, and you average across experts and items. The result is a cut score with a documented rationale and a named panel standing behind it.

check relations to something outside the test

Correlate scores with an independent criterion collected separately: work samples, output data, or ratings from someone who never saw the assessment result. Expect a correlation between 0.20 and 0.40. Treat anything above 0.60 as a reason to hunt for contamination rather than a reason to celebrate.

check consequences

Compute pass rates by gender, age band, location, and any group your jurisdiction protects. Apply the four-fifths screen. Where a difference appears, investigate the items before defending the instrument. Then repeat the pilot analysis, the criterion check, and the consequence check annually, because populations and jobs both move.

what self-report can and cannot tell you

Self-rated skill inventories are widely deployed because they scale to thousands of people in a week. They are useful, but not for the purpose they are usually given. A self-rating measures self-perception. That is informative about interest, confidence, and intent, and only weakly related to demonstrated capability.

Use self-report to route people toward opportunities and to surface interests the organization cannot see. Do not use it as the input to a capability gap number that will be presented to an executive committee. Where assessment has to scale, the workable compromise is a short structured instrument with real item statistics for the skills that carry decisions, plus self-report and work evidence for everything else. Platforms that run workforce assessments at enterprise scale make that cheaper to operate, but they do not remove the obligation to produce the evidence.

the honest posture

Good workplace assessment predicts performance at around 0.4. That explains a meaningful share of the variance and leaves most of it unexplained. Stated plainly to a leadership team, that number builds more credibility than a claim of precision the instrument cannot support.

The organizations that get this right are rarely the ones with the most sophisticated psychometrics. They are the ones that wrote down what each assessment claims, tested whether the evidence supports the claim, and stopped using instruments for decisions they were never built to carry.