Finding Where Training Is Needed: A Model and a Test

The Problem

Companies decide what to train on mostly from opinion: a manager’s request, the development section of an appraisal, a compliance rule, a new product. The textbook alternative is a training needs analysis, and CIPD’s guidance lists the data it should draw on, from performance figures and LMS records to business plans. In practice it rarely does. A review of 51 studies found needs analysis is done “in an unsystematic manner”, relies mostly on questionnaires, and reacts to problems rather than looking for them (Ferreira and Abbad, 2013). In CIPD’s 2023 survey of 1,108 learning and development (L&D) practitioners, fewer than one in ten strongly agreed that they make evidence-informed decisions, and only half had a process for assessing whether learning had any impact.

The figures that would answer the question already exist. They sit in other departments’ systems (sales records, quality scores, incident logs) and are rarely linked to a skill.

What Already Exists

Parts of this are on the market. Contact-centre tools such as Observe.AI score every call and flag coaching needs. Gong Enable finds skill gaps in recorded sales calls and assigns training. TechWolf infers each employee’s skills from their work and HR data. All three work inside one department or one profile, and all score individuals.

The Design Rules

The model starts from what the research on feedback and monitoring at work supports (see Data at Work: A Check, Not a Verdict):

• Figures are read by team and skill, never by name, and no group under five people is ever shown.
• A short check finds the skill. A figure alone says where something went wrong, not why.
• A person decides the cause before any training is built. A rise in errors may come from a tool, a process or staffing. A skill gap does not automatically mean a training need (Ferreira and Abbad, 2013).
• Nothing feeds a performance review.
• The same check and the same figure are read again afterwards, so the company learns whether the training worked.

The Model

At the centre is a needs register: one row for each possible need. Each row holds the evidence, the cause a person judged, the decision, the measure agreed in advance, and the result. Five parts feed it:

1. A skill map. Each role is broken into tasks, skills and the errors people in that role usually make.
2. Signals. Monthly figures from existing systems, stored by team only.
3. Links. Which figure each skill is believed to affect. Every link is a hypothesis, and the outcomes either support it or disprove it.
4. Check results. These are kept in two stores that are never joined. Each person’s own answers drive their adaptive practice. Managers see team totals only.
5. Outcomes. The same check and the same figure after the programme.

A need moves through it in five steps. A figure rises above what the team’s own history predicts, and a row opens. The team takes a ten-minute check. A person judges the cause. If it is a skill gap, practice runs on that skill. At the review date, the check and the figure are read again.

The Test

No public data links skill checks to work figures, so the test used a fake company with the answers hidden in it. The staff came from IBM’s fictional HR dataset: 1,470 people, split at random into about 170 teams, about 19 of them under five people. A year of monthly figures and skill-check answers was generated on top. It was then made messy, with duplicated rows, team names written three ways, missing months, seasons, a reorganisation, a company-wide price rise, and people skipping checks.

Six problems were planted, and the model never saw where:

• a real skill gap in one sales team
• a broken label printer in one lab team
• a one-month spike with nothing behind it
• a gap in a team under five people
• a gap that moves no figure
• a figure that rises for a reason the skill map wrongly blames on a skill

The model ran blind, and was then marked against the hidden answers. This was repeated 20 times with different random companies.

Case	Right
Real skill gap found and trained	Found 20/20; closed 15/20
Broken printer sent away from training	20/20
Company-wide price rise not mistaken for a training need	20/20
One-month spike ignored	20/20
Wrong skill link spotted	13/20
False alarms	About 2.2 a year, every one closed by the ten-minute check

What It Got Wrong

Privacy works against detection. Teams under five people rarely produce enough work to show a change. When one did, merging it into its whole role to protect individuals diluted the gap, and it was missed every time.

Small groups can fake a success. In one run, chance made training look as if it had worked in a team of about six.

A gap that moves no figure is invisible. The register only looks where a figure moves.

Performance ratings carry no signal. In IBM’s file, 85% of staff are rated 3 and the rest 4. A rating with two values cannot point at a skill.

What Would Fix It

• Read small teams over a quarter rather than a month, and estimate their rates by borrowing strength from similar teams.
• Give everyone a light check twice a year, which also works as retrieval practice.
• Do not judge a cause when too few people took the check.
• Stagger training so half the affected teams start a quarter later and act as the comparison group. This is the strongest way to show training worked.
• Set the alarm level from the company’s own history.

What a Company Would Need

The software is the small part. The model needs a skill map with common errors for each role, a bank of short questions tagged to those skills, team figures that include counts, rules agreed in advance, legal groundwork (a data protection impact assessment, the works council, and the EU AI Act’s duties for AI that evaluates workers, which apply from 2 December 2027), and an L&D lead to judge causes. The skill maps cost the most to build, and they are the part a learning designer is best placed to build.

The test shows the logic and the privacy rules behave as designed on messy data. Whether the model leads to better training decisions in a real company has not been tested.

The next piece takes one row of the register and designs the course for it: Refunds Done Right: From a Training Need to a Course.