What Kind of Information Do You Want to Collect?

Starting from 1984

You have already read Keith’s 1984 paper. In Chapter Two, he turned from asking whether video had a use to asking what a coach or analyst might actually do with it.

His starting question was practical: “What kind of information do you want to collect?” He invited the reader to identify a particular problem, decide what to observe and produce a recording that might go some way towards resolving it.

We can read “go some way” as a boundary around the task. The recording is not presented as a complete answer. In Keith’s rugby example, a repeated claim about line-outs becomes something that can be examined through a match-analysis framework. Yet he also showed the fragility of the resulting information: if the recording misses the action, “objective analysis can become guesswork”.

The springboard

The analyst’s task is not simply to collect more information. Before a number, category, event map or model can inform practice, someone has decided what to notice, what to call it, how to record it and what question it might help answer.

Clyde Street gives us several opportunities to watch Keith reconsider those practical decisions in different circumstances. The examples do not form a method that Keith announced in advance, and they do not describe a straight line of progress.

Question to carry

When is collected information ready to inform practice?

At each stop, notice:

What was selected?
How was it produced?
What can it support?
Who is it for, and what are they trying to decide?

Do not assume that publication order is the same as the order in which Keith’s thinking or practice developed. One later Clyde Street post looks back at observational work undertaken between 1987 and 1992.

Choosing what will count

2010 — two parameters, sixty-four games and the cases that do not fit

During the 2010 FIFA World Cup, Keith examined all sixty-four matches through two deliberately limited parameters: which team scored first and the teams’ FIFA rankings. Of the teams that scored first, forty-six won, eight drew and three lost; seven matches finished 0–0. Keith also listed the thirteen occasions on which the higher-ranked team lost. Rather than removing awkward cases, he kept them visible and explained how he classified unusual outcomes.

Keith’s figures support a limited statement: in this tournament, the team scoring first usually won. They do not show that the first goal caused the victory. A sending-off, the relative strength of the teams, what happened after the goal and many other features of the match could contribute to the result. The pattern is information; deciding what it can reasonably support is the analytical judgement.

Think: A first goal was scored in 57 of the 64 matches. The team scoring first won 46 of those 57. Which conclusion is justified: “in this tournament, the team scoring first usually won when a goal was scored” or “scoring first makes a team win”? What extra evidence would you need before making the stronger claim?

Judging information collected by somebody else

2011 — official data are still evidence to be questioned

At the 2011 Asian Cup, Keith noted the remarkable opportunities created by official data supplied through STATS: formations, match statistics, event charts and heat maps. The analyst no longer had to produce every observation personally.

But availability did not settle the matter. Keith said that the reliability of such data still needed to be considered. Detail, authority and an official label do not remove the need to ask how the information was produced and whether it is suitable for the present purpose.

Think: If you did not collect the data, what would you need to know before using it—and who would be affected if it were wrong?

Making the production of evidence visible

2013 — Keith looked back at hand notation from 1987–1992

Keith’s later account of England–Wales rugby matches from 1987 to 1992 brings the observer and the procedure into view. He described the categories he selected, his operational definitions, the practical equipment, the cognitive demands of observing, the risk of observer drift and his deliberate decision not to pursue greater granularity.

He also stated a significant limit: he did not conduct intra-observer or inter-observer reliability studies. His confidence in the observations was therefore not the same as a formal reliability test.

Keith linked this earlier work to the period in which he was researching and writing Using Video in Sport. The 2013 post therefore acts as a retrospective window onto some of the practical work behind the information that the 1984 paper encouraged readers to collect.

Think: Which details would another analyst need in order to inspect, repeat or challenge this analysis?

Returning the analysis to its purpose

2016 — tools, reasons and real-world observation

In Coming to Terms with Sport Analytics, Keith brought other writers into conversation. Richard Whittall distinguishes analytical tools from the reasons for using them. Seth Partnow brought Charles Goodhart’s work into the sports-analytics discussion Keith was reading: when achieving a measure becomes the target, the measure may stop telling us what we thought it did. Richard Lipsey’s work returns attention to observation and testing in the real world.

These ideas remain the intellectual property of Richard Whittall, Seth Partnow, Charles Goodhart and Richard Lipsey. Keith returned them to coaching and athlete learning. A team might improve a measured benchmark without improving the performance that the benchmark was intended to represent. The analyst must therefore keep asking what practical problem the measure serves, whether it still represents that problem and how the finding will help a coach or athlete decide what to do.

Think: Imagine that a team begins training mainly to improve a dashboard score. How would you check whether the score still represents better performance—or whether achieving the number has replaced the original purpose?

Data are recorded and transformed

2019 — questioning the idea of raw data

In Discussing data, Keith brought several writers into conversation to question the idea of “raw” data. Their ideas remain their own; his post uses them to prompt more reflexive analytical practice. They include Cassie Kozyrkov, Nick Barrowman, Will Koehrsen and E. H. Carr.

Read together in Keith’s post, these externally owned ideas challenge the assumption that data are an untouched copy of reality. Before a dataset reaches an analyst, someone has decided what counts as an event, when recording begins and ends, where the observation takes place, which categories are available and how an uncertain case will be entered. The dataset contains the results of those choices. This does not make it useless; it means those choices affect what the analyst can later claim.

Think: Before you analyse a dataset, what has already been decided for you? Consider who chose the events to record, how each event was defined, what was left unrecorded and how uncertain cases were handled. Which of those choices would you need to check before trusting the result?

Letting the inquiry change the question

2019 — exploration, surprise and translation

Later that year, Keith brought together work concerned with exploring data, asking questions and communicating findings. The contributing ideas remain owned by John Tukey, Sara Alspaugh and colleagues, Anne Fisher, Nathan Yau and Sean Taylor.

Drawing these writers together, Keith concluded that “we are explorers”, need skills of translation and “become detectives too”. The explorer looks beyond the first expected result and remains open to something surprising. The translator turns complex analysis into clear, practical choices for the people who will use it. The detective follows clues, compares possible explanations and accepts that analysis is rarely a neat step-by-step process. Together, these roles make the analyst more than a producer of numbers.

Set beside the 1984 practical task, this later post adds a further possibility. Choosing information remains necessary, but the first question need not remain untouched. An unexpected result might reveal an error in the data, an unsuitable definition, an unusual event or a genuine pattern that the original question could not explain. The analyst’s task is to investigate which possibility is most plausible before deciding whether to repair the analysis or ask a better question.

Think: Your analysis produces a result you did not expect. What would you check first—the recording, definitions, missing cases or calculation? If those checks hold, what new question might the unexpected result invite you to ask?

What changed across the journey?

Read together, the selected encounters place further demands on the analyst. They do not abandon Keith’s 1984 concern with practical use; they make more of the work behind that use visible.

1984: identify a practical problem and decide what information might help.
2010: define a limited view and retain the cases that complicate it.
2011: treat inherited data as something to check, not simply receive.
2013: disclose the categories, procedures, labour and limits behind observation.
2016: question why the analysis exists and what its measures assume.
2019: recognise data as recorded and transformed rather than raw reality.
2019: let exploration and surprise generate or revise questions.

The educational movement we can construct from these selected sources is from collecting information to investigating how information becomes evidence. This is a learning translation, not a phrase Keith used or a curriculum he authored.

The journey does not turn evidence into certainty. It makes the analyst more able to explain how information was made, what it can support, who might use it, where it might fail and what should happen next.

What can the evidence support?

Use the evidence summarised in this encounter to make an initial judgement about each statement. You have enough information here to begin. The linked Clyde Street posts are available if you want to inspect the full context or test the summary. If you follow a link, notice whether the fuller source confirms, qualifies or changes your first judgement.

Use one of these judgements:

SUPPORTED AS WRITTEN — the selected source warrants the complete claim.
SUPPORTS PART ONLY — some words are warranted, but others exceed the evidence or leave an important condition unstated.
NOT YET SUPPORTABLE — the selected source does not warrant the claim.

Some statements deliberately combine an observation with an overreach. For each one, ask: Which words can the source support? Which require qualification? What would you need next?

In the 2010 FIFA World Cup, the team scoring first won 46 of the 57 matches in which a goal was scored.
Scoring first caused those teams to win.
Official competition data can be used without checking how they were produced.
An experienced observer’s confidence is equivalent to a reliability test.
A measure may become less useful when achieving it becomes the target.
Data provide direct access to what happened in performance.
An unexpected result may justify changing the original question.

Choose one judgement to share. Point to the detail in this encounter that supports it. If you followed a link, you may also identify the passage or table you used. Explain what the evidence allows you to say, what it does not, who might use the claim and what you would check or investigate next.

Bring it back to practice

Choose one analysis, dashboard or secondary dataset you have recently used. Name:

the decision it is meant to inform
what was counted or produced
one detail about its production you do not know
one claim you would refuse to make

Then revisit one of your judgements above. Would the same evidence be sufficient for an analyst, coach and athlete making different decisions? This perspective comparison is a learning invitation arising from Keith’s return to coaching and athlete learning; it is not an exercise Keith proposed.

Learning return

Complete the three sentences and name at least one source:

In 1984, Keith asked us to think about …
Across the selected Clyde Street encounters, I saw this concern reappear, alter or meet a limit when …
By the end of Clyde Street, analysts still need to judge …

Your learning is not demonstrated by choosing the expected category. It is demonstrated by making a judgement that another person can inspect: naming the source, showing your reasoning, preserving its limits and deciding what follows.

Sources followed

Keith Lyons, Use of Video in Sport, Resource Pack 9, National Coaching Foundation, Chapter 2, especially paragraphs 2.7–2.9.

Keith Lyons, “Analysing Performance at the 2010 FIFA World Cup” — 10/06/12

Keith Lyons, “Opportunities For Secondary Data Analysis At The Asian Cup 2010” — 11/01/10

Keith Lyons, “England v Wales Rugby Union Matches 1987–1992” — 13/01/01

Keith Lyons, “Coming to Terms with Sport Analytics” — 16/04/21

Keith Lyons, “Discussing data” — 19/01/21

Keith Lyons, “Working with data in sport” — 19/10/25

The named writers and organisations encountered in these posts retain ownership of their ideas and data. This encounter follows Keith’s acts of connection and practical translation; it does not attribute those external ideas to him.