You collect data from an official source and begin the analysis.

Then you look again.

The numbers have changed.

Which version is now your evidence?

In October 2019, Keith Lyons encountered that problem while analysing the Rugby World Cup.

Twenty-nine games had been played. Keith was collecting match data from the World Rugby website and experimenting with a way of describing each game with a single number. He combined kicks, passes, lineouts and scrums into a ratio that he hoped might say something about the mobility of a match. He was also using the tournament to improve his work with R and to experiment with different visualisations.

Then he noticed a problem.

The official numbers did not stay still.

In #RWC2019: patterns after 29 games, Keith noted that figures changed after matches had been played, particularly the number of passes.

So he made a choice.

For his own record of each game, he used the World Rugby data as they stood twenty-four hours after the final whistle.

That decision is easy to skim past.

For this article, it is the hinge.

The problem was not that Keith had no data. He had plenty.

The problem was that the data he was analysing could change after he had collected them.

Anyone working with live or continually updated information can recognise the difficulty. You download figures from a dashboard on Monday. By Wednesday, some values have been corrected. An API revises historical records. A public dataset is updated. A spreadsheet supplied by somebody else changes after you have begun the analysis.

Which version is now your evidence?

Keith's answer was not a universal rule.

There is nothing in the Clyde Street post to show that twenty-four hours was the ideal waiting period, or that later World Rugby figures were somehow worse than earlier ones.

He needed a record he could work with, and he made the point at which he captured that record visible.

That matters.

Because once the evidence itself is moving, saying I used the official data may no longer tell another person exactly what you analysed.

When the source keeps changing

Suppose Keith had collected the figures immediately after every match and never recorded when he collected them.

Someone attempting to reproduce his analysis later might retrieve different numbers from the same official source.

Both people could truthfully say that they had used World Rugby data.

They might still be analysing different records.

The problem is much larger than sport.

A researcher using government statistics may discover that historical figures have been revised. An analyst may build a report from a live business dashboard whose underlying data are later corrected. A teacher using an automatically generated performance report may find that results are recalculated after late submissions arrive.

None of this makes changing data inherently bad.

Correction can improve a dataset.

But it creates a judgement:

Do I need the latest available record, or do I need to preserve the record on which this analysis was actually based?

Those are not always the same requirement.

Keith's post does not solve that problem for us.

What it shows is the moment when the instability of the evidence becomes consequential for the work.

He noticed that the source values moved.

And he changed the way he constructed his own record because of it.

The limitation has changed the work.

Not every uncertainty changes the work

That boundary matters.

Clyde Street contains much more ordinary uncertainty.

During the CCK08 course, Keith once struggled to access a paper before finding it online about thirty minutes later. In the same post he acknowledged that he might be missing an important economic point in an argument.

Those are real difficulties, but they are not shown having the same consequence.

One problem disappears when the paper becomes available.

The other is an appropriately cautious acknowledgement that Keith might have missed something.

Neither is shown changing the method, narrowing the claim or redirecting the inquiry.

That gives us a useful distinction.

The question is not simply:

Is there uncertainty here?

It is:

What, if anything, does this uncertainty change?

Sometimes the answer may be: nothing yet.

Sometimes the answer is much more substantial.

What happens when the missing evidence is a person?

One day after the Rugby World Cup post, Keith published something very different.

Charles, Richard, Neil and Simon: the stories we craft is concerned with the history of sport analytics and, particularly, the ways Charles Reep and his work had been represented.

Keith began with different accounts already in circulation.

Tom Fenton and Rob Carroll had published a story about Reep. Richard Pollard had challenged later interpretations of Reep's passing-sequence work and some of the claims built upon them.

Keith did not settle that dispute for the reader.

Instead, the competing accounts take him towards another problem:

How does a historical story get made at all?

Keith already had substantial material.

Richard Pollard had preserved correspondence from Reep, match analyses and other documents. Keith had corresponded with Richard and Neil Lanham about their experiences and careers. Neil had shared material of his own. Keith was also reading work concerned with historical documents, ethnography, authorial voice and collaborative interpretation.

But there was an important gap.

Keith wrote:

“I have not spoken with Simon Hartley.”

Simon was not incidental to the history Keith was considering.

Keith was thinking about Charles, Richard, Neil and Simon.

The absence therefore mattered.

It did not mean there was no history to write.

But the evidence Keith already possessed could not silently become Simon's perspective.

A gap inside the work

This is a very different problem from the Rugby World Cup case.

There, the source itself keeps moving.

Here, the available record is substantial but incomplete.

Keith described an approach to the history that drew on documents, correspondence and participation. He envisaged sharing stories with the people involved before posting them publicly. He described an iterative process and paid attention to who produced documents, what kind of documents they were and how they related to other evidence.

An important restraint is needed here.

The post does not establish that Simon Hartley's absence caused Keith to devise that entire approach.

We should not manufacture that causal sequence.

What the source does show is that Keith left the missing perspective visible within an inquiry that was already concerned with participation, provenance, interpretation and the construction of historical accounts.

He did not make the gap disappear by writing around it.

And the intellectual ownership around that inquiry matters too.

Keith explicitly drew on Clifford Geertz, John Van Maanen, Jean-François Rouet and colleagues, Sam Wineburg, Erin Comollo, Arthur Costa and Bena Kallick.

Those ideas remain theirs.

Richard Pollard and Neil Lanham remain owners of their testimony and materials.

Simon Hartley's missing perspective cannot be supplied simply by combining everybody else's.

Nor does an iterative or participatory approach guarantee that the history eventually produced would be true.

It offers no magical escape from interpretation.

In the process Keith described, that incompleteness remains visible rather than being written out of the account.

That is enough for the distinction we need here.

When is incomplete actually a problem?

Every inquiry is incomplete in some sense.

No analyst has every possible variable.

No historian has every possible document.

No teacher hears every student's private experience.

No coach sees everything that influenced a performance.

If complete knowledge were the standard, very little work could ever finish.

So the existence of missing evidence cannot, by itself, tell us what to do.

The harder judgement is whether what is missing matters to the claim we want to make.

Imagine interviewing people about a failed project.

One important participant declines to speak.

You still have meeting notes, emails, performance data and interviews with six other people.

That evidence may establish some things extremely well.

But if you begin writing as though the seventh person's motives and experiences are already known, the missing perspective has become consequential.

That does not automatically mean abandoning the project.

It might mean narrowing a claim.

Making the absent perspective visible.

Looking for another kind of evidence.

Or leaving part of the account unresolved.

Clyde Street does not turn those possibilities into a procedure.

The more modest point is this:

If missing evidence matters to what you want to claim, the way you make the claim should reflect that absence.

Two evidence problems, two different consequences

The two October 2019 posts sit next to each other in Clyde Street.

But they are not one project.

And they do not form a causal sequence.

That is precisely why they work well together.

In the Rugby World Cup analysis, Keith had an official data source but discovered that its values changed.

His response was explicit: he defined the point at which he would capture the figures used in his analysis.

In the historical post, Keith had extensive documentary and personal material but lacked a perspective he regarded as relevant.

He did not manufacture that perspective.

Its absence remains visible within the historical account he was trying to construct.

In one case, the evidence needs stabilising for the purpose of the analysis.

In the other, no capture rule can make the available evidence complete.

So the useful lesson is not:

When uncertain, do this.

It is:

Find out what the evidence problem changes.

That may sound straightforward until the pressure to finish the work arrives.

A deadline encourages us to carry on with the dataset already downloaded.

A polished dashboard makes revised values look definitive.

A strong collection of interviews makes the absent participant easier to forget.

A persuasive account written by somebody else can begin to sound like a substitute for the perspective we do not have.

A fluent AI-generated explanation can smooth conflicting evidence into a single story before we have decided whether the conflict should be resolved at all.

The problem is not uncertainty itself.

It is allowing an uncertainty that matters to the claim to disappear from the way the work is done.

What should the limitation change?

There is a familiar way of dealing with evidence problems after the event.

Finish the analysis.

Then add a limitations paragraph.

The data may have changed.

A participant was unavailable.

Some records were incomplete.

There may be alternative interpretations.

Sometimes that is entirely appropriate.

Sometimes it is not enough.

If an evidence problem could materially alter the conclusion, comparison, historical account or the confidence a reader should place in what is being claimed, then a closing disclaimer may be insufficient.

The important question arrived earlier:

Should this limitation have changed the work itself?

Keith's two cases give us different ways to inspect that question.

In one, instability changes how the evidence is recorded.

In the other, an absent perspective remains visible within an inquiry concerned with how a historical account is constructed.

Neither case tells us that uncertainty is virtuous.

Neither tells us to delay indefinitely.

Neither tells us that every missing datum requires a new methodology.

And neither gives us a universal procedure.

They show something narrower:

When a problem in the evidence matters to the work, let it have consequences for the work.

For an analyst, that might mean recording exactly when a changing dataset was captured.

For a researcher, it might mean narrowing a conclusion when an important perspective is unavailable.

For a teacher or practitioner, it might mean resisting a confident explanation while consequential information has not yet arrived.

Sometimes it may mean leaving something unresolved rather than asking the evidence to say more than it can.

That is not necessarily a failure to finish.

It may simply be the most accurate description of where the inquiry has reached.

Clyde Street leaves us with a question that is useful before the analysis is polished and before the limitations section is written:

Question to carry

What is wrong, missing or still moving in the evidence — and what does that actually require me to change?

Sources followed

Keith Lyons, “#RWC2019: patterns after 29 games” — 19/10/09

Keith Lyons, “CCK08: Week 11 Possessive Individualism Revisited in the Post Historical University” — 08/11/17

Keith Lyons, “Charles, Richard, Neil and Simon: the stories we craft” — 19/10/10