How a Reference Library Is Researched Before It Is Written
Automated research for a reference library: a layered literature review, an emergent knowledge map, an editor, and a platform that learns from its readers.
I run a consulting practice, and the practice runs on a research platform we built. Its public half is agility-at-scale.com: a growing library of reference papers on enterprise AI, organisational change and scaled agile, written from practice and open to anyone. The map behind it spans more than twenty fields and several hundred topics, and the library is still being written into it.
Every paper in that library came out of research. The engine behind it follows the research lifecycle any researcher would recognise, with most of the labour automated. It starts from what a practitioner believes about a field and tests those beliefs against the literature. The literature is reviewed in layers, and a review is accepted only when it brings the counter-arguments with it. Thousands of concepts have been extracted from those reviews so far, and the map of each field is clustered out of them. Every paper is planned before it is written, down to what each section contributes that no other section does, and every fact it cites is traced back to its source and confirmed there. Then an editor takes over. All of that work produces a proposed draft: I read it, improve it, send it round again when it needs it, and publish it when it is right.
After publication the platform learns from its readers: where they are in their own work, and what they need next.
Where a knowledge map comes from
What the engine draws for each field is a knowledge map: the concepts of the field, how they relate to each other, and the topics they group into.
1. A thesis, stated and tested. A knowledge map starts from what an expert practitioner already believes about the field, stated as assertions. Each assertion is validated against the literature. The thesis sets the questions the research is commissioned against. The map itself is built from what the research returns.
2. A layered literature review. Research briefs run across three layers: the topic, the academic literature, and the named thought leaders of the field. A brief is accepted only when it carries high-strength evidence, a critical mass of counter-arguments and novel intersections: points where two bodies of work meet and nothing published joins them. The counter-arguments are required so that a review cannot simply confirm what commissioned it.
3. Concept extraction. The briefs are mined for concepts: the named ideas, methods, roles, artefacts and standards of the field. A field needs at least a hundred distinct concepts before a map is drawn from it. Each of the thousands of concepts tracked today carries its category, its relations to other concepts and the places it appears.
4. The map emerges. Concepts cluster into topics. Each topic is placed by how central it is to the field, from the core outwards to the periphery, and is given the reader it serves and the perspective it will be written from. An editor shapes the map before anything is commissioned from it.
5. A demand check. Each topic is then read against what people are actually asking, which adds topics the literature missed and removes topics nobody is asking about. Depending on the field, that can remove up to half of the first list of central topics.
The thresholds in steps 2 and 3 are working values, set from practice and written into the method so that every field is held to the same ones.
Where a paper comes from
6. Gap and seam analysis. The engine looks for three kinds of gap, each with its own instrument.
| Where the gap is | What is compared | What comes out |
|---|---|---|
| Within a map | each topic against what is already published elsewhere on it, and against how often its concepts appear | the gaps a new paper can fill, which steer the search for sources |
| Between maps | the concepts two fields share | one canonical home per shared concept and a differentiated treatment everywhere else; where the overlap is rich enough, a seam field built as a field in its own right, such as scaled agile and AI |
| Between the literature and the library | every concept against every topic, across every field in the map | the places where the structure calls for a paper that does not exist yet |
7. A specified, sourced brief for every paper. Before anything is written, a paper exists as a specification. Each section states its perspective, what it must cover, its concepts and the depth each is owed, the questions it answers, the lens it is written through, where it came from, and its claim to information gain: what this section adds that no other section does.
In the library’s guide to AI use-case prioritisation, the opening section’s perspective reads:
Most guides treat identification and prioritization as one step. Separate them: identification is discovery (finding candidates), prioritization is the portfolio bet (choosing which few to fund). The value is in the second.
Its claim to information gain:
Only section to source Lewis Walker’s (Founder, Scan AI) explicit ordering principle, “value first, feasibility second, governance always”, as the sequence the rest of the page follows; every later section operationalizes one of these three steps rather than restating the ordering itself.
The published paper can be checked against its brief, section by section.
8. Research and writing against a dossier. Each paper gets its own research pass, in several phases and across a range of search and research providers: topic search, academic search, the discovery of thought leaders, and a reading of what others have already published. By the time writing starts the brief has grown into a dossier that is usually larger than the paper it will produce. It holds the paper’s position in the map, its step on the reader’s journey, the audience, the priority sources already read with their key data extracted, the vetted sources ranked by strength, the gaps in what others cover, and the practice’s own perspective. The draft is then checked at length before anyone is asked to read it: for repetition, for structure, for quality, that its links hold, and that its sources are distinct from each other.
What happens before anything is published
9. The citation check. A reader should be able to follow any citation in the library and find what we said was there. So the engine follows every one first: it goes to the source, reads it, and confirms the fact the paper attributes to it.
10. The editor. By this point a paper has been researched, sourced and checked, and it is still a proposed draft. It goes to an editor. I read the whole paper as a practitioner in the field: I test its argument against what I have seen in practice, rework what is weak, send sections back for more research where that is what they need, and give the paper its final shape. Then I release it for publication. The same principle runs through our assessment service: we automate as much of the labour as we can, so that more of the time goes to expert judgement, which is what the reader or the client came for.
The loop closes
11. The map updates. Each published paper reports back which concepts it made prominent, and that is written into the map it came from. The next paper in that field is planned against a map that knows what has already been said.
What a reader meets
The library is served through components, each built from the same map.
| Component | What it does |
|---|---|
| Map room | The interactive graph at the head of a paper: where you are in the field and what connects to it. |
| Seven-step journey | Every paper is placed on a reader’s journey. For enterprise AI: readiness, use cases, ROI, pilots, KPIs, operationalising, scale. |
| Pivot points | One question at the point in a paper where it becomes live, multiple answers, and a chosen destination for each. |
| Field findings | A finding from real engagement data, set into the reference papers on the same subject. |
| Internal links | Placed from the map’s own structure, then verified on the live paper. |
| Stable section anchors | Every heading carries a stable address, so a report can link to a section and the link holds. |
| Tools | Working tools on the platform. The first is a NIST AI RMF profile drafter. |
| Research series | Measured in the field: a growing series of studies, one measured finding each, from anonymised engagement data. |
The platform learns from its readers
The library is written to be read as one body of work. Reading across it, and following the journeys readers take, we map the common pathways through a field. Where those pathways fork, the platform offers the reader a pivot point: a question, and the most useful next step for each answer, drawn from our research and our experience of the field.
Part-way down the use-case prioritisation guide quoted above, once the reader has seen how use cases are scored and ranked, the paper asks: “now that your use cases are scored and ranked: what happens next?”
| The reader answers | They are guided to | Step on the journey |
|---|---|---|
| Proving the ranking pays off in dollars | ROI and success metrics | ROI |
| Testing the top pick before betting big | Pilot projects | pilots |
| Knowing what to track once it is live | Performance metrics and KPIs | KPIs |
Each question comes from the paper itself, its place in the map and its step on the journey, and a destination always sits at the same step as the paper or a later one. The answers are anonymous and they accumulate. They tell us where the people reading a paper are in their own work. That is how the platform learns from its readers: what they tell us becomes research, and the research goes back into the library.
This makes the library an interactive corpus. The papers are written so that a reader can follow one thread across ten of them, reading a few sections in each, and the moves between papers stay out of the way. You can join at any point and leave at any point, and get what you need when you need it, without combing through thousands of words for the one part you are looking for.
Where the field data meets the library
The same method runs inside organisations. In an engagement, the words of dozens, hundreds or thousands of people are coded, and the organisation becomes a network of what moves with what. The framework emerges there too: a survey goes in with the dimensions we designed and comes out with more, and in one engagement about half as many again surfaced from what people chose to write. One of them is written up in Two teams cleaned the same data twice: no question asked about it, and people raised it anyway. The service is described in the assessment post.
On both sides the structure emerges from the evidence. A knowledge map is drawn from the concepts in the literature. A picture of an organisation is drawn from what its people say.
The two sides meet in the library. Every theme that surfaces in an engagement is read against it: does the library already have a home for this, is the substance scattered across many papers, or is it missing? A missing theme becomes a research commission. And a measured finding, such as the relation between technical debt and stakeholder trust, is set into the reference papers on the same subject, at the section where it applies. The library says what the literature holds. The studies say what was measured. A disagreement between the two goes to the front of the research queue.
The judgement at the top
Judgement stays with a person at fixed points: the thesis that starts a map, the shaping of the map, the choice of sources, and the editing of every paper before it is published. The tooling never writes up a measured correlation as a cause. Whether it is one is a judgement, and a person makes it.
A living library
Learning from readers has only just begun. Today the answers readers give are read by people and acted on by people. The direction is for them to commission research the way a gap in the map already does: a pivot point whose answers keep choosing a destination the library serves thinly is a request for a paper. Further out, the direction is a corpus that assembles itself around the reader, with sections drawn from many papers and put together for the place that reader is in. The same direction holds on the field side, where years of engagements are being turned into further studies.
How the operation underneath is built and run, agents working under review with an editor deciding what is published, is told in the field report.
If this is your field
If you work on research method or knowledge engineering, and any of this touches your work, write to me.