annex to the change-map demo · software
The same map, two questions, opposite answers
The change-map demo asks which files a fix that doesn't exist yet will touch. This annex asks something that sounds similar and isn't: who breaks if I change a shared piece today.
We measured both, with the same dependency map. And the answers came out opposite: for knowing who breaks, the map wins hands down. For predicting what the fix will touch, it adds nothing. Both measurements are here, with their numbers and their limits.
where this comes from
This page wasn't planned. On 7 August 2026, Francisco Borja Escalona opened a debate on LinkedIn about how to get artificial intelligence to understand a company's code, and several people who have spent a long time wrestling with the problem gathered there. We were missing a number to have an opinion, so we went and measured it. Our thanks to him for opening the conversation, and to those who argued in it: a good part of the limits listed further down come from their objections. None of them endorses what is on this page.
The question, minus the jargon
A big program has pieces that many other parts use. Changing one of those pieces is like changing a pipe in a building: before touching it you want to know which floors the water reaches. We call that list of affected places the blast radius.
There are two ways to find out. The first, and the one almost everyone uses, is a text search (grep): looking for the piece's name across the program's whole text, the way you search for a word in a document. The second is to ask a dependency map: a chart, drawn by a machine, of which part calls which part.
It's usually taken for granted that the second is more reliable and the first is the workaround. We went and checked, applying both to the same code.
Why this costs a company money
When someone changes a shared piece and misses a part of the program that depended on it, nothing happens that day. It happens two weeks later, in production, in a part of the system nobody linked to that change. It's the kind of failure that gets paid for dearly precisely because nobody saw it coming.
And that calculation is now increasingly done by a machine. A coding assistant that gets the scope of a change wrong can write code that compiles, passes the tests and breaks a part nobody was watching. The question of what breaks stops being a manual check and becomes the piece of data you have to hand the machine before letting it work.
Hence the interest in knowing which of the two lenses gets it wrong, by how much, and in which direction.
What came out
Four repositories, six pieces from each, chosen for being the most used and not cherry-picked. The rules were fixed in writing before looking at any result. Three ways of searching by text are compared against the dependency map.
| Application | By its name | Only what imports it | Map | Only the map sees it | Only the text search sees it |
|---|---|---|---|---|---|
| Application A | 241 | 198 | 293 | 106 | 11 |
| Application B | 605 | 357 | 333 | 205 | 229 |
| Application C | 703 | 521 | 474 | 97 | 144 |
| Application D (the largest) | 1,430 | 668 | 2,018 | 1,578 | 228 |
| Total | 2,979 | 1,744 | 3,118 | 1,986 | 612 |
Six pieces per application, 24 in total. Of the 2,979 appearances that searching for the name finds, 1,378 are noise: comments or text where the word shows up for some other reason.
- Refining the text search removes noise and doesn't close the gap. Requiring the file to import the piece rather than just name it drops the count from 2,979 to 1,744: 1,235 places that looked affected weren't. That's more than the 1,378 mentions the table flags as noise, because the noise only covers the ones sitting in comments or inside a string; the rest are uses of the name that aren't a dependency. And even so, the ones only the map sees are still 1,986: refining the search cuts the false positives, but recovers nothing of what gets left out.
- The map misses 612 cases too. It happens mostly when a piece is offered again from another file, or is reached by an indirect path. Which is why the conclusion isn't to switch lenses: it's that you need both.
- The two extreme cases. In one, the map finds 365 affected places and searching for the name finds one. In another, the map finds 561 and the search finds two. They're the most conclusive examples because there's nothing to classify: either you import the piece or you don't.
Where they agree and what each one misses
If one of the two lenses were enough, one of the bands at the edges wouldn't exist. Both exist, in all four applications.
The bars share the same scale across applications, so the relative size shows too: the biggest program doesn't just have more pieces, it has the widest amber band in proportion. The bigger the code, the more a text search misses.
Why the text search fails
It isn't that it searches badly. It's that pieces travel in company. A file exports several things, and whoever uses any of them breaks just the same when you touch the file, even if it never mentions the name you're searching for. Those are most of the 1,986 that only the map sees, and by definition no text search can find them.
The second reason is the name. A piece called user shows up
everywhere for reasons that have nothing to do with it: in our case, 759 appearances of
which 737 were noise. With names like that, the text census isn't imprecise. It's useless.
The flaw we found by measuring ourselves
The first time we ran this, the largest application gave 827 appearances of one piece. Repeating it on a clean copy gave 417. Half were the same file counted several times: the working directory had copies of the repository itself inside, 3,770 duplicate files nobody had put there to fool anyone.
The dependency map, on both passes, gave 427. Exactly the same.
This isn't a quirk of our machine. It happens to anyone with working copies, third-party libraries in the tree or build folders with sources inside. And the dangerous part is that it leaves no trace: it doesn't fail, it doesn't warn. It just returns a bigger number, which is exactly what you expect from a census.
The 365 case, drawn
A single piece from the largest application. Each dot is a file that would stop working if it changed, coloured by the layer it belongs to. The amber dot is the only one that searching the code for the name finds. The other 364 never mention that piece anywhere, and they break just the same.
The split by layers says something the number alone doesn't: the radius isn't concentrated in one corner. There are 162 services, 100 tests, 37 utilities, 23 background jobs and 9 entry routes. It cuts across the entire application.
The other question: here the map is no use
With the above in hand, the natural thought was that the map would also improve the predictions of the change-map demo. We went and checked, on the same fifteen years of pandas' public history that demo uses, with its same exam and its same frozen parameters.
The change-map system scores with two signals: the text of the reported problem and the memory of which files change together. We added a third, structure: who imports whom, read from the code exactly as it stood at each cutoff date, so as not to give the system information from the future.
The weight of the new signal was chosen looking only at the calibration years, never the exam years. And that calibration chose on its own not to use it: the winning weight was zero.
forcing the signal in the exam years (1,125 cases)
| Right 1st try | In 3 | In 5 | |
|---|---|---|---|
| The change-map system | 0.276 | 0.483 | 0.581 |
| With structure added | 0.254 | 0.473 | 0.572 |
Worse on all three measures, and worse in each of the four years taken separately. It isn't noise.
It makes sense once you think about it: the memory of what changes together already captures the dependencies that matter in practice, and it also captures the ones not written in the code, like two files that always get touched together even though neither mentions the other.
Along the way, this test served to rerun the original experiment from scratch, with one more month of data: it gave 0.276 against the 0.277 published in July. It reproduces.
And that is the result we were after: the same map that finds 365 places where searching for the name finds 1, adds nothing when the question is what a future fix will touch. It works for the structural present, not for the probable future. It's a distinction almost nobody makes, and getting wrong which of the two questions you're facing is what makes a good tool look useless, or the other way round.
Why we built the graph ourselves
This test needed a dependency map of pandas frozen at each cutoff date. The first thing we tried was graphify, the tool the thread that started all this talked about, with over a hundred thousand stars. Before building anything on top of it, we measured how much it saw.
On pandas frozen at 1 January 2021, with 1,227 Python files: the parser the language itself ships finds 1,088 files with internal dependencies, and the tool sees 569. It misses about 530, half of them.
Had we built the signal on that extraction, the experiment would have measured the gaps of one tool on Python instead of the map's value, and the conclusion would have been false without anyone noticing. So we built the graph with the language's own parser: exact, dependent on nothing and reproducible by anyone.
Two warnings we owe ourselves: we ran it in its local mode, and the tool itself warns that without an API key it does no semantic extraction; and it could be attributing dependencies to modules instead of files, in which case our count would underestimate it. We haven't ruled out either, so this is not a verdict on the tool: it's the reason we didn't use it here.
The caveats
- One piece per module, the first one the script finds. Not necessarily the most representative.
- Only the scope visible in the written code gets measured. What is decided while the program runs is invisible to both lenses, and that's a real limitation.
- 24 modules from four applications of ours. It's a measurement, not an evaluation: it serves to show an effect and its size, not to give a general rate.
- The code is private, so nobody outside can reproduce these exact figures. That's why we publish the script: what's reproducible is the method, on the code of whoever runs it.
- The earlier table of this same measurement, with 17 modules, is not comparable with this one: the instrument and the sample changed at the same time.
try it on your code
You don't have to believe our numbers. This is the script that did the measuring, as is: censo.py. It runs on any repository with its dependency graph and tells you how many places you are missing. It carries the classification protocol inside, written before looking at the results, so you can argue with it. If you get something different, we want to know.
Built with Python. The dependency map is generated by graphify, an open project that extracts symbols and relations locally. The classification rules were fixed in writing before looking at the results. No language model takes part in the measurement.
About this demo
An internal measurement on our own code, published with its method and its limits. It doesn't evaluate third-party tools and doesn't claim one lens replaces the other: the conclusion is the opposite, that you need both. The figures correspond to the measurement of 8 August 2026 and don't recalculate themselves.