Deep dive · Hole detection
How the hole detector was built
GroupLab's detector was not trained, and it was not guessed. It was measured into existence: 343 real hole measurements first, a written test it had to pass second, synthetic sheets to practice on third, and then every miss on real paper turned into a rule with its reason written beside it.
343
hole measurements on 15 scans of commercial targets, before GroupLab's detector was written
5
findings that changed the design: 4 assumptions overturned, one confirmed
3 to 9×
a hole is wider than any printed line, on every target in the survey
0
learned models: every decision is a rule you can read
The idea
Subtract what was printed
The photo, minus what was printed, leaves the holes.
A generic detector has to tell a bullet hole from a printed ring, a numeral, a barcode and a pen stroke. GroupLab printed the sheet, so it knows where every printed line is to a fraction of a millimeter. It draws that artwork again, lays it exactly over the photo through the markers, and subtracts it. Rings, numbers and bulls cancel out. What is left is everything that was not printed: the holes, and anything written on the sheet, which the definition's exclusion zones and the classifier then deal with.
This is called render-and-difference. The rest of this page is how its details were decided.
Step 1 · By 13 September · Measure first
343 holes, and five findings that changed the design
Before a line of GroupLab's detector was written, seventeen scanned files of fourteen commercial targets were measured by a Python and OpenCV harness: 343 hole measurements on 15 of them. Every number in the survey is labeled measured, estimated, eyeballed or not measured. The design had listed what a hole looks like. The survey checked each item.
DROPPED
A hole has a bright center
The design said every hole shows the white scanner lid through it. Measured, the center is on average darker than paper, and a quarter of holes have a center within 5 gray levels of the paper around them. No brightness cut separates holes from paper.
Core 53 levels darker than paper · 24% at paper level
DROPPED
A hole is round
A bullet hole in paper is a lobed star. Any roundness test throws away real holes in proportion to how strictly it is set, which is exactly why a filled-circle detector fails.
Rim radius varies by 0.48 of its mean
RAISED
The naive baseline is weak
Tuned for one picture, a Hough circle detector finds nearly every hole. The real problem is that its best setting changes from picture to picture, so the test had to change.
26 of 27 at one setting · 0 of 27 at another
SPLIT
Color tells a hole apart
Chroma works on color-printed targets and does nothing on black or gray ones; brightness is the reverse. On a grayscale target the ring and the hole are the same gray. The channel must be chosen for each picture, from a measurement.
Best separation 0.14 on the grayscale target
KEPT
Scale holds everywhere
Every printed feature on every target is at most 0.0567 in wide; every hole is 0.15 to 0.54 in across. The gap holds on every style, including the grayscale one, and survives any resolution because it is in inches.
Holes are 3 to 9 times wider than any printed line
What that left
Not brightness, not roundness, not color. Scale, measured in inches, is the primary test, and everything else refines it.
Widths on real targets, inches. An opening with a disk wider than the widest printed line erases all printed line work while every hole survives it. That is why one opening erases the printed lines of every style in the survey, whatever the artwork. Hand ink overlaps both, so it is a job for classification, not morphology.
Step 2 · 13 September · The test comes first
A gate that rewards stability, not luck
The original plan said the detector must beat a naive baseline of about 25 percent. The survey showed a tuned Hough circle detector finds 26 of 27 holes on the reference scan, but only at one setting: at an accumulator threshold of 25 it finds 26, at 40 it finds 10, at 55 it finds none. Naive methods are not bad at finding holes. They are bad at finding them without being tuned for each picture, which is useless in an app. So the gate was rewritten.
| Gate | Requirement | Why |
|---|---|---|
| G1 | At least 95% of holes across all 15 files, with one set of settings and no per-image tuning | Beats the tuned baseline while forbidding what makes it useless: tuning per picture. |
| G2 | At most one false mark per target, and none from the printed artwork | Rings, numerals and dots were already rejected on size; what remained was text, barcodes and handwriting. |
| G3 | Still above 90% when every threshold moves by 30% either way | The test the naive methods fail. It is what separates a shippable detector from a lucky one. |
| G4 | The same settings pass at 93, 300 and 600 dpi | Proves no absolute gray-level threshold survived. |
| G5 | At least 95% on the grayscale target | The hardest file: no color at all, ring and hole the same gray. If it works there it works anywhere. |
Step 3 · 14 September · Practice on synthetic sheets
Holes drawn from the survey's own numbers
None of the seventeen scans was a GroupLab sheet, and render-and-difference needs one. So GroupLab draws its own shot sheets: the real render of GL-CF25-LTR, with holes whose shapes come from the survey. Development used seeds 1 to 3. Seeds 1001 to 1003 were kept aside and run once, at the end, so the result is not the product of tuning to the test. Every synthetic score on this page is an upper bound, not a prediction for paper.
What a synthetic hole is
A dark rim at a set radius, lobed by four patterns of narrow outward spikes, with 3 to 6 lobes around the rim, so it is a ragged star, not a circle. A rim band of measured width and darkness. A core where the paper is gone, showing the scanner lid or, in the dark-backing case, a board. A disturbed zone outside the rim that recovers to paper. Holes on printed ink get the different rim the survey measured there. Over the page: hand ink, a paper gradient, a 0.6 pixel blur and noise.
src/GroupLab.Core/Detection/SyntheticSheet.cs
| How real they are | Real | Synthetic |
|---|---|---|
| Hull diameter, in | 0.272 | 0.324 |
| Core brightness, V | 195.6 | 175.6 |
| Darkest rim, V | 33.8 | 32.1 |
| Raggedness, in | 0.0456 | 0.0370 |
Development, one change at a time.
| Change, and why | One per bull | Two per bull | Registration off |
|---|---|---|---|
| The survey's opening, radius 0.032 in. Every miss was a thin rim around a pale center, erased by a disk wider than the rim. | about 85% | ||
| Opening cut to radius 0.012 in. In a difference the artwork has already cancelled, so the opening only removes registration slivers. It broke registration tolerance instead. | 97.6% | 73.8% | 60.7% |
| Each bull cell aligned by phase correlation. The printed rings are fiducials too, so each cell lines itself up locally. | 97.6% | 73.8% | 92.9% |
| Merged neighbors split; oversized holes flagged. Single holes stretch to 1.26 to 1.37 at the 99th percentile (1.61 at worst); merged pairs from 1.50. The split point sits between them. | 97.6% | 95.2% | 92.9% |
The held-out run, once.
| Sheets | Render-diff | Baseline | Median error, in |
|---|---|---|---|
| 600 dpi, one per bull | 100.0% | 85.7% | 0.0063 / 0.0112 |
| 600 dpi, two per bull | 94.0% | 67.3% | 0.0074 / 0.0147 |
| 300 dpi, one per bull | 100.0% | 83.3% | 0.0067 / 0.0122 |
| 300 dpi, two per bull | 96.4% | 72.0% | 0.0082 / 0.0125 |
The hard cases.
| Case | Found | Strays | Baseline |
|---|---|---|---|
| Overlapping pairs | 48.2% | 0 | 47.3% |
| X marks over holes | 94.6% | 0 | 89.3% |
| Arrowheads, letter bowls | 98.2% | 53 | 87.5% |
| Holes on the ring's edge | 100.0% | 0 | 83.9% |
| Dark backing | 98.2% | 0 | 100.0% |
| Shots in the next cell | 98.2% | 0 | 91.1% |
Step 4 · 16 to 24 September · Real paper overrules
Every miss became a rule, with its reason
Synthetic sheets set the starting point. The developer's own range sheets then found what synthesis could not draw. The rule that came out of it: GroupLab may miss a hole, but it may never miss one silently. A missed shot is visible to the shooter who fired it; an invented one is not, so inventing is the worse mistake.
16 September
Do not cut real holes in two
What happened. The split point between one hole and two merged holes was set at an elongation of 1.45 from synthetic holes. On real paper, a hole joined to printed or photographed ink was cut in two, inventing a shot nobody could see.
The rule now. The split is at 1.80, where no real hole in the collection is split. A merged pair left whole is the louder failure: one mark about twice a hole's area, which the oversize flag reports.
src/GroupLab.Core/Detection/RenderDifferenceHoleDetector.cs21 September
A torn hole is still a hole
What happened. Fifteen shots fired, fourteen found, and an empty review queue. The fifteenth was seen, measured at 0.236 in, and refused for a solidity of 0.54 against a floor of 0.55. Lowering the floor would have let in handwriting at 0.53 to 0.41.
The rule now. A ragged mark is rescued only when it is the right size for the stated caliber, inside a bull's cell, and near the cell's middle. This is the defect that started the never-silent rule.
src/GroupLab.Core/Detection/TornHole.cs21 September
Holes are smaller than bullets
What happened. Five real .22 LR holes of 0.11 to 0.14 in were refused by a fixed 0.15 in floor. Paper stretches ahead of the bullet and closes behind it; the smallest measured 0.49 of its caliber.
The rule now. A stated caliber sets the floor at 0.45 of the caliber, a little below the worst seen so the next slightly worse hole is not refused. Without a caliber the old floor stands.
src/GroupLab.Core/Detection/HoleSizeGate.cs21 September
A shot that missed the bull still counts
What happened. Two holes just off the bull grid were refused by the position rule that had removed every false mark in the survey. Dropping the rule would trade two misses for an unknown number of inventions.
The rule now. A mark within one cell width of the grid is kept, marked as outside the grid, left unassigned and shown to the shooter. Far out in the margins is still refused.
src/GroupLab.Core/Detection/OutsideTheGrid.cs21 September
Half a hole at the edge of the picture
What happened. A hole cut by the edge of the scan has half the area, reads as about seven tenths the width, and was refused as a speck.
The rule now. For a mark touching the edge, the diameter comes from its extent along that edge, and it is marked partial because its center rests on less evidence.
src/GroupLab.Core/Detection/PartialHole.cs22 September
One hole or two, on photographs
What happened. Judged against the stated caliber, one 15-shot photograph had all fifteen holes flagged as possibly two shots.
The rule now. Each mark is compared with the sheet's own other marks instead. That brought fifteen flags to one, and left scans exactly as they were.
website/research/one-hole-or-two.md24 September
A center from the edge, not the shadow
What happened. A flatbed lights from one side, so a hole in card stock carries a shadow on the same side every time, and a center from the dark area is pulled toward it.
The rule now. Edges are found along rays from the center, each weighted by how sharp its crossing is, and a circle fitted to them. It measures alongside the detector's center until the comparison decides.
src/GroupLab.Core/Detection/HoleEdgeFit.cs
Step 5 · 28 September · Stress it
Shadows, glare, blur, curl
A rendered GL-CF25-LTR with one synthetic hole on each of its 25 scoring bulls, two seeds, each degraded one way and read by the whole pipeline as a photograph. A hole counts as found within 0.1 in. Synthetic holes, so an upper bound.
| Condition | Found, two seeds | False marks | Median error, in |
|---|---|---|---|
| Clean | 24, 25 | 0 | 0.006 to 0.007 |
| Hard shadow, lower third at 55% | 24, 25 | 0 | 0.006 to 0.007 |
| Soft shadow, a ramp to 55% | 23, 24 | 0 | 0.006 to 0.007 |
| A hand's shadow | 24, 25 | 0 | 0.006 to 0.007 |
| Dim, 45% light | 24, 25 | 0 | 0.006 to 0.007 |
| Uneven, a diagonal to 65% | 24, 24 | 0 | 0.006 to 0.007 |
| Glare, a bright ellipse | 22, 23 | 0 | 0.006 to 0.008 |
| Wave, 8 px, two across | 24, 24 | 0 | 0.007 to 0.008 |
| Blur, sigma 1.5 px | 25, 25 | 0 | 0.006 to 0.007 |
| Blur, sigma 3 px | 25, 25 | 0 | 0.008 to 0.009 |
| Noise, 12 levels | 25, 24 | 0 | 0.007 to 0.008 |
| JPEG quality 40 | 24, 25 | 0 | 0.007 |
| Curl, 15 px across | 0, 0 | 0 | did not register |
Curl: 0 of 25, and fixed the same day
A sheet bowed by 15 pixels across its width at 300 dpi, about 0.05 in, read all 38 of its markers and still did not register, because a homography, even with lens terms, is flat. That was geometry, not detection, and not something learning would fix.
The fix is a thin-plate spline through every marker corner: exact at each marker, bending no more than it must between them. A corner a misread marker put far from the rest is dropped and the spline refitted. The error a person is told is each marker as predicted by a mesh fitted without it. A test now requires at least 23 of 25 holes found on the curled sheet.
src/GroupLab.Core/Registration/MarkerMesh.cs
Why not machine learning?
Not yet, and here is the test for when
The failures measured so far are geometry (curl) and physics (two holes in one place), not a classifier that cannot tell a hole from a mark. A learned model also needs thousands of labeled holes across calibers, papers, lights and phones, with hundreds of the hard cases. GroupLab has about 400 distinct labeled holes, an estimate. So the plan is a scoreboard first, tuning second, and a model only if both run out.
NOW
A scoreboard
The degradations above, and later every target people choose to send, re-read by every build. Recall, false marks and center error by condition, so no step backward goes unseen. Fully explainable.
NEXT
Automatic tuning
The classical settings searched against the scoreboard, keeping the rule that recall holds when every threshold moves 30%. The same code with other constants, each with its reason.
ONLY IF NEEDED
A small learned classifier
Only accepting or refining candidates; markers, codes, geometry and every measurement stay classical. Worth it when tuning cannot move a failure, most likely overlaps, and a few thousand labeled holes exist.
Where it stands
What is met, and what is not
MET
Real commercial scan
25 of 27 hand-verified holes with no false marks on the reference scan, by the detector GroupLab's own grew from.
MET
Registration on paper
Ten of ten printed sheets inside the 0.005 in gate, worst bull 0.00319 in.
NOT YET
Synthetic gate
99 percent of holes with no false marks: one hole per bull is found completely with no strays, two per bull reaches 87.5 to 90.5 percent at today's split. Synthetic, so an upper bound.
NOT YET
Photographs
Flat photographs nearly pass the 0.005 in gate; mounted ones do not yet, and wait on photographs of mounted GroupLab sheets.
OPEN
Overlapping holes
The largest real failure. On eight photographs of a commercial grid target shot in tight five-shot groups, the detector built from the scans, for sheets GroupLab did not print, found 54 of 400 shots as marks of their own.
→ How GroupLab reads a target→ A bullet hole is not the bullet→ One hole or two?→ Curled, angled and wrinkled paper→ The detector on GitHub→ The full pipeline specification→ How GroupLab uses OpenCV→ Follow one target, stage by stage→ Every feature