GROUPLAB DOWNLOAD
More

Deep dive · Hole detection

How the hole detector was built

GroupLab's detector was not trained, and it was not guessed. It was measured into existence: 343 real hole measurements first, a written test it had to pass second, synthetic sheets to practice on third, and then every miss on real paper turned into a rule with its reason written beside it.

343

hole measurements on 15 scans of commercial targets, before GroupLab's detector was written

5

findings that changed the design: 4 assumptions overturned, one confirmed

3 to 9×

a hole is wider than any printed line, on every target in the survey

0

learned models: every decision is a rule you can read

The idea

Subtract what was printed

The photo, minus what was printed, leaves the holes.

A generic detector has to tell a bullet hole from a printed ring, a numeral, a barcode and a pen stroke. GroupLab printed the sheet, so it knows where every printed line is to a fraction of a millimeter. It draws that artwork again, lays it exactly over the photo through the markers, and subtracts it. Rings, numbers and bulls cancel out. What is left is everything that was not printed: the holes, and anything written on the sheet, which the definition's exclusion zones and the classifier then deal with.

This is called render-and-difference. The rest of this page is how its details were decided.

Step 1 · By 13 September · Measure first

343 holes, and five findings that changed the design

Before a line of GroupLab's detector was written, seventeen scanned files of fourteen commercial targets were measured by a Python and OpenCV harness: 343 hole measurements on 15 of them. Every number in the survey is labeled measured, estimated, eyeballed or not measured. The design had listed what a hole looks like. The survey checked each item.

DROPPED

A hole has a bright center

The design said every hole shows the white scanner lid through it. Measured, the center is on average darker than paper, and a quarter of holes have a center within 5 gray levels of the paper around them. No brightness cut separates holes from paper.

Core 53 levels darker than paper · 24% at paper level

DROPPED

A hole is round

A bullet hole in paper is a lobed star. Any roundness test throws away real holes in proportion to how strictly it is set, which is exactly why a filled-circle detector fails.

Rim radius varies by 0.48 of its mean

RAISED

The naive baseline is weak

Tuned for one picture, a Hough circle detector finds nearly every hole. The real problem is that its best setting changes from picture to picture, so the test had to change.

26 of 27 at one setting · 0 of 27 at another

SPLIT

Color tells a hole apart

Chroma works on color-printed targets and does nothing on black or gray ones; brightness is the reverse. On a grayscale target the ring and the hole are the same gray. The channel must be chosen for each picture, from a measurement.

Best separation 0.14 on the grayscale target

KEPT

Scale holds everywhere

Every printed feature on every target is at most 0.0567 in wide; every hole is 0.15 to 0.54 in across. The gap holds on every style, including the grayscale one, and survives any resolution because it is in inches.

Holes are 3 to 9 times wider than any printed line

What that left

Not brightness, not roundness, not color. Scale, measured in inches, is the primary test, and everything else refines it.

Widths on real targets, inches. An opening with a disk wider than the widest printed line erases all printed line work while every hole survives it. That is why one opening erases the printed lines of every style in the survey, whatever the artwork. Hand ink overlaps both, so it is a job for classification, not morphology.

Step 2 · 13 September · The test comes first

A gate that rewards stability, not luck

The original plan said the detector must beat a naive baseline of about 25 percent. The survey showed a tuned Hough circle detector finds 26 of 27 holes on the reference scan, but only at one setting: at an accumulator threshold of 25 it finds 26, at 40 it finds 10, at 55 it finds none. Naive methods are not bad at finding holes. They are bad at finding them without being tuned for each picture, which is useless in an app. So the gate was rewritten.

GateRequirementWhy
G1At least 95% of holes across all 15 files, with one set of settings and no per-image tuningBeats the tuned baseline while forbidding what makes it useless: tuning per picture.
G2At most one false mark per target, and none from the printed artworkRings, numerals and dots were already rejected on size; what remained was text, barcodes and handwriting.
G3Still above 90% when every threshold moves by 30% either wayThe test the naive methods fail. It is what separates a shippable detector from a lucky one.
G4The same settings pass at 93, 300 and 600 dpiProves no absolute gray-level threshold survived.
G5At least 95% on the grayscale targetThe hardest file: no color at all, ring and hole the same gray. If it works there it works anywhere.

Step 3 · 14 September · Practice on synthetic sheets

Holes drawn from the survey's own numbers

None of the seventeen scans was a GroupLab sheet, and render-and-difference needs one. So GroupLab draws its own shot sheets: the real render of GL-CF25-LTR, with holes whose shapes come from the survey. Development used seeds 1 to 3. Seeds 1001 to 1003 were kept aside and run once, at the end, so the result is not the product of tuning to the test. Every synthetic score on this page is an upper bound, not a prediction for paper.

What a synthetic hole is

A dark rim at a set radius, lobed by four patterns of narrow outward spikes, with 3 to 6 lobes around the rim, so it is a ragged star, not a circle. A rim band of measured width and darkness. A core where the paper is gone, showing the scanner lid or, in the dark-backing case, a board. A disturbed zone outside the rim that recovers to paper. Holes on printed ink get the different rim the survey measured there. Over the page: hand ink, a paper gradient, a 0.6 pixel blur and noise.

src/GroupLab.Core/Detection/SyntheticSheet.cs

Measured by the same baseline, on 229 real holes on bare paper. Calibration was limited to four rounds and the gap was published, not hidden: synthetic holes are 19 percent wider, 20 gray levels darker in the core and 19 percent less ragged, all of which make them easier. Every synthetic score is an upper bound.
How real they areRealSynthetic
Hull diameter, in0.2720.324
Core brightness, V195.6175.6
Darkest rim, V33.832.1
Raggedness, in0.04560.0370

Development, one change at a time.

Found within 0.15 in, 600 dpi, development seeds; the last column with the registration off by 0.02 in. Synthetic sheets, an upper bound, as first read on 14 September at a split of 1.45.
Change, and whyOne per bullTwo per bullRegistration off
The survey's opening, radius 0.032 in. Every miss was a thin rim around a pale center, erased by a disk wider than the rim.about 85%
Opening cut to radius 0.012 in. In a difference the artwork has already cancelled, so the opening only removes registration slivers. It broke registration tolerance instead.97.6%73.8%60.7%
Each bull cell aligned by phase correlation. The printed rings are fiducials too, so each cell lines itself up locally.97.6%73.8%92.9%
Merged neighbors split; oversized holes flagged. Single holes stretch to 1.26 to 1.37 at the 99th percentile (1.61 at worst); merged pairs from 1.50. The split point sits between them.97.6%95.2%92.9%

The held-out run, once.

Found within 0.15 in, as first read on 14 September at a split of 1.45. Median center error for render-and-difference against the baseline's. It kept every hole with the registration off by up to 0.08 in, and lost everything from 0.12 in, which says exactly how good registration must be. Re-recorded at today's split of 1.80: two per bull 87.5% at 600 dpi and 90.5% at 300 dpi; one per bull unchanged.
SheetsRender-diffBaselineMedian error, in
600 dpi, one per bull100.0%85.7%0.0063 / 0.0112
600 dpi, two per bull94.0%67.3%0.0074 / 0.0147
300 dpi, one per bull100.0%83.3%0.0067 / 0.0122
300 dpi, two per bull96.4%72.0%0.0082 / 0.0125

The hard cases.

As first read at a split of 1.45. Overlapping pairs are found about half the time, by either method. Arrowheads drawn in the cells made 53 strays at first, 50 with today's settings, which is why position and exclusion zones matter as much as shape. At today's split overlapping pairs read 47.3%.
CaseFoundStraysBaseline
Overlapping pairs48.2%047.3%
X marks over holes94.6%089.3%
Arrowheads, letter bowls98.2%5387.5%
Holes on the ring's edge100.0%083.9%
Dark backing98.2%0100.0%
Shots in the next cell98.2%091.1%

Step 4 · 16 to 24 September · Real paper overrules

Every miss became a rule, with its reason

Synthetic sheets set the starting point. The developer's own range sheets then found what synthesis could not draw. The rule that came out of it: GroupLab may miss a hole, but it may never miss one silently. A missed shot is visible to the shooter who fired it; an invented one is not, so inventing is the worse mistake.

  1. 16 September

    Do not cut real holes in two

    What happened. The split point between one hole and two merged holes was set at an elongation of 1.45 from synthetic holes. On real paper, a hole joined to printed or photographed ink was cut in two, inventing a shot nobody could see.

    The rule now. The split is at 1.80, where no real hole in the collection is split. A merged pair left whole is the louder failure: one mark about twice a hole's area, which the oversize flag reports.

    src/GroupLab.Core/Detection/RenderDifferenceHoleDetector.cs

  2. 21 September

    A torn hole is still a hole

    What happened. Fifteen shots fired, fourteen found, and an empty review queue. The fifteenth was seen, measured at 0.236 in, and refused for a solidity of 0.54 against a floor of 0.55. Lowering the floor would have let in handwriting at 0.53 to 0.41.

    The rule now. A ragged mark is rescued only when it is the right size for the stated caliber, inside a bull's cell, and near the cell's middle. This is the defect that started the never-silent rule.

    src/GroupLab.Core/Detection/TornHole.cs

  3. 21 September

    Holes are smaller than bullets

    What happened. Five real .22 LR holes of 0.11 to 0.14 in were refused by a fixed 0.15 in floor. Paper stretches ahead of the bullet and closes behind it; the smallest measured 0.49 of its caliber.

    The rule now. A stated caliber sets the floor at 0.45 of the caliber, a little below the worst seen so the next slightly worse hole is not refused. Without a caliber the old floor stands.

    src/GroupLab.Core/Detection/HoleSizeGate.cs

  4. 21 September

    A shot that missed the bull still counts

    What happened. Two holes just off the bull grid were refused by the position rule that had removed every false mark in the survey. Dropping the rule would trade two misses for an unknown number of inventions.

    The rule now. A mark within one cell width of the grid is kept, marked as outside the grid, left unassigned and shown to the shooter. Far out in the margins is still refused.

    src/GroupLab.Core/Detection/OutsideTheGrid.cs

  5. 21 September

    Half a hole at the edge of the picture

    What happened. A hole cut by the edge of the scan has half the area, reads as about seven tenths the width, and was refused as a speck.

    The rule now. For a mark touching the edge, the diameter comes from its extent along that edge, and it is marked partial because its center rests on less evidence.

    src/GroupLab.Core/Detection/PartialHole.cs

  6. 22 September

    One hole or two, on photographs

    What happened. Judged against the stated caliber, one 15-shot photograph had all fifteen holes flagged as possibly two shots.

    The rule now. Each mark is compared with the sheet's own other marks instead. That brought fifteen flags to one, and left scans exactly as they were.

    website/research/one-hole-or-two.md

  7. 24 September

    A center from the edge, not the shadow

    What happened. A flatbed lights from one side, so a hole in card stock carries a shadow on the same side every time, and a center from the dark area is pulled toward it.

    The rule now. Edges are found along rays from the center, each weighted by how sharp its crossing is, and a circle fitted to them. It measures alongside the detector's center until the comparison decides.

    src/GroupLab.Core/Detection/HoleEdgeFit.cs

Step 5 · 28 September · Stress it

Shadows, glare, blur, curl

A rendered GL-CF25-LTR with one synthetic hole on each of its 25 scoring bulls, two seeds, each degraded one way and read by the whole pipeline as a photograph. A hole counts as found within 0.1 in. Synthetic holes, so an upper bound.

ConditionFound, two seedsFalse marksMedian error, in
Clean24, 2500.006 to 0.007
Hard shadow, lower third at 55%24, 2500.006 to 0.007
Soft shadow, a ramp to 55%23, 2400.006 to 0.007
A hand's shadow24, 2500.006 to 0.007
Dim, 45% light24, 2500.006 to 0.007
Uneven, a diagonal to 65%24, 2400.006 to 0.007
Glare, a bright ellipse22, 2300.006 to 0.008
Wave, 8 px, two across24, 2400.007 to 0.008
Blur, sigma 1.5 px25, 2500.006 to 0.007
Blur, sigma 3 px25, 2500.008 to 0.009
Noise, 12 levels25, 2400.007 to 0.008
JPEG quality 4024, 2500.007
Curl, 15 px across0, 00did not register

Curl: 0 of 25, and fixed the same day

A sheet bowed by 15 pixels across its width at 300 dpi, about 0.05 in, read all 38 of its markers and still did not register, because a homography, even with lens terms, is flat. That was geometry, not detection, and not something learning would fix.

The fix is a thin-plate spline through every marker corner: exact at each marker, bending no more than it must between them. A corner a misread marker put far from the rest is dropped and the spline refitted. The error a person is told is each marker as predicted by a mesh fitted without it. A test now requires at least 23 of 25 holes found on the curled sheet.

src/GroupLab.Core/Registration/MarkerMesh.cs

Why not machine learning?

Not yet, and here is the test for when

The failures measured so far are geometry (curl) and physics (two holes in one place), not a classifier that cannot tell a hole from a mark. A learned model also needs thousands of labeled holes across calibers, papers, lights and phones, with hundreds of the hard cases. GroupLab has about 400 distinct labeled holes, an estimate. So the plan is a scoreboard first, tuning second, and a model only if both run out.

NOW

A scoreboard

The degradations above, and later every target people choose to send, re-read by every build. Recall, false marks and center error by condition, so no step backward goes unseen. Fully explainable.

NEXT

Automatic tuning

The classical settings searched against the scoreboard, keeping the rule that recall holds when every threshold moves 30%. The same code with other constants, each with its reason.

ONLY IF NEEDED

A small learned classifier

Only accepting or refining candidates; markers, codes, geometry and every measurement stay classical. Worth it when tuning cannot move a failure, most likely overlaps, and a few thousand labeled holes exist.

Where it stands

What is met, and what is not

MET

Real commercial scan

25 of 27 hand-verified holes with no false marks on the reference scan, by the detector GroupLab's own grew from.

MET

Registration on paper

Ten of ten printed sheets inside the 0.005 in gate, worst bull 0.00319 in.

NOT YET

Synthetic gate

99 percent of holes with no false marks: one hole per bull is found completely with no strays, two per bull reaches 87.5 to 90.5 percent at today's split. Synthetic, so an upper bound.

NOT YET

Photographs

Flat photographs nearly pass the 0.005 in gate; mounted ones do not yet, and wait on photographs of mounted GroupLab sheets.

OPEN

Overlapping holes

The largest real failure. On eight photographs of a commercial grid target shot in tight five-shot groups, the detector built from the scans, for sheets GroupLab did not print, found 54 of 400 shots as marks of their own.