03 · Software · data engineering · Jul 2026
Lead warehouse
A warehouse of 4,203,138 companies, 95.93% classified by niche for $27.93, so a new client can pull the exact companies in their market.
The decision
Map the tokens, not the companies
The obvious plan was to send 4,203,138 rows to a model. Before paying for that I looked at the data. The Google Maps category field is a comma-separated list, and split apart it collapses to 4,475 distinct tokens. I labelled the tokens instead of the companies, which settled most of the warehouse at no marginal cost. A model only saw the residue, and its answer was rejected unless the evidence appeared word for word in the input.[38] SourceLeadMasters/_warehouse LLM_CLASSIFY.md; engineering notes (company classification universe)As of2026-07-26StatusVerifiedAll sources →
What I built
- The ingest, rollup and classification pipeline in Node over Postgres.
- A two-tier model pass for the residue: a cheap model over everything, a better one re-asking only the uncertain band. 112,486 classifications in total.[41] SourceLLM_CLASSIFY.mdAs of2026-07-26StatusVerifiedAll sources →
- Structural guarantees: slugs are a strict schema enum, and evidence must appear verbatim in the input. 3,618 fabricated answers were rejected by that rule.[42] SourceLLM_CLASSIFY.md (evidence must appear verbatim in input)As of2026-07-26StatusVerifiedAll sources →
What was hard
It was slow, and it wasn't the model
Every "classification is slow" symptom traced to I/O. The only partial index covered rows that were already classified, exactly backwards for finding work. One index the right way round took the count from 13,709 ms to 702 ms.[39] Sourceengineering notes (company classification universe) (count(*) before idx_companies_unclassified)As of2026-07-25StatusVerifiedAll sources →
Paging that deadlocked
Hash-bucket paging fought the workers writing to the same table in a different order and died with a deadlock. Paging by primary-key range fixed it with zero retries.
What I learned
- Look at the shape of the data before you pay to process it.
- When something is slow, measure where the time goes before you change the expensive part.
- Store the model's below-threshold answers too, so the threshold is a policy you can change later instead of a decision baked in.
Results
- 95.93% of 4,203,138 companies classified.[30] SourceMeasured by DanielAs of2026-07-26StatusVerifiedAll sources →
- Total model spend $27.93.[31] SourceMeasured by DanielAs of2026-07-26StatusVerifiedAll sources →
- A person index of 2,875,646 rows the CRM searches for new client audiences.[33] SourceVPS provena_leads person_search_indexAs of2026-09-19StatusVerifiedAll sources →