History

James 6ba57df6ae refactor: clean up reference data and remove rate limiting Reference data simplification (choke point pattern): - Remove RefSave/RefDelete from lib (import-time only, not runtime) - Remove LabTestSave, LabRefSave from lib/lab_reference.go - Remove PopulateReferences (LLM-based ref generation) - Keep only RefQuery() for runtime reads - Import tools handle their own SQL inserts Rate limiting removed: - Delete new_signups table and all rate limit code - Solved via different approach (not in codebase) Database consolidation (on staging): - Moved genotypes table (30K SNPs) to reference.db - Deleted empty DBs: portal.db, rate_limit.db, snpedia.db, ratelimit.db Net -293 lines. Runtime code now only reads reference data via RefQuery(). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>	2026-02-11 01:20:18 -05:00
..
README.md	Initial commit	2026-02-01 02:43:27 -05:00
main.go	refactor: clean up reference data and remove rate limiting	2026-02-11 01:20:18 -05:00

James 6ba57df6ae refactor: clean up reference data and remove rate limiting

Reference data simplification (choke point pattern):
- Remove RefSave/RefDelete from lib (import-time only, not runtime)
- Remove LabTestSave*, LabRefSave* from lib/lab_reference.go
- Remove PopulateReferences (LLM-based ref generation)
- Keep only RefQuery() for runtime reads
- Import tools handle their own SQL inserts

Rate limiting removed:
- Delete new_signups table and all rate limit code
- Solved via different approach (not in codebase)

Database consolidation (on staging):
- Moved genotypes table (30K SNPs) to reference.db
- Deleted empty DBs: portal.db, rate_limit.db, snpedia.db, ratelimit.db

Net -293 lines. Runtime code now only reads reference data via RefQuery().

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

2026-02-11 01:20:18 -05:00

README.md

Initial commit

2026-02-01 02:43:27 -05:00

main.go

refactor: clean up reference data and remove rate limiting

2026-02-11 01:20:18 -05:00

README.md

import-genome

Fast genetic data importer using lib.Save() for direct database access.

Performance

~1.5 seconds to:

Read 18MB file
Parse 674,160 variants
Sort by rsid
Match against 9,403 SNPedia rsids
Insert 5,382 entries via lib.Save()

Installation

cd ~/dev/inou
make import-genome

Usage

import-genome <plain-file> <dossier-id>

# Help
import-genome --help

Supported Formats

Format	Delimiter	Columns	Alleles
AncestryDNA	Tab	5	Split
23andMe	Tab	4	Combined
MyHeritage	CSV+Quotes	4	Combined
FTDNA	CSV	4	Combined

Auto-detected from file structure.

Data Model

Creates hierarchical entries:

Parent (genome/extraction):
  id: 3b38234f2b0f7ee6
  data: {"source": "ancestry", "variants": 5381}

Children (genome/variant):
  parent_id: 3b38234f2b0f7ee6
  type: rs1801133 (rsid)
  value: TT (genotype)

Databases

SNPedia reference: ~/dev/inou/snpedia-genotypes/genotypes.db (read-only, direct SQL)
Entries: via lib.Save() to /tank/inou/data/inou.db (single transaction)

Algorithm

Read plain-text genome file
Auto-detect format from first data line
Parse all variants (rsid + genotype)
Sort by rsid
Load SNPedia rsid set into memory
Match user variants against SNPedia (O(1) lookup)
Delete existing genome entries for dossier
Build []lib.Entry slice
lib.Save() - single transaction with prepared statements

Example

./bin/import-genome /path/to/ancestry.txt 3b38234f2b0f7ee6

# Output:
# Phase 1 - Read: 24ms (18320431 bytes)
# Detected format: ancestry
# Phase 2 - Parse: 162ms (674160 variants)
# Phase 3 - Sort: 306ms
# Phase 4 - Load SNPedia: 47ms (9403 rsids)
# Phase 5 - Match & normalize: 40ms (5381 matched)
# Phase 6 - Init & delete existing: 15ms
# Phase 7 - Build entries: 8ms (5382 entries)
# Phase 8 - lib.Save: 850ms (5382 entries saved)
#
# TOTAL: 1.5s
# Parent ID: c286564f3195445a