Commit Graph

32 Commits

Author SHA1 Message Date
James 20c33d2fb2 Document processing pipeline: OpenRouter OCR + Stepfun extraction
- Switch OCR from Fireworks Qwen to Gemini 3 Flash via OpenRouter
- Switch extraction/translation to Stepfun Step 3.5 Flash via OpenRouter
- Add CallOpenRouter() to lib/llm.go (OpenAI-compatible API)
- Add OpenRouter API key support to lib/config.go
- Fix extraction preamble causing Stepfun reasoning model to return null content
  (heavy user-message preamble → short system message)
- Fix timestamp fallback: no date = 0, not today
- Fix document type: use file extension instead of hardcoded "pdf"
- Fix findUploadByFilename: Value filter was silently ignored, returning all uploads
- Fix entryQuery: skip category filter when EntryID is specified
- Fix extraction prompt contamination: replace real patient examples with generic ones
- Add reprocess button (↻) for document uploads
- Add delete cascade for upload entries (removes created document/extraction entries)
- Add upload/reprocess/delete logging throughout pipeline
- Consolidated templates, translations, email templates, landing pages
- Portal: defense, genome, dossier sections, MCP, OAuth updates
- Tools: translate pipeline improvements

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 05:28:05 -04:00
James a2141bb5d3 Checkpoint: all pending changes across lib, portal, api, tools
Extraction prompts refined, dossier sections expanded, MCP tools
enhanced, genome/oauth/upload improvements, health-poller added,
import-genome removed, landing/pricing/dashboard template updates,
carousel images, consent/docs templates, rquery/dbquery tools,
CLAUDE.md and docs updates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 23:37:44 -04:00
James 155d24ec2e Add comprehensive test suite: labs, entries, dossiers, readings
Test coverage:
- lib/lib_test.go (36 test functions):
  - Crypto: Pack/Unpack, CryptoEncrypt/Decrypt, deterministic encryption
  - Tokens: Create, Parse, expiration handling, invalid token handling
  - Lab references: LabScale conversions, MakeRefID, LabRefLookup with sex/age matching
  - Categories: FromString, CategoryName, CategoryKey, Categories() list
  - IDs: NewID uniqueness, FormatID/ParseID roundtrip
  - Entry CRUD: Write, Read, Delete, filter by type, filter by date range
  - Trackers: Add, Query, Respond, Dismiss
  - RBAC: CheckAccess for own dossier, system user, granted access, no access
  - Dossier helpers: DossierFromEntry, SexKey
  - Normalize: normalizeKey test name cleaning
  - JSON marshaling: Tracker data, DossierPreferences

- api/api_test.go (23 test functions):
  - Helper functions: intPtr, getString, seriesPriority
  - v2 helpers: deterministicID, metricLabel
  - Version endpoint
  - Error/JSON responses
  - Request/Response JSON marshaling: Entry, Dossier, Tracker, Parse, Readings
  - Method handling: token endpoint
  - Categories endpoint
  - isLocalhost detection
  - Schedule calculation

All tests use in-memory SQLite for isolation.
No external dependencies required.
2026-02-28 07:20:44 -05:00
James 93643d285b Code review 2026-02-28: fix critical/high findings + full review report
Critical fixes:
- CR-001: Remove auth backdoor code 250365 (lib/dbcore.go)

High fixes:
- HI-001: Fix CORS wildcard to use origin allowlist (portal/api_mobile.go, portal/mcp_http.go)
- HI-002: Fix LOINC skip logic - skip only if BOTH SearchKey2 AND LOINC are set (lib/normalize.go)

Also added:
- Full code review report at docs/CODE-REVIEW-2026-02-28.md

14 issues found: 2 critical, 4 high, 5 medium, 3 low
3 fixes applied, remaining are documented for follow-up
2026-02-28 07:20:38 -05:00
James 033d558535 Portal: upload handler, dossier rework, dashboard updates, normalize fixes
- portal/upload.go: new file upload handler (360 lines)
- portal/templates/dossier.tmpl: major rework (469 changes)
- portal/templates/upload.tmpl, dashboard.tmpl: UI updates
- lib/normalize.go, llm.go, config.go: library fixes
- portal/dossier_sections.go, main.go: portal logic
- portal/lang/en.yaml: string updates
- cmd/populate-search-key: search key population tool
- Makefile, style.css: build and style updates
2026-02-25 20:01:11 -05:00
James cc1dd7690c Lab reference charts, import tracking, DossierFromEntry consolidation
- Fix lab chart reference bands: parse DOB in DossierFromEntry, generate
  deterministic ref_ids in import-caliper (was collapsing 4363 rows to 1)
- Consolidate DossierFromEntry into lib/dbcore.go (eliminate portal duplicate)
- Add Import field to entries for batch undo (NextImportID, all import paths)
- MyChart direct JSON parsing (skip Gemini for structured lab data)
- Multi-order extraction from markdown/text tables
- Normalize progress callback for UI feedback
- DICOM import, genome import, API, portal, MCP, translation updates
- Remove test DICOM data from repo

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 05:15:03 -05:00
James d5133fd56f WIP: DICOM import improvements and database query optimizations 2026-02-14 18:00:30 -05:00
James 75e9ec7722 Refactor: Remove legacy v2, data, roles modules; update auth, queries, translations, portal MCP tools 2026-02-13 15:30:22 -05:00
James 6ba57df6ae refactor: clean up reference data and remove rate limiting
Reference data simplification (choke point pattern):
- Remove RefSave/RefDelete from lib (import-time only, not runtime)
- Remove LabTestSave*, LabRefSave* from lib/lab_reference.go
- Remove PopulateReferences (LLM-based ref generation)
- Keep only RefQuery() for runtime reads
- Import tools handle their own SQL inserts

Rate limiting removed:
- Delete new_signups table and all rate limit code
- Solved via different approach (not in codebase)

Database consolidation (on staging):
- Moved genotypes table (30K SNPs) to reference.db
- Deleted empty DBs: portal.db, rate_limit.db, snpedia.db, ratelimit.db

Net -293 lines. Runtime code now only reads reference data via RefQuery().

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-11 01:20:18 -05:00
James 6486a52ad9 refactor: complete RBAC redesign with hierarchical permissions
Simplify access control from 500+ lines to ~50 lines of core logic:
- New permission bitmask (PermRead/Write/Delete/Manage)
- Hierarchical access (dossier → category → entry)
- Single choke points: CheckAccess(), EntryQuery(), DossierQuery()
- All data access now enforced through lib RBAC layer
- Removed complex role templates and permission caching

Also improved NewID() to use UUID v4 + SHA-256 hash for better
randomness distribution (was limited to 0-7 hex start).

Net -210 lines across 28 files. Ready for staging deployment.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-11 00:06:02 -05:00
James e1b40ab872 refactor: unexport raw DB functions, enforce RBAC at data layer
Rename Query→dbQuery, Save→dbSave, Load→dbLoad, Delete→dbDelete,
Count→dbCount in lib/db_queries.go. Go compiler now prevents any code
outside lib/ from bypassing RBAC checks.

All external callers migrated to RBAC-checked functions:
- EntryCategoryCounts, EntryCount, EntryListByDossier (new)
- LabTestList, LabEntryListForIndex, LabRefLookupAll (new)
- GenomeQuery now requires AccessContext
- EntryDeleteByCategory/EntryDeleteTree now require AccessContext

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-10 02:34:51 -05:00
James 77db02a6eb feat: optimize genome queries with IN clauses, dedup, repute filter
- Replace N separate SQL queries with single IN clause for rsids and genes
- Dedup results by rsid, merging categories from multiple tiers
- Add repute filter (Good/Bad/Clear) to genome queries
- Expose limit/offset as MCP parameters
- Add genotype to search check
- Fix category filter in genomeEntriesToResult
- Remove deprecated api/api_categories.go and api/api_genome.go
- Change GenomeMatch to use Categories []string instead of Category+Subcategory

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-10 02:13:18 -05:00
James d2d77d1503 refactor: move genome query to lib, add sensitive variant redaction
- Move GenomeQuery logic from api/api_genome.go to lib/v2.go so MCP
  handler calls lib directly instead of HTTP round-trip (fixes 403 on
  genome queries via Claude.ai MCP - was hitting RBAC table mismatch)
- Generate CategoryFromString from categoryNames in init() (single
  source of truth, removes 9 unused aliases)
- Redact sensitive variants (Bad repute, magnitude >4) in targeted
  queries: genotype/summary replaced with "hidden" + hint to use
  include_hidden=true. Broad queries still suppress entirely.
- API handler is now a thin wrapper parsing query params

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-09 17:23:25 -05:00
James c88c50bc12 Revert 'fix: add legacy genome data format support to API' - investigating actual issue 2026-02-09 14:04:50 -05:00
James 0ac8f0f42d fix: add legacy genome data format support to API
Root cause: Sophia's genome data was stored in legacy flat format where
all variants were top-level category 4 entries with rsID as the type
field. The API expected new hierarchical format (extraction → tier →
variant), causing "no genome data" errors.

Changes:
- GenomeGetExtraction(): Detect legacy data, return synthetic extraction
- GenomeGetTiers(): Return synthetic "all" tier for legacy format
- GenomeGetVariants(): Query top-level entries for legacy format

Test results:
- Sophia's 4960 genome variants now accessible via /api/genome
- Gene queries work: ?gene=MTHFR,COMT,MTR,MTRR returns 5 matches
- MCP tools (query_genome) now function correctly

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-09 12:21:08 -05:00
James 131a41037b fix: update /prompts/respond route to /trackers/respond 2026-02-09 05:21:35 -05:00
James 96fec23e22 refactor: rename prompt to tracker everywhere
- Rename prompts table to trackers
- Rename all Prompt types/functions to Tracker
- Rename prompt_id to tracker_id throughout
- Rename API endpoints /api/prompts -> /api/trackers
- Rename URL paths /dossier/{id}/prompts -> /dossier/{id}/trackers
- Rename template files and references
- Add migration script for schema changes
- Next: implement self-contained entries with metadata
2026-02-09 02:05:17 -05:00
James 9781b31c7d feat: fix year interpretation, new schedule format, exclude today from backfill 2026-02-09 02:00:18 -05:00
James 3014f21d72 refactor: prompts UI and LLM API cleanup 2026-02-08 08:30:27 -05:00
James 35e9e2a84b feat: add Terms of Service page and legal page updates
- Add /legal/terms with comprehensive ToS content
- Add terms link to footer navigation
- Add /legal/terms to defense.go whitelist for external access
- Update privacy policy and DPA templates with improved styling
- Refactor RBAC editor template formatting
- Add prompts AI setup documentation
- Include database migration scripts

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-08 04:59:59 -05:00
James 7192f39bc1 fix: remove backward compat, migrate old access to proper RBAC grants
Removed the backward compatibility fallback that checked the old
dossier_access table from CanManageDossier/CanAccessDossier - it was
a security risk (hidden path that bypassed the new RBAC system).

Instead, added MigrateOldAccess() that converts existing dossier_access
entries to proper access grants on startup (idempotent - skips existing).

Migration rules:
- Self-references (accessor == target) skipped (owner access is automatic)
- can_edit = 1 → "rwdm" root grant
- can_edit = 0 → "r" root grant
- Role set to "Migrated" for tracking

Result: 12 grants migrated from old table.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-07 23:27:23 -05:00
James c3b5381c4c fix: add backward compatibility for old dossier_access table
RBAC editor was failing with 403 Forbidden when trying to edit permissions
for users who have access via the old dossier_access table but not the new
access grants table.

Added fallback logic to CanManageDossier and CanAccessDossier:
1. Check new RBAC system (access table) first
2. If no grant found, check old dossier_access table
3. For manage: check can_edit = 1
4. For access: check status = 1

This allows existing access relationships to work with the new RBAC editor
while we migrate data from old to new system.

Fixes: "Forbidden" error when editing permissions for legacy access grants

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-07 18:24:27 -05:00
James b684612797 security: use 16-char hex system accessor ID from .env
Replaced hardcoded "system-internal" with proper 16-char hex ID loaded
from environment, matching dossier ID format. Created actual dossier
for system accessor with name "System".

Changes:
- Generated random 16-char hex: 7b3a3ee1c2776dcd
- Added SYSTEM_ACCESSOR_ID to anthropic.env (staging & production)
- Created dossier for system accessor (name: "System", email: "system@internal")
- Load SystemAccessorID from config in ConfigInit()
- Initialize SystemContext after config load with proper ID
- Default fallback value in config.go

Benefits:
- Proper 16-char hex format matches all other dossier IDs
- Won't break code expecting 16-char IDs
- System operations show as "System" in audit logs
- Can be changed via .env without code changes
- Has actual dossier entry for referential integrity

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-07 17:25:10 -05:00
James 45a6445c3b security: replace empty string bypass with explicit system accessor ID
Changed from empty accessorID bypassing checks to explicit SystemAccessorID
for better security and audit trail.

Before: accessorID == "" → bypass all checks (security risk)
After: accessorID == "system-internal" → bypass (explicit, auditable)

Changes:
- Added SystemAccessorID constant = "system-internal"
- Updated SystemContext to use SystemAccessorID
- Updated checkAccess() to check for specific ID
- Updated accessorIDFromContext() to return SystemAccessorID
- Updated all EntryList calls to use SystemAccessorID
- Updated auth.go helpers to use SystemAccessorID

Benefits:
- Explicit backdoor ID visible in audit logs
- No accidental bypass from empty strings
- Clear intent for system operations
- Can't collide with real hex dossier IDs (uses "system" prefix)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-07 17:20:24 -05:00
James 86e72b4f28 refactor: simplify RBAC to use accessorID string parameter
Complete refactor from AccessContext struct to simple accessorID string
parameter for RBAC enforcement, as requested. All access control remains
in lib layer - API/Portal just pass accessor + dossier to lib functions.

Changes:
- Added accessorIDFromContext() helper in lib/v2.go
- Updated all checkAccess() calls to extract accessorID from context
- Updated all EntryList() calls (nil → "" for system context)
- Fixed auth.go helper functions to extract accessorID
- Updated categories API to pass accessor through to lib

All RBAC enforcement stays in lib - no API-level access checks.
Empty accessorID bypasses checks (system/internal operations).

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-07 17:15:09 -05:00
James d5be120058 refactor: enforce RBAC only in lib layer
- Remove API-level access checks (requireDossierAccess)
- Pass user context to lib functions instead of system context
- Single enforcement point: lib.EntryList/EntryGet/etc check access
- Fixes EnsureCategoryEntry to use EntryWrite (correct function name)

All access control now happens at the lowest level in lib.
API and MCP layers just pass context through.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-07 17:05:05 -05:00
James 8ccab9581d feat: add RBAC helper functions for granting access
- EnsureCategoryEntry: creates category entry if needed
- GrantAccess: creates access grant with cache invalidation
- RevokeAccess: removes grant with cache invalidation

Category entries are automatically created when granting category-level access.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-07 17:02:31 -05:00
James f7e6c32e30 refactor: simplify RBAC - categories are entries
- Remove special cat:{id} handling from permission resolution
- Categories are now just entries with parent_id=""
- Access flows naturally through parent_id chain hierarchy
- Three levels: root (entry_id="") > categories > individual entries
- Explicit denial supported with ops=""
- Updated documentation to reflect cleaner model

Next: deprecate dossier_access table, migrate to access grants

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-07 17:01:59 -05:00
James c1cd76559d fix: genome query RBAC - use system context for data access
- Updated all genome functions to accept AccessContext parameter
- GenomeGetExtraction, GenomeGetTiers, GenomeGetTierByCategory,
  GenomeGetVariants, GenomeGetVariantsByTier now pass context to EntryList
- API genome handler uses system context after dossier access check
- Categories endpoint uses system context for counting operations
- Fixes MCP query_genome returning 403/no data errors

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-07 16:31:17 -05:00
Johan 9190ca1443 Merge remote 'Initial commit from dev' with local master
Conflicts resolved:
- soc2 docs: used remote (updated versions)
- go.mod/go.sum: kept local (full dependencies)
- lib/*.go: kept local (production FIPS, no hardcoded keys)
- .gitignore: kept local (comprehensive)
- test/*.sh: kept local (executable permissions)

Includes: Flutter app, design system, templates, static assets
2026-02-01 04:00:45 -05:00
Johan Jongsma dd160f8451 Initial commit from dev 2026-02-01 08:03:12 +00:00
Johan 94946baf00 Initial commit 2026-02-01 02:43:27 -05:00