Congruence

Overview

Congruence is a documentation OS for therapy practices of 5 to 25 clinicians. A session is recorded in the browser, uploaded, transcribed and turned into an insurance-ready progress note a clinician edits and signs. Supervisors and admins review in a dashboard, and insurance packets with ICD-10 suggestions come out of the same record.

It replaces the spreadsheets and PDFs most clinics run on with one structured, reviewable workflow: intake, recording, analysis and insurance, with scheduling, billing and an admin portal around them.

The production path is transcript-first: chunked parallel Whisper, one isolation-guarded gpt-4o-mini note, structured session facts and a 90-day clinical state. A tri-modal affect engine (face, voice, language) was built first and is still in the codebase, parked behind the fast path. Every number here is counted from the two repositories.

What I built

• Built the session-to-note pipeline: media download, ffmpeg audio extraction, whisper-1 transcription in 120-second chunks on a 6-thread pool with a local faster-whisper fallback, and a single gpt-4o-mini call that writes the progress note in JSON mode.
• Implemented session-fact extraction and clinical-state aggregation with FastAPI and Supabase, with local-first artifacts and retried database writes so a session is never lost.
• Supported 100+ patients with provider-specific note styles: clinicians upload a sample note and generation follows its format and tone.
• Built the LangChain copilot agent with 13 role-filtered tools and intent-routed prompts, and the tri-modal congruence engine (DeepFace, vocal and language emotion, 10 Hz cosine scoring) that ships parked behind the transcript-first path.
• Containerized the backend and shipped it to a DigitalOcean droplet with GitHub Actions, a disk-pressure guard and a monitoring script.

By the numbers, counted from the repo

Architecture

Pipeline

ffmpeg 16 kHz mono, OpenAI whisper-1 in 120 s chunks (faster-whisper fallback), gpt-4o-mini for notes and facts, gpt-3.5-turbo for per-segment affect, DeepFace + cosine TECS engine (parked)

API

FastAPI, 25 routes: process_session with background jobs and status polling, note-style upload (PyPDF2, python-docx), copilot chat and conversations, data and analytics reads

Data

Supabase Postgres via supabase-py: session facts, patient clinical state, note styles, copilot tables; local-first JSON and Markdown artifacts per session; 3 retries on upload

Product

React 18 + Vite + Tailwind, Supabase Auth with PKCE, 47 RLS tables, 4 private buckets, 15 Deno edge functions (analysis worker, forms, booking, billing, Gemini 2.5 Flash packets), Realtime

Infra

python:3.13-slim + ffmpeg on Docker Compose, one DigitalOcean droplet, GitHub Actions rsync deploy with env from secrets, health checks, retention and cleanup scripts; frontend on Vercel

How it's built

1. 01

   Ingestion and the worker

   Upload success is recorded before analysis is attempted, and the job survives restarts of the browser, the edge function or the backend.

   Durable upload and queue

   • The session video row gets status uploaded and analysis status queued, then an analysis job row with retry count, max retries and next retry time. Files over 64 MB use tus resumable uploads in 6 MB chunks with retry delays of 0, 1, 3, 5 and 10 s and fingerprint-based resume.
   • A retry resets both rows, so a failed analysis never requires re-uploading the recording.

   The edge worker

   • The analysis-worker edge function marks the job processing, mints a 4-hour signed URL, loads the clinician's active note style and posts it to the backend as a reference, then returns 200 immediately through EdgeRuntime.waitUntil.
   • It polls the backend job endpoint every 10 s for a window scaled to the recording length, between 30 and 90 minutes; failures reschedule at 60, 300, 900 and 3,600 s up to four attempts; success maps the response into the session analysis table and the workspace auto-opens it over Realtime.
2. 02

   Transcription

   Chunked, parallel, forgiving

   • The backend streams media to disk with a 1,800 s timeout in 8 KB chunks, extracts PCM 16 kHz mono with ffmpeg (8 kHz in fast mode), and reads the duration with ffprobe.
   • Audio is split into 120 s WAV chunks and sent to whisper-1 with verbose JSON and segment timestamps on a 6-thread pool; offsets are added per chunk and segments sorted. A 10 to 15 minute transcription dropped to 20 to 40 s.
   • A failed chunk is skipped with a warning instead of failing the job. Without an API key, local faster-whisper takes over: audio over 10 minutes is chunked, over 30 minutes drops to a smaller model, beam width 1 to 5 by mode.
3. 03

   The progress note, session facts and clinical state

   The note prompt is isolation-guarded. Emotion data and prior sessions are excluded on purpose so nothing contaminates what a clinician signs.

   One call, no inferred facts

   • One gpt-4o-mini call in JSON mode at temperature 0.2 with a 4,000-token cap produces identifying data, subjective, mental status exam and assessment-and-plan as narrative prose, plus a transcript summary of themes, events, tone and decisions.
   • Rules in the prompt: zero context outside the transcript, a fixed Not-discussed-in-this-session sentinel, a banned-hedging list (appears to, may, might), and zero tolerance for fabricated or inferred information. Invalid JSON yields no note rather than a malformed one.

   Note styles

   • Clinicians upload a sample note as PDF, DOCX or text; it is extracted, validated at 50 characters or more, scanned for SOAP, HPI and MSE markers, and classified by format and by tone from average sentence length.
   • A style-matched variant conditions generation on the first 2,000 characters at temperature 0.3 and falls back to the standard note on any error. One active style per clinician.

   Facts and state

   • A second gpt-4o-mini call at temperature 0.1 extracts only explicitly stated symptoms, interventions, homework, adherence, risk (suicide, self-harm, harm to others), stressors, progress markers and uncertainty, idempotent per session video.
   • Clinical state needs no model: the last 90 days of facts aggregate into the top 10 problems and top 5 stressors plus a risk history, upserted per patient. Every artifact is written locally first, the Supabase write retries three times with exponential backoff, and the ffprobe duration is cross-checked within 5 s.
4. 04

   The copilot agent

   Tools, roles, intent

   • A LangChain agent over gpt-4o-mini at temperature 0.1 with 13 tools filtered by role: clinicians and admins see 8, practice owners all 13. History persists in copilot tables with an in-memory fallback trimmed to 10 turns.
   • A keyword intent classifier picks one of three system prompts (evidence, summary, action) so answers stay predictable; evidence search is title-prioritized matching over themes, observations, risk fields and transcript segments.
   • Stub tools for claim status, scheduling and intake forms return pending-integration payloads instead of pretending; multiple patient-name matches return a disambiguation payload; an unknown role gets zero tools.
5. 05

   The affect engine, built and parked

   The first two months of the backend were a tri-modal pipeline. It still runs end to end in code; the orchestrator feeds it empty face and voice timelines today.

   Signals and fusion

   • ffmpeg frame extraction (1 FPS, later 0.3 and 0.1 FPS with a 10 to 20 frame cap), DeepFace 7-emotion classification with the OpenCV detector, per-frame failures normalized to neutral, a per-file vocal distribution, and an LLM 7-emotion score with valence, arousal and style per transcript segment.
   • Signals interpolate to 10 Hz; the congruence score is the mean of three pairwise cosine similarities across 7-dimensional vectors, on a 0 to 1 scale. A step is incongruent when valence signs disagree or the score is under 0.6, intensity is over 0.2 and the client is speaking, smoothed over a 7-step window; runs under 0.3 s are dropped and each gets a one-line LLM reason citing the three valences.
   • A simplified three-signal layer names five incongruence types (positive words with negative physiology, emotional flattening, smiling with a stressed voice, and so on) and detects cross-session repetition by correlating 20-point intensity signatures at 0.7 or above.
6. 06

   Platform and deployment

   Supabase as the product platform

   • 47 tables all under RLS with 195 policies; policy helpers (has_role, is_user_active, is_assigned_to_patient, is_super_admin) are SECURITY DEFINER SQL functions so a rule changes in one place. Invites are 32 random bytes with a 7-day expiry, single use; client-form links are stored only as a SHA-256 hash.
   • 15 Deno edge functions verify the caller's JWT with the anon client and then write with the service role; booking checks has_time_conflict() in SQL; insurance packets are written by Gemini 2.5 Flash with metrics passed as internal-only and missing facts bracketed.

   Deploy and operate

   • python:3.13-slim with ffmpeg on Docker Compose; a 30 s health check with a 40 s start period restarts the API. GitHub Actions rsyncs on push, writes the env from secrets, rebuilds with no cache and curls the health endpoint; a disk-pressure guard prunes Docker and keeps the newest 10 session directories.
   • A monitor script reports disk, containers, Docker usage, session count and age, memory, API health and the last cleanup; a cleanup script and a 7-day retention policy keep a small droplet healthy.

Decisions

1. 01

   No invented facts

   Anything the transcript doesn't support is rendered as Not discussed in this session, never guessed. Insurance audits punish confident gaps, and inferred prose eroded clinician trust faster than a visible one.

2. 02

   Provider-specific note styles

   Clinicians upload an example note and generation follows their structure and phrasing, so notes read like theirs.

3. 03

   Transcript isolation

   Emotion data and prior sessions are excluded from the note prompt. Tradeoff: the affect analysis the product is named for does not reach the clinical note yet.

4. 04

   Four-step chain to one call

   A sequential pipeline (extraction, emotion, synthesis, recommendations on gpt-4o) collapsed into one JSON call on gpt-4o-mini, removing 731 lines of prompts. Faster, cheaper, easier to validate.

5. 05

   Cloud Whisper over local

   The transcription API with parallel chunks took a 10 to 15 minute job to 20 to 40 seconds; local faster-whisper stays as the fallback when the key is missing or a call fails.

6. 06

   Async by default, local first

   Transcription and analysis run as background jobs with status polling, so a 50-minute session never blocks the UI. Every artifact is written to disk before the database, and every outcome is surfaced in the processing status.

7. 07

   Park vision, keep the code

   At 0.1 FPS the facial branch added minutes and little signal, so it was disabled in the orchestrator rather than deleted. It can return with better hardware and a clearer clinical use.

Edge cases the code handles

• Audio-only input takes a separate path and skips frame extraction entirely; iPhone footage gets rotation metadata reset and variable frame rate handling.
• A failed Whisper chunk degrades to a partial transcript; a missing API key falls back to local faster-whisper with a smaller model for long audio.
• Invalid LLM JSON yields no note rather than a malformed one; the pipeline continues and records the outcome in the processing status.
• A Supabase write that fails after three retries keeps the local artifacts and a flag so a backfill script can recover the session.
• Session directories use a lock-guarded counter plus an 8-hex UUID so concurrent sessions for one patient never collide; thread-safe singletons guard the database client and the agent.
• Note-style uploads reject .doc files, image-only PDFs and anything under 50 characters, and warn above 50 KB.
• The booking conflict check runs in the edge function before insert, not as a database exclusion constraint; a simultaneous double-booking is a known gap.