Aaveg Shangari — SWE · AI/ML

Work

What I've built and shipped

Ordered by what best shows what I can do. The top entries have expandable case studies covering why they exist, the architectural decisions behind them, and the tradeoffs I knowingly made.

Apr 2025 — Present / Founder & Solo Developer

Full-Stack ML

LIVE AT SAVVANT.CA

Savvant

Canadian personal finance on live bank data. Links a user's accounts through Plaid, pulls transactions, debts, and balances continuously, and layers per-user machine learning, anomaly detection, forecasting, and a chatbot whose numbers are never hallucinated.

Production Plaid integration: Canadian banks syncing live with Fernet-encrypted access tokens, verified ES256 JWT webhooks, and cursor-based sync isolated per row so one bad transaction can never wedge the pipeline.
Hybrid chatbot where the LLM never does the math. GPT-5.4-mini classifies intent and formats answers; a deterministic Python engine computes every number from real data.
Per-user ML with disciplined ground truth: only explicit user confirmations train the model, uncertain predictions route to a verification queue, and every correction retrains in the background.
Double-count-proof by design: in-place pending-to-posted promotion and a typed transaction taxonomy keep spend totals correct across chequing and credit accounts.

FastAPI · PostgreSQL · React · Plaid · LightGBM · IsolationForest · Prophet · GPT-5.4-mini · Railway · Vercel · Cloudflare

Bank sync

Plaid · verified webhooks · live

Categorization

One model per user · retrains on every correction

Anomaly detection

IsolationForest · 4 features

user question
└→ LLM intent classification
└→ Python calculation engine
└→ LLM formatting → trustworthy answer

Why it exists

A bank app tells you where money went, a category total and a pie chart, but not the patterns behind it. Savvant links a user's Canadian bank accounts through Plaid, keeps transactions, debts, and balances in continuous sync, and answers the pattern question: behavioral spending tags with dollar-quantified tips, anomaly flags with plain-language reasons, subscription tracking that notices price creep, and a month-over-month view of what actually changed.

The project began as a privacy-first PDF statement parser and pivoted to Plaid bank aggregation as the primary data source: continuous, webhook-driven sync replaced monthly uploads, and the PDF path was retired behind a feature flag. Killing my own original architecture when the better design became clear was part of the education.

The other motivation was personal: proving I could take an ML system all the way to production alone. Real third-party bank integration, per-user modeling, API design, frontend, auth, deployment, and the unglamorous edge cases in between, like pending transactions that change their ID when they post.

Architecture decisions

LLM classifies, Python calculates GPT-5.4-mini + deterministic engine

LLMs produce confident but wrong numbers. In a finance product that is disqualifying. So the LLM only maps a question to an intent and formats the final answer. Every dollar figure comes from a deterministic Python engine querying real data, which keeps outputs trustworthy and auditable.

Guarded SQL for freeform questions validated read-only SELECT

Structured intents cannot cover every question ("how much at this merchant in the last 3 months?"). The chatbot generates SQL, but it only executes if it is a SELECT, touches only the transactions table, and scopes to the user's own user_id as a bound parameter. Write operations are rejected, results are capped, and parameterization eliminates injection.

One model per user, features beyond text char n-gram TF-IDF + amount, recurrence, day-of-month via ColumnTransformer → LightGBM

Merchant strings are messy and personal ("TIMHORTONS#1985MARKHAM"). Character n-grams handle abbreviations that word tokens miss, but text alone cannot separate every Apple charge arriving as just "Apple", so amount, recurring-series linkage, and day of month carry the separation text cannot. The model activates at 20 qualifying labels; below that, keyword rules take over. Retraining runs as a background task after every confirmation.

Ground truth discipline: silence is not consent verification queue · retroactive propagation · ambiguity demotion

Only explicit approvals and corrections train the model; ignored queue items train nothing. Confident predictions apply silently, uncertain ones surface in a small queue that never blocks or nags. Correcting a merchant offers to recategorize its history in one tap, and a merchant whose confirmed corrections span multiple categories is demoted to ambiguous: no more propagation, every transaction classified individually. The training set stays honest by construction.

Strict categorization priority chain user override → overrides table → ML above threshold → rules → uncategorized

User corrections are ground truth and always win. Remembered past corrections (keyed on normalized merchant, skipped for merchants marked ambiguous) come next, then the ML prediction but only above a confidence threshold, then keyword rules, then "Uncategorized". Predictable resolution order means the system never silently overrides a human.

Pending-to-posted promotion Plaid issues a new transaction id on posting

Plaid frequently gives a transaction a brand-new id when it moves from pending to posted, which naively creates a duplicate and double-counts the spend. The normalizer detects this via pending_transaction_id and promotes the existing row in place: new id, refreshed fields, user-corrected categories preserved. Historical duplicates in production were cleared with a cleanup script that supports a dry run before --confirm, executed inside the Railway container.

Transaction taxonomy vs. double-counting purchase · fee · debt_payment · income · cc_payment · refund · transfer

With both a chequing and a credit account linked, the card's bill payment appears on one side and the purchases on the other; naive summing double-counts spending. Typed transactions exclude cc_payments and transfers from spend totals, a transfer into savings maps to transfer rather than income, and cash flow returns null when it genuinely cannot be determined instead of pretending to know.

Subscriptions are an attribute, not a category recurring_series · cadence detection · permanent user overrides

An Oura membership is Health and a subscription, so "Subscriptions" was removed as a spending category and rebuilt as a cross-cutting recurring-series engine: cadence detection with tolerance, amount history, and next-expected dates. User assertions are permanent ground truth the detector can never flip, and a forced subscription that stops charging surfaces a staleness card. Totals never change from a heuristic alone, only from explicit user action or real transaction data.

Money is Numeric, never float signed amounts: spending negative, income positive

Floating point drift is unacceptable in financial sums. Plaid's sign convention (positive = outflow) is flipped once at normalization so every downstream calculation reads one consistent scheme, and money is stored as Numeric everywhere.

Tradeoffs & known limits

ML models live on ephemeral disk. A redeploy wipes them. Because per-user retraining is fast and fully automatic, the practical cost is a brief cold start, not data loss. Database-backed model storage is the planned fix.
Tests run against the real dev database. The test client executes the actual startup lifespan, which once leaked a Postgres advisory lock and hung the suite with nine zombie connections. Lock acquisition now has a timeout and the suite enforces a global timeout, but a test-scoped database is the correct eventual fix.
The verification loop's acceptance test is lived, not unit-tested. The queue badge should trend toward empty over weeks of normal use as the model learns. If it doesn't, the diagnosis path is labels reaching training, features separating merchants, then the confidence threshold.
Investments are a read-through only. Nothing is modeled or persisted. Deliberately deferred rather than half-built.

Sep 2024 — Present / Research Software Engineer

AI/ML Computer Vision Full-Stack

ParaTracker

Visual Computing Lab · View repo ↗

A full-stack tool that tracks C. elegans worms in microscopy video, holds their identity across frames, and turns skeleton-based motion into cross-video metrics a biology lab uses to compare experimental conditions.

Trained a custom YOLOv8-seg model for worm instance segmentation, cutting false-split detections roughly 5x versus the classical CV pipeline on translucent, self-overlapping worms.
Fixed the training set, not the model: a silent RLE-mask bug was starving YOLO of data. Decoding those masks took validation mask mAP50 from ~0.50 to ~0.87.
Built the tracking pipeline and the metrics layer: skeletonization plus Hungarian ID matching, then per-worm head/tail/midbody motion aggregated into a cross-video comparison API and Recharts UI.
Killed a concurrency bug that broke the sequential job queue, by moving the worker into FastAPI lifespan so it runs in one process instead of two.

Python · PyTorch · YOLOv8-seg · OpenCV · scikit-image · FastAPI · React · Recharts · SQLite

Validation mask mAP50

0.50 → 0.87 after RLE fix

Unique IDs per video

62 classical → 12 YOLO

Mean track length

108 → 254 frames

video upload
└→ YOLOv8-seg worm masks
  └→ skeletonize keypoints
    └→ Hungarian match track IDs
      └→ motion metrics head/tail/midbody
        └→ cross-video compare

Why it exists

A biology lab needed to quantify how C. elegans worms move (head, tail, and midbody) across many videos and experimental conditions. The worms are translucent, fast, and constantly overlap themselves, which is exactly where a classical CV pipeline breaks: it splits one worm into several false detections and loses identity the moment two worms touch. Doing the analysis by hand does not scale, and any tool the lab depends on has to produce numbers a biologist could defend in a paper.

So the job had two halves: get detection and tracking accurate enough to trust, and turn that into metrics a non-technical researcher can run and compare themselves. I built the tracking pipeline, the YOLO segmentation, the metrics and aggregation layer, and the React UI and charts on top of it.

Architecture decisions

YOLOv8-seg instead of pushing classical CV harder custom instance segmentation, 149 annotated frames

The classical thresholding pipeline over-fragmented translucent worms and could not hold identity through overlap. A learned segmentation model was the right tool for that specific failure. On a real video, unique IDs dropped from 62 (classical) to 12 (YOLO) with mean track length more than doubling, so identity held far longer. YOLOv8-seg specifically because it trains and runs on modest hardware (a 4GB laptop GPU during early work).

The accuracy win was data, not model size RLE mask decode bug

Training stalled around 0.50 mAP. The cause was not the model: the COCO-to-YOLO converter was silently dropping every RLE-format mask and keeping only polygons, quietly gutting the training set. Decoding the RLE masks properly took validation mask mAP50 to ~0.87 by the first epoch. With annotation count as the real bottleneck, more data beat any change to the model.

Refuse identity claims you cannot verify edge-truncation of partial worms

When a worm touches the frame edge and leaves view, you cannot prove a worm re-entering later is the same one. Rather than fabricate that identity, the pipeline truncates the track at first edge-touch and filters on the truncated length. Choosing an honest data gap over a confident guess is the correct default for a scientific instrument.

Never blend non-comparable data into one number per-pipeline split, auditable grouping

Cross-video comparison keys on an explicit, user-confirmed list of job IDs, never on a loose keyword match, and never averages classical and YOLO results together. Aggregation dedups on (filename, pipeline) so the same video run on both pipelines is not silently collapsed. The boundary of what goes into a number is auditable, which is what makes the output trustworthy.

Worker in FastAPI lifespan, not module scope the real concurrency fix

The sequential job queue was intermittently processing two videos at once, and cancels were not stopping jobs. Root cause, found by logging PIDs at runtime: under the reload server the worker was starting at module top level, so it ran in both the reloader and the serving process. Two workers, two PIDs, one database. Moving the worker start into lifespan startup means it runs in exactly one process. An atomic job-claim (UPDATE ... WHERE status='pending' with a rowcount check) backstops it, but the process count is what actually fixes the bug, not the SQL.

What I'd note

Self-overlap still defeats skeletonization. When a worm crosses itself the mask is fine but the skeleton collapses onto a partial segment until it untangles. The honest interim behavior is to flag those frames as data gaps rather than emit a wrong pose.
Two tracking pipelines duplicate their output logic. Classical and YOLO reimplement the same filter-and-export tail, which has drifted and produced wrong output before. Unifying them behind one shared write path is the main open refactor.
I deliberately did not build manual worm deletion. Hand-picking which worms to drop introduces selection bias a biologist could not defend. The right fix for a bad track is an upstream rule, not downstream curation.

I list these on purpose. On a tool whose output goes into research, knowing exactly where it is allowed to be uncertain is the point, not a footnote.

Personal Project

AI/ML Systems

BravoBot

A local-first voice assistant that runs entirely offline: speech-to-text, intent classification, semantic memory, and LLM responses, all on-device.

End-to-end voice pipeline: Whisper STT, fine-tuned DistilBERT intent classifier (13 classes), and Llama 3.2 served via Ollama.
Semantic memory with RAG: FAISS exact search over 384-dim Sentence Transformer embeddings, persisted across sessions.
Modular handler dispatch: new voice commands are single-file additions with a uniform signature.
Natural conversation flow via RMS silence detection and timezone-aware session logs queryable in natural language.

Python · Whisper · DistilBERT · FAISS · Ollama · Llama 3.2 · SQLite

Why it exists

Commercial voice assistants stream your audio to someone else's servers. BravoBot was built to test the opposite premise: can a useful assistant run entirely on a consumer machine, with no data leaving the device? It was also a deliberate exercise in composing multiple ML components (STT, intent classification, embeddings, retrieval, generation) into one coherent, low-latency pipeline on CPU-class hardware.

Architecture decisions

Fine-tuned DistilBERT for intent, not the LLM 13 classes, HuggingFace Trainer

Routing every utterance through Llama for classification would be slow on CPU and non-deterministic. A small fine-tuned classifier is fast, cheap, and predictable, and it lets the LLM be reserved for what it is actually good at: generating responses.

FAISS IndexFlatL2 for memory exact search, 384-dim MiniLM embeddings

Brute-force exact search is O(n), which sounds bad until you look at the actual scale: a personal assistant stores hundreds to low thousands of memories. At that scale, exact search is real-time and guarantees correct nearest neighbors. Approximate indexes would be premature optimization.

Dictionary-based handler dispatch uniform handle(user_input, context) signature

Every handler receives a context dict holding all service instances, so there is no global state. Adding a command is: add training examples, retrain the classifier, write one handler. The core dispatch logic never changes.

RMS silence detection over push-to-talk energy thresholding with block counting

Requiring a sustained run of low-energy blocks before stopping recording avoids cutting speech off during natural pauses, while still ending turns automatically. It makes the interaction feel conversational rather than transactional.

Tradeoffs & known limits

No confidence threshold on intent classification. Every input routes somewhere, so gibberish maps to the nearest training example instead of being rejected.
No distance threshold on retrieval. FAISS returns the closest match regardless of relevance; a jazz question could surface a hiking memory.
Small training set. 119 examples across 13 classes, some classes with as few as 3, which limits generalization for those intents.
The OpenAI fallback quietly breaks the offline promise. If Ollama is down, the system routes to GPT-3.5-turbo without notifying the user. Correct fix: notify and ask, or fail loudly.

I list these deliberately. Knowing exactly where your system fails is half of engineering it.

Sep 2023 — Dec 2024
Teaching Assistant
Python · Java

Teaching Assistant

Ontario Tech University

Mentored 50+ students in Python and Java across Introductory Programming and Data Structures.
Led weekly technical labs, debugging sessions, and drop-in office hours.
Graded assignments and exams with detailed feedback on algorithmic efficiency and code design.

Aaveg Shangari
builds and ships ML products.

What I've built and shipped

Savvant

Why it exists

Architecture decisions

Tradeoffs & known limits

ParaTracker

Why it exists

Architecture decisions

What I'd note

BravoBot

Why it exists

Architecture decisions

Tradeoffs & known limits

Teaching Assistant

Tools I work with

</>Languages07

✦AI / ML & Data08

{ }Frameworks06

#Infrastructure11

Foundation

B.Sc. (Honours) Computer Science, Minor in Mathematics

Let's build something