A model that scores well and a model that works are different things. From data leakage in phishing detection to a security system that flagged itself as an attack — these are the gaps, and how each one closed.
1.00 → 0.937
Perfect recall from a leaked shortcut
The legitimate-URL training set contained only bare domains, so the classifier learned that any URL with a path was phishing. It scored flawlessly in cross-validation and failed instantly on real traffic. Diagnosed through feature importances, fixed by replacing the dataset — and a third candidate set was rejected first for having the same zero path diversity that caused the problem.
0.40 – 0.50
A UI tier that contradicted the model
The extension's risk tiers and the backend's binary threshold were set independently, opening a band where the interface warned "medium risk" for URLs the model had already called legitimate. Fixed by moving the tier boundary onto the model's own decision threshold so the two can no longer disagree.
0.14 precision
Deleting the leak by deleting the features
One attempted fix removed the two features carrying most of the leaked signal outright. Precision collapsed to 0.14 — those features were genuinely informative, not merely leaky. Reverted, and solved at the data layer instead of the feature layer.
OOM → 12 chunks
A memory fix that fixed nothing
Transcribing a 4-hour file in one call crashed Colab on system RAM. The first patch bolted on VAD filtering and a CPU-thread cap — plausible-sounding flags that left the real cause untouched, and it crashed the same way again. The actual problem was loading the whole file at once, not decoding overhead; splitting the audio into twelve 20-minute pieces solved it. The lesson stuck harder than the bug: understand why something failed before trusting the fix, or you just move the crash.
restated ≠ answered
Retrieval that was right and useless
Vector search correctly found the intro to "selecting data" — a 2-second, six-word fragment. With only that in context, the model could restate the topic but not explain it. The retrieval wasn't wrong; the chunk was too small to be useful. Fixed by expanding each hit into a centred window of neighbouring segments, with explicit boundary clamping so a match near row 0 doesn't wrap around and pull chunks from the end of the video.
59 → 42 score
Training data that didn't match real traffic
Ghost Guard's Isolation Forest was flagging legitimate GET /api/products requests as anomalous (score 59, threshold 45). The synthetic training data assumed all requests had ~200-byte payloads, but real GET requests carry zero payload — a 4-standard-deviation gap the model correctly treated as unusual. Fixed by making payload generation method-dependent: GET payloads ~0, POST ~200. False-positive rate dropped to 1% on a 200-sample retest.
49 ≥ 45 but "normal"
Two ML signals that silently disagreed
The router AND-ed two independent outputs from Isolation Forest — anomaly_score (a manually scaled 0–100 number) and is_anomalous (the model's own internal binary). Score was 49 (above the 45 threshold), but is_anomalous returned False because the model's internal cutoff is calibrated differently. The result: requests that should have been honeypot-routed slipped through as normal. Fixed by using a single authoritative signal — score only.
self → honeypot
A dashboard that flagged itself as an attack
Ghost Guard's own dashboard endpoints (/events, /status, /honeypot-log) were being scored by the middleware and routed to honeypot — because their paths weren't in the known-safe whitelist. The threat level showed "HIGH" with zero real attacks, purely from the dashboard polling itself. Fixed by adding an EXCLUDED_PATHS list in the middleware that skips internal endpoints entirely.