# NLU model training

How the transcript → JSON model is trained, published and switched. There are two ways to train, and both are
described step by step: **Google Colab** (recommended, GPU) and **directly on the PC** (CPU, slow).

Code: `nlu_model/` at the repo root (see its `README.md`). Training uses **only this server's
database** (`server/data/laxmi.db`) plus generated sentences.

[TOC]

## 1. The whole flow

```
app saves an expense ──► POST /v2/nlu/results ──► nlu_events.final_json  (server/data/laxmi.db)
                                 │                        │
                                 │ user changed a value?  │ no edit: step 1, scripts/label_nlu_events.py (nightly)
                                 ▼                        │
       nlu_edits (old → new) + nlu_reviews "pending"      │
                                 │ step 2: admin reviews   │
                                 │ /admin/reviews          │
                                 │ approve → gold          │
                                 │ reject / skip → never   ▼
                                 └──────────────────► nlu_labels (gold / silver / skip)
old server v1 logs (imported once) ──► nlu_training_logs ─┤
synthetic sentences (nlu_model/lexicon) ──────────────────┴─► dataset: train · val · hard
                                                                   │
                         ┌─────────────── Colab (GPU) ─────────────┴──────────── PC (CPU) ───────────────┐
                         │ colab_pack → upload zip → notebook → result zip        retrain (one command)   │
                         └───────────────────────── publish ─────────────────────────────────────────────┘
                                                      │
                                   server/models/nlu/<new version>/  (+ vectors.jsonl)
                                                      │  /admin/models: check → compare → switch (latest.json)
                                                      ▼
                        server serves it (/v2/nlu/parse) · apps download it (GET /v2/models → update_available)
```

Training always starts from the pretrained base model (`google/mt5-small`) with **all** data, never from the
previous release.

## 2. How the model works

```
inputs : transcript "ત્રણ દિવસ પહેલા ચારસો ના કેળા લીધા", reference_date "2026-09-15"
          │
   laxmi_nlu.onnx (one file, ~55-60 MB)
     tokenizer → mT5-small (INT8) → date code "3 days ago" → exact date with integer maths → JSON
          │
output : {"amount":400,"category":"Groceries","merchant":null,"note":"કેળા","language":"gu",
          "spent_at":"2026-09-12","confidence":1.00,"needs_review":false}
```

- **Languages:** gu, hi and en. Add more one at a time (section 12).
- **Runtime:** it runs with `onnxruntime` + `onnxruntime-extensions` only (server, iOS, Android). There is no
  app-side logic.
- **Dates:** the model never computes dates. It outputs a code (days ago / last weekday / day of month / weeks ago),
  and the graph turns that into `spent_at`.

## 3. Training data

| Source | Where | Used when |
|---|---|---|
| Synthetic sentences | generated from `nlu_model/lexicon/*.json`: full sentences plus partial ones (amount only → `Misc`, no amount → `0`, nothing useful), ASR spelling variants and Romanized gu/hi | always (default 50k for Colab, 30k for `retrain`) |
| Hard set | `data/hard.jsonl`, from the same generator with many partial / Romanized / misspelled sentences | never trained on: calibrates confidence and is reported separately |
| Labels from real users | `nlu_labels` in `server/data/laxmi.db`, written by `scripts/label_nlu_events.py` from what users saved (online and offline) | `label_quality` = gold / silver (gold rows are repeated ×3) |
| v1 logs (old server history) | `nlu_training_logs` in `server/data/laxmi.db`, imported once from `server-old/`, never written again | the rules re-derive the same amount and date → silver |
| Golden test set | `nlu_model/testset/golden.jsonl` (git-ignored, real transcripts checked by a person) | evaluation and the release gate only, never training |

Per device, at most 200 real rows are used (one device cannot dominate). Sentences in the golden set are removed
from training.

**Before the first real release:**

1. **Native-speaker review** of `nlu_model/lexicon/gu.json`, `hi.json`, `items.json` and `merchants.json`.
2. **Golden test set.** Without it, no model is promoted automatically. Candidates come from what users saved
   (`nlu_events`) first, then the v1 logs, with the saved values filled in as a starting point:

```bash
cd nlu_model
../server/.venv/bin/python -m laxmi_nlu.golden candidates --per-language 150   # → testset/golden_candidates.jsonl
# edit each row: fix "target", set "checked": true
../server/.venv/bin/python -m laxmi_nlu.golden freeze                          # → testset/golden.jsonl
```

Running `candidates` again only adds new sentences.

### 3a. Add words and sentence patterns (`lexicon_additions`)

The synthetic sentences can only use the words and patterns in `nlu_model/lexicon/`. To teach the model new
item words, shops, sentence patterns, date phrases or currency words, add them as a **batch**, never by editing
`lexicon/` directly:

```
nlu_model/lexicon_additions/batch_NNN/
  items.json       {"<category>": {"gu": [...], "hi": [...], "en": [...]}}
  merchants.json   {"merchants": [{"name": "Zepto", "category": "Groceries", "gu": "ઝેપ્ટો", "hi": "ज़ेप्टो"}]}
  templates.json   {"gu": {"templates": [], "templates_amount_only": [], "templates_no_amount": [], "templates_no_info": []}, ...}
  dates.json       {"gu": {"today": [], "yesterday": [], ... existing keys only}, ...}     (optional)
  currency.json    {"gu": [], "hi": [], "en": []}                                             (optional)
  loanwords.json   {"<category>": [{"en": "milk", "gu": "મિલ્ક", "hi": "मिल्क"}]}            (optional)
  review.md        what was checked / corrected
```

Rules the batches must follow:
- the 15 category names exactly;
- gu / hi in native script (the generator romanizes and misspells by itself);
- one clear category per word;
- templates only use `{date} {amount} {currency} {item} {qty} {merchant}` and fit their group;
- templates use **neutral spending verbs**, because each template is filled with items from every category;
- only money already spent.

Batches 001–006 (written with an AI assistant, reviewed, then merged on 2026-09-22) added about 1,060 item words,
50 shops, 573 templates, 59 date phrases and 5 currency words.

**Merge** (validates everything first and writes nothing if anything is wrong; running it again only adds new
batches):

```bash
cd nlu_model
../server/.venv/bin/python -m laxmi_nlu.merge_additions --dry-run   # check + counts
cp ../server/data/indic_categories.json ../server/data/indic_categories.backup.json   # not in git
../server/.venv/bin/python -m laxmi_nlu.merge_additions             # lexicon/*.json + server keywords
../server/.venv/bin/python -m pytest -q                             # test_lexicon_is_consistent
```

**See what is in the lists:** the admin page **`/admin/lexicon`** shows every category with its item words per
language, the English words used in gu / hi speech, the shops and the rule keywords, plus every sentence pattern
with a filled example, the date phrases and currency words. Search a word to find which category it belongs to,
and use **Test a sentence** to see what the rules alone make of it.

**English words inside Gujarati / Hindi (code-mixing).** People say "aaje 300 rupiya nu **milk** lidhu". Speech
recognition writes such a word in the native script (મિલ્ક), a typed or English-mode sentence in Latin letters
(milk). `lexicon/loanwords.json` lists these words per category in the three forms (`gu` / `hi` = the English word
spelled in that script, not a translation). For each gu / hi sentence with an item the generator picks the native
word (60 %), the loanword in native script (20 %) or the English word (20 %); the expected note is exactly what
was said. When the whole sentence is romanized, a loanword is written the English way ("gift", not "gipht").
Evaluation reports category / note accuracy per form (`category|item_form=loanword_script`, ...).

The merge also adds every new item, shop and loanword (all three forms) to `server/data/indic_categories.json`, so the rules (the fallback
parser, the verifier and the labelling job) know the same words as the model. Restart the server afterwards.
Then retrain (section 5 or 6), and compare the new version with the current one on real phrases
(`/admin/models` → **Compare**) before switching.

## 4. Step 1: label the new user data (both flows)

**Two kinds of saved results:**
- **Not edited** (the user saved what the model showed): the rules below make the label automatically.
- **Edited** (the user changed any value or the transcript, or an offline entry had no model output): **never
  labelled automatically.** It waits in the review queue until an admin approves it (section 4a). The rules' decision
  is only shown there as a hint.

Run this from `server/`:

```bash
cd server
.venv/bin/python scripts/label_nlu_events.py --status     # how much is waiting / ready
.venv/bin/python scripts/label_nlu_events.py --dry-run    # see the decisions without writing
.venv/bin/python scripts/label_nlu_events.py              # write labels for events not labelled yet
```

**Output:** labels written by quality (`gold` / `silver` / `skip`), edit types, skip reasons, and a status block with
`new_usable_labels_since_served_model` and `retrain_suggested` (`true` at ≥ 1000 new gold/silver labels).

Only events **with a saved result** are labelled: an online parse that came back through `POST /v2/nlu/results`
(with its `event_id`), or an offline entry sent there. A parse the user never saved is not training data.

### How a label is decided

The rules live in `server/app/services/labelling.py`. For **edited** results the table below is only the
**suggestion** shown on the review page; the training label comes from the admin (section 4a).

| User edit | Label |
|---|---|
| no edit | what was shown → **gold** (silver if confirmed in < 1.5 s on a low-confidence result) |
| the new value is in the transcript, the shown value is not (the model was wrong) | user value → **gold** |
| the shown value is in the transcript, the new value is not (the user changed their mind) | shown value → **silver** |
| both values are in the transcript | **skip** |
| cannot be checked (e.g. Gujarati date words) | user value → silver if `edit_reason = model_error`, else **skip** |
| no amount anywhere in the transcript | amount `0` ("not said") → silver; skip if `edit_reason = model_error` |
| the user removed a merchant/note that was never said | `null` → **gold** |
| the user added a merchant/note that was not said | `null` → silver |
| category changed | the shown category (a personal preference), unless ≥ 20 devices made the same change for that item (≥ 70%) or Gemini chose it → silver |
| transcript edited | the label uses the edited text, `edit_type = asr_error` |
| a device whose edits are mostly not in the transcript (≥ 10 edits, ≥ 60%) | **skip** all its labels |
| same transcript + date + label already stored | **skip** (duplicate) |
| no model output to compare (offline entry without `model_json`) | the saved values → silver |

**Other options:** `--relabel` deletes all labels and labels everything again (use it after many new category votes
or a rule change).

**Nightly (cron):** the job also puts older edited results into the review queue (with their history), refreshes
the suggestions of pending ones, and applies admin decisions.

```bash
crontab -e
# 02:15 every night
15 2 * * *  cd /Users/imac/Documents/ios-project/laxmi-expense-manager/server && .venv/bin/python scripts/label_nlu_events.py >> data/labelling.log 2>&1
```

## 4a. Step 2: review what users edited (admin page)

**Only edited results that an admin approved are used for training.**

**Where:** `http://<server>:8082/admin/reviews`. It asks for the admin key (`LAXMI_ADMIN_API_KEY`) and your name
(shown in the history). The key stays only in that browser tab.

**What is recorded, the moment the app saves a result** (`POST /v2/nlu/results`):
- **`nlu_edits`:** one row per changed field, with the **old value** (what the model showed, or the previous save)
  and the **new value**. Transcript corrections are included. Later admin corrections are added as `changed_by =
  admin`.
- **`nlu_reviews`:** status `pending`, the edited fields, and the rules' suggestion.

**On the page:**

| Part | What you do |
|---|---|
| Tabs | Pending / Approved / Rejected / Skipped / All, with counts. Skipped and rejected results stay there for another look |
| Filters | language · edited field (amount, category, merchant, note, date, transcript) · the rules' suggestion (gold = the edit looks right, silver = it looks wrong, skip = unclear) · online / offline · source (model / Gemini / rules) · device · search in transcript and notes · newest / oldest first |
| List | each result's transcript and every change as ~~old~~ → **new** |
| Detail | the context (date said about, device, model version and confidence, the user's reason, confirm time, whether it is in training now), and per field: **model (old) · user (new) · rules check · training value** |
| Training value | prefilled with the user's value; edit it, or use **Use model** / **Use user**. The transcript training will use is editable too |
| Approve | the values in "training value" become a **gold** training label. Every value you changed is added to the history |
| Reject | wrong: never used for training |
| Skip | not used (unsure, test data, …); you can come back to it |
| Back to pending | undo a decision |
| Bulk | tick results in the list → **Approve as saved** (the users' values, unchanged; asks to confirm) / Skip / Reject |

**Rules that keep it safe:**
- A pending result has **no** training label.
- A result the user changes again after a decision goes back to **pending**, and its label is removed.
- `--relabel` never loses admin decisions; they are re-applied from `nlu_reviews`.
- Labels the rules made for edited results before this review step existed are removed by the next labelling run.
- `DELETE /v2/user-data` also deletes the device's history and reviews.

**Before the next training run:**

```bash
cd server
.venv/bin/python scripts/label_nlu_events.py --status   # user_edited_reviews: how many are still pending
sqlite3 data/laxmi.db "select status, count(*) from nlu_reviews group by 1"
```

The same data is available as JSON for scripts (with `X-Admin-Key`): `GET /v2/admin/reviews`,
`GET /v2/admin/reviews/{event_id}`, `POST /v2/admin/reviews/{event_id}` and `POST /v2/admin/reviews/bulk`
([API reference](/guide/api#get-v2adminreviews)).

## 5. Option A: train on Google Colab (recommended)

Roughly 25-35 min on a T4 for 50k sentences.

```
PC                                     Google Colab (GPU)                         PC
colab_pack  ── laxmi_nlu_colab.zip ─► laxmi_nlu_train.ipynb ── result zip ──►  publish
(dataset built here: needs the DB)     train → export .onnx → calibrate → eval    (gate vs current release)
```

### Step A1: PC, build the upload zip

Plain `python3` is enough, because the dataset code needs only the standard library.

```bash
cd nlu_model
python3 -m laxmi_nlu.colab_pack                   # synthetic + real labels + v1 logs + golden set (if present)
# or, to keep real user transcripts on the PC:
python3 -m laxmi_nlu.colab_pack --no-real-data
```

**Output:** `nlu_model/work/colab/laxmi_nlu_colab.zip` (a few MB). It contains:
- the `laxmi_nlu` code, lexicons and the `colab/` folder
- the prebuilt `data/` (`train`, `val`, `hard`, `summary.json`)
- `testset/golden.jsonl`, if you have one
- `pack_info.json`: creation time, git commit and dataset counts

`device_id` is removed from the rows.

**Options:**

| option | effect |
|---|---|
| `--synthetic-count 50000` | number of generated sentences |
| `--seed 7` | random seed |
| `--no-golden` | leave the golden set out |
| `--out PATH` | write the zip elsewhere |

Check `train_by_source` in the printed summary: `v2_label` is the user labels, `v1_log` the old history.

> The zip can contain real user transcripts (labels, golden set). Upload it only to the team's Google account,
> and delete it from Colab / Drive when done.

### Step A2: Colab, run the notebook

1. Open <https://colab.research.google.com> → **File → Upload notebook** → `nlu_model/colab/laxmi_nlu_train.ipynb`.
   Upload it again whenever it changes in the repo.
2. **Runtime → Change runtime type → T4 GPU** (L4 / A100 are faster if your plan has them).
3. **Runtime → Run all**. At step 2, choose `laxmi_nlu_colab.zip`.

| Notebook step | What happens |
|---|---|
| 1 | check the GPU |
| 2 | upload / unzip (or set `USE_DRIVE = True` to read the zip from Google Drive; the result zip is then saved there too) |
| 3 | create a Python 3.10 venv with the tested versions (`colab/setup_env.sh`, 3-5 min) |
| 4 | settings: `EPOCHS=6`, `BATCH_SIZE=32`, `LR=5e-4`, `KEEP_FF_OUT_FP32=False`, `TORCH_EVAL_SAMPLES=500` |
| 5 | `retrain --no-publish --data-dir data`: train on the GPU → export `laxmi_nlu.onnx` (INT8) → calibrate → evaluate |
| 6 | prints accuracy per language, INT8 vs FP32, accuracy by date kind / amount style, wrong answers, test sentences |
| 7 | downloads `laxmi_nlu_result_<RUN_ID>.zip` (the `.onnx` + results) |

**Reading step 6:**

- `amount`, `spent_at`, `category` per language: the targets are 0.97 / 0.97 / 0.90 (on the golden set).
- `INT8 file vs FP32 PyTorch`: the same accuracy means quantization is fine. If INT8 is clearly worse, set
  `KEEP_FF_OUT_FP32 = True` (bigger file) and run again.
- `category|item_form=…` / `note|item_form=…`: accuracy when the item was a native word, an English word written in
  the native script (મિલ્ક) or an English word (milk). `loanword_script` should be close to `native`.
- Scores on synthetic validation data are only a guide; the golden set is the real test.

### Step A3: PC, publish the result

This needs onnxruntime + onnxruntime-extensions, so use the server's venv:

```bash
cd nlu_model
../server/.venv/bin/python -m laxmi_nlu.publish --result ~/Downloads/laxmi_nlu_result_<RUN_ID>.zip
```

What `publish` does:
1. Extracts the zip to `nlu_model/work/runs/<RUN_ID>/`.
2. Runs 3 test sentences.
3. Checks the release gate against the current release.
4. Creates the next version, e.g. `server/models/nlu/1.3.0/`.
5. Writes that version's test vectors (`vectors.jsonl`).

Add `--promote` to also make it `latest`. That happens only when the gate passes on the golden set;
`--allow-synthetic-gate` lets a gate on validation data count. Then switch the server to it (section 8).

## 6. Option B: train directly on the PC

On the CPU this is slow: ~0.85 s per step at batch 8 on this iMac, several hours for a full run. A GPU is picked
automatically if there is one.

```bash
cd nlu_model
../server/.venv/bin/python -m laxmi_nlu.retrain --force --synthetic-count 50000 --epochs 6 --batch-size 8
```

One command runs every step and publishes into `server/models/nlu/<next version>/`. It is not promoted unless you
pass `--promote` and the gate passes.

| Step | Module | Output (under `nlu_model/work/runs/<run-id>/`) |
|---|---|---|
| 1. Trigger: enough new labels since the served model? (`--force` skips it) | `sources.py` | — |
| 2. Dataset: synthetic + real labels + v1 logs, golden set removed, gold ×3, device cap | `generate.py`, `sources.py`, `dataset.py` | `data/train.jsonl`, `val.jsonl`, `hard.jsonl`, `summary.json` |
| 3. Trim the mT5 vocabulary to the corpus, fine-tune **from the pretrained base** | `vocab.py`, `train.py` | `model/`, `train_summary.json` |
| 4. Export one end-to-end `.onnx` (INT8) | `export.py` | `export/uncalibrated.onnx` |
| 5. Calibrate confidence (temperature) on the hard / validation data | `evaluate.py` | temperature in the file metadata |
| 6. Evaluate the final file (golden set, else validation) | `evaluate.py` | `laxmi_nlu.onnx`, `evaluation.json` |
| 7. Gate: per-language targets + no regression vs the current release | `gate.py` | pass / fail + reasons |
| 8. Publish a new version; move `latest.json` only if the gate passed on the golden set and `--promote` | `registry.py` | `server/models/nlu/<version>/` |

| Flag | Meaning |
|---|---|
| `--promote` | move `latest.json` when the gate passes (scheduled use: `retrain --promote` without `--force` only trains when ≥ `--min-new-labels` 1000 new labels arrived) |
| `--max-steps 40` | quick smoke test (the model's output will be nonsense) |
| `--no-publish` | stop after evaluation; publish later with `laxmi_nlu.publish --run-dir work/runs/<id>` |
| `--skip-train --run-id <id>` | redo export / calibration / evaluation of an existing run |
| `--keep-ff-out-fp32` | bigger file, closer to PyTorch |
| `--no-quantize` | FP32 export |
| `--data-dir DIR` | use a dataset built elsewhere (this is what the Colab notebook does) |
| `--db`, `--registry DIR`, `--work-dir DIR` | read or write somewhere else (e.g. experiments) |

Keep the PC awake for long runs: `caffeinate -i ../server/.venv/bin/python -m laxmi_nlu.retrain ...`

**Individual steps** (for debugging):

```bash
cd nlu_model
PY=../server/.venv/bin/python
$PY -m laxmi_nlu.generate --count 50000 --out work/data/synthetic.jsonl
$PY -m laxmi_nlu.dataset  --out work/data/run1 --synthetic-count 50000
$PY -m laxmi_nlu.train    --data work/data/run1 --out work/runs/run1 --epochs 6 --batch-size 8
$PY -m laxmi_nlu.export   --model-dir work/runs/run1/model --out work/runs/run1/laxmi_nlu.onnx
$PY -m laxmi_nlu.evaluate --model work/runs/run1/laxmi_nlu.onnx --data work/data/run1/val.jsonl --limit 300
```

## 7. Where published models live

```
server/models/nlu/
  latest.json                 {"version": "1.2.0"}   the version served and offered to apps
  1.0.0/laxmi_nlu.onnx
  1.0.0/manifest.json         version, sha256, bytes, languages, metrics, gate result, dataset, training
  1.0.0/vectors.jsonl         test vectors for the app teams (written by publish / retrain)
  1.1.0/...
```

Versions go up automatically: 1.0.0 → 1.1.0 → 1.2.0. Old versions stay (for rollback).

Check what is published:

```bash
ls server/models/nlu
python3 -c "import json;m=json.load(open('server/models/nlu/1.2.0/manifest.json'));print(m['version'],m['file'],m.get('gate'),m['evaluation']['metrics']['overall'])"
```

## 8. Switch the server to a new model

**Option 0 (easiest): the admin page `/admin/models`.** Open `http://<server>:8082/admin/models`, enter the admin key.

| On the page | What it does |
|---|---|
| Served now | the version the server parses with and apps download; says if `.env` pins it |
| Installed versions | every folder in `server/models/nlu/`: trained date, languages, size, accuracy per language (red = lower than the served one), whether apps are offered it |
| **Check** | files present, sha256 matches the manifest, the file loads, valid JSON on a sample phrase per language. Changes nothing |
| **Switch to this** | runs the checks, then writes `latest.json` (nothing changes if a check fails). The server uses it on the next request, apps download it on their next start. No restart |
| **Roll back to X** | back to the version served before the current one (from the switch history; else the newest older one) |
| Compare versions on a phrase | the raw output of 2–4 versions side by side for any transcript; rows where they disagree are highlighted |
| Switch history | who switched, from → to, when, and why (the note) |

Typical test: publish **without** `--promote`, compare it on real phrases, **Switch to this**, try it in the apps /
`/admin/console`; if it is worse, **Roll back**. The same actions are API endpoints (`X-Admin-Key`):
`GET /v2/admin/nlu-models`, `POST /v2/admin/nlu-models/{version}/check`, `POST /v2/admin/nlu-models/{version}/activate`,
`POST /v2/admin/nlu-models/rollback`, `POST /v2/admin/nlu-models/try`.

If `LAXMI_NLU_MODEL_VERSION` is set in `server/.env`, the page shows it and cannot switch: the pin wins. Remove the
line and restart to manage the version from the page again.

**Option 1: promote.** This writes `latest.json` and needs no restart: the server re-reads it and loads the new
file on the next request.

```bash
cd nlu_model
../server/.venv/bin/python -c "from laxmi_nlu import registry; registry.promote('1.3.0')"
```

**Option 2: pin the version** (useful until a golden set exists). In `server/.env`:

```
LAXMI_NLU_MODEL_VERSION=1.3.0
```

Restart the server: `./start.sh`.

**Rollback:** `registry.promote('1.2.0')`, or pin the old version and restart.

**Check:**

```bash
curl -s http://127.0.0.1:8082/v2/health                              # nlu_model_version
curl -s "http://127.0.0.1:8082/v2/models?type=nlu" | python3 -m json.tool   # version, sha256, files
```

Apps only see released versions: the served one and older ones. A newly published version stays hidden until you
promote or pin it. When it is served, apps get `update_available: true` from
`GET /v2/models?installed=nlu@<their version>` and download it (see the [mobile guide](/guide/mobile)).

Vectors for an older version:

```bash
cd nlu_model
../server/.venv/bin/python -m laxmi_nlu.vectors --model ../server/models/nlu/<v>/laxmi_nlu.onnx \
    --out ../server/models/nlu/<v>/vectors.jsonl
```

## 9. Retraining with new user data (repeat whenever needed)

Step by step. The details of each step are in the section named in brackets.

**Before training**

1. **New words merged?** If you added `lexicon_additions/` batches, run `merge_additions` (3a), then
   **restart the server** so the rules load the new keyword / number files.
2. **Label** the real user data (4):
   ```bash
   cd server
   .venv/bin/python scripts/label_nlu_events.py --status
   .venv/bin/python scripts/label_nlu_events.py
   ```
3. **Review** what users edited on `/admin/reviews` (4a). Only approved edits are trained on; pending ones are
   left out.

**Train: Colab (recommended, ~30 min) or PC (hours)**

4. **Colab** (5):
   - PC: `cd nlu_model && python3 -m laxmi_nlu.colab_pack` → `work/colab/laxmi_nlu_colab.zip`
     (`--no-real-data` keeps user sentences on the PC). The zip contains the whole `lexicon/` folder, so the
     loanwords and every merged batch are used.
   - Colab: upload `colab/laxmi_nlu_train.ipynb` → T4 GPU → Run all → choose the zip → download
     `laxmi_nlu_result_<RUN_ID>.zip`.
   - PC: publish **without** `--promote`:
     ```bash
     cd nlu_model
     ../server/.venv/bin/python -m laxmi_nlu.publish --result ~/Downloads/laxmi_nlu_result_<RUN_ID>.zip
     ```

   **or PC** (6):
   ```bash
   cd nlu_model
   ../server/.venv/bin/python -m laxmi_nlu.retrain --force --synthetic-count 50000 --epochs 6 --batch-size 8
   ```

   Both create the next version (e.g. `server/models/nlu/1.3.0/`). The server keeps serving the old one.

**Test and switch** (8, admin page `/admin/models` → NLU model tab)

5. **Check** the new version: files, sha256, loads, valid JSON on sample phrases.
6. **Compare** the old and new version on real phrases, including code-mixed ones:
   ```
   આજે મેં ત્રણસો રૂપિયાનું મિલ્ક લીધું
   કાલે 200 નું ગ્રોસરી લીધું
   aje me tanso rupiya nu milk lidhu
   आज 300 रुपये का milk लिया
   ફાફડા 120 · ટોલ ટેક્સ 150 · मैनीक्योर 700
   ```
   plus sentences from real users. The accuracy shown on the page is measured on synthetic data: after a lexicon
   change the synthetic test is harder, so a lower number there does not mean a worse model.
7. **Switch to this** when it is better. The server and the apps follow on the next request / start; no restart.
8. **Watch** `/admin/reviews` for a few days. If users edit more than before, **Roll back**.

After a new model is served, `--status` counts new labels from that model's data cut-off
(`manifest.json` → `dataset.real_data_until`).

## 10. How `/v2/nlu/parse` decides (for support questions)

```
cache hit? ─► return the cached result (still logged)
language (sent, else guessed from the script) not in LAXMI_NLU_LANGUAGES ─► fallback
note / merchant not in the transcript ─► removed (category Misc if nothing is left), needs_review, no Gemini
invalid model output / amount 0 / rules disagree on amount or date / no amount in the transcript
   (LAXMI_NLU_REQUIRE_AMOUNT_EVIDENCE) / low confidence ─► fallback
fallback: Gemini (if enabled, key set, under the daily + monthly limits, not failing repeatedly)
   Gemini ok        ─► source "gemini"
   Gemini not used  ─► the model result with needs_review=true (if confidence ≥ 0.3 and it found an amount, item or
                        merchant; an amount with no evidence in the transcript is returned as 0 = "ask the user")
                    ─► else the rules (source "rules", needs_review=true)
                    ─► else 422 no_expense_found
```

## 11. Useful database queries

```bash
cd server
sqlite3 data/laxmi.db "select created_at, source, fallback_reason, transcript from nlu_events order by created_at desc limit 20"
sqlite3 data/laxmi.db "select count(*) from nlu_events where gemini_called=1 and created_at >= strftime('%Y-%m-01')"   # Gemini calls this month
sqlite3 data/laxmi.db "select label_quality, count(*) from nlu_labels group by 1"
sqlite3 data/laxmi.db "select channel, count(*), sum(final_json is not null) from nlu_events group by 1"   # saved results
```

Or through the API: `GET /v2/admin/training-data` (with `X-Admin-Key`).

## 12. Adding a language

1. In `nlu_model/`, add `lexicon/<code>.json` (numbers 0-99, scale words, dates, templates), its items and merchant
   spellings.
2. Add the code to `LANGUAGES` and its script to `SCRIPT_RANGES` in `laxmi_nlu/config.py`.
3. Add golden rows, then `retrain --force` (or Colab).
4. Publish and serve the model, then add the code to `LAXMI_NLU_LANGUAGES` in `server/.env`.

The tokenizer lives inside the `.onnx`, so the apps need no change. Speech for the language:
[setup guide](/guide/setup#82-add-a-new-language-eg-marathi).

## 13. Tests

```bash
cd nlu_model && ../server/.venv/bin/python -m pytest -q      # the pipeline, no model download
cd server    && .venv/bin/python -m pytest -q                # includes the labelling rules
```

## 14. Troubleshooting

| Problem | Cause / fix |
|---|---|
| `/v2/nlu/parse` always `source: gemini` or `fallback_reason: model_unavailable` | no NLU version served: check `ls server/models/nlu` and `latest.json`, or pin `LAXMI_NLU_MODEL_VERSION` and restart |
| `/v2/health` → `nlu_model_version: null` | same as above |
| changed `.env` but nothing changed | restart `./start.sh` |
| `fallback_reason` ends with `gemini_disabled` | no `LAXMI_GEMINI_API_KEY`, `LAXMI_GEMINI_ENABLED=false`, or monthly limit `0` |
| `gemini_daily_device_limit_reached` / `gemini_monthly_budget_reached` | the limits in `.env` were reached (by design) |
| `train_by_source` has no `v2_label` | run `scripts/label_nlu_events.py` first; check `--status` (are there saved results?) |
| `ModuleNotFoundError: pydantic` running `colab_pack` | the server's rule files must stay standard-library only (`server/app/engines/rules/`): see nlu_model's `test_server_rules_load_without_server_dependencies` |
| `publish needs onnxruntime + onnxruntime-extensions` | run publish with `../server/.venv/bin/python` |
| a new version is published but apps don't get it | it isn't served yet: promote or pin (section 8) |
| Colab: `No GPU found` | Runtime → Change runtime type → T4 GPU |
| Colab: CUDA out of memory | `BATCH_SIZE = 16` |
| Colab: package install fails in step 3 | re-run; don't upgrade transformers / onnxruntime (the export depends on the pinned versions) |
| Colab disconnected during training | `USE_DRIVE = True` and run again |
| Romanized Gujarati/Hindi parsed badly ("gaya somvare 500 nu petrol") | only partly in the training data; results are flagged `needs_review` |