# Laxmi server

The Laxmi API: 12 endpoints, one error shape, and the device id always in the `X-Device-Id` header. It is
**self-contained**: its own code, models (`models/`), Python environment (`.venv/`) and database (`data/laxmi.db`).
It replaced the old server on 2026-09-21. The old one is kept in `../server-old/` for reference only and nothing
here reads it (its data was imported once, see the [setup guide](SETUP_GUIDE.md#71-old-server-data-imported)).

```bash
./start.sh            # http://127.0.0.1:8082  (docs home)   /docs = Swagger
```

Docs as web pages (served by the API itself):

| page | URL | file |
|---|---|---|
| Home: status + all endpoints | `/` | |
| **API console**: call every API from the browser, step by step | `/admin/console` | |
| Review user edits (admin) | `/admin/reviews` | |
| Models (admin): NLU switch / roll back, speech packs + .nemo | `/admin/models` | |
| Word lists (admin): training words, patterns, rule keywords | `/admin/lexicon` | |
| API reference | `/guide/api` | [docs/API_REFERENCE.md](docs/API_REFERENCE.md) |
| Mobile guide | `/guide/mobile` | [docs/MOBILE_GUIDE.md](docs/MOBILE_GUIDE.md) |
| Setup and operations guide | `/guide/setup` | [SETUP_GUIDE.md](SETUP_GUIDE.md) |
| NLU training (Colab + PC) | `/guide/training` | [docs/NLU_TRAINING.md](docs/NLU_TRAINING.md) |
| Changes from the old server | `/guide/changes` | [docs/MIGRATION.md](docs/MIGRATION.md) |

Add `.md` to any guide URL for the raw Markdown. New machine: see the [setup guide](SETUP_GUIDE.md#3-first-time-setup).

## Endpoints

| # | Point | Endpoint | Purpose |
|---|---|---|---|
| 1+2 | Audio → text | `POST /v2/asr/transcribe` | multipart: `audio`, `language`, `engine` (auto / onnx / nemo), `romanize`, `consent_store_audio` |
| 1+2 | Audio → expense | `POST /v2/voice/parse` | the same speech step, then the NLU. Response = expense + `asr` block |
| 3+4 | Model list | `GET /v2/models` | ASR packs (`asr-gu`, `asr-hi`) and the NLU model (`nlu`) in one list. `?type=`, `?lang=`, `?installed=asr-gu@…,nlu@1.1.0` sets `update_available`, `?max_nlu_schema=` |
| 3+4 | Model file | `GET /v2/models/{id}/files/{name}` | one file. HEAD + Range, so downloads resume. `?version=` for older NLU versions |
| 4 | Text → expense | `POST /v2/nlu/parse` | JSON `{transcript, reference_date, currency}`: ONNX model → rule checks → Gemini (capped) → rules |
| 5 | Saved results | `POST /v2/nlu/results` | what the user saved. Online entries carry `event_id`; offline entries carry `transcript`, `reference_date`, `created_at`. Idempotent on entry `id` |
| 6 | Languages | `GET /v2/languages` | every language: `supported` (offered now), `offline_asr`, `server_asr`, `requested_by_device` |
| 6 | Request a language | `POST /v2/languages/requests` | `{language: "mr" / "Marathi" / "मराठी"}`. One row per device per language; asking again raises that row's `request_count` |
| admin | Demand | `GET /v2/admin/language-requests` | unique devices per language, most wanted first (`X-Admin-Key`) |
| admin | Training export | `GET /v2/admin/training-data` | NLU events + what users saved (`X-Admin-Key`) |
| admin | Review user edits | `GET/POST /v2/admin/reviews…` + page **`/admin/reviews`** | user-edited results (old → new); only approved ones are used for training |
| | Health | `GET /v2/health` | what is installed; loads no model |
| | Privacy | `DELETE /v2/user-data` | deletes this device's audio, events and language requests |

Every app endpoint (everything under `/v2/` except `/v2/health` and `/v2/admin/*`) needs the app key in
**`X-API-Key`** (`LAXMI_APP_API_KEYS`; see the setup guide, 10.5).

`X-Device-Id` (1-128 chars `A-Za-z0-9._:-`) is required on voice/parse, nlu/parse, nlu/results,
languages/requests and user-data. It is optional elsewhere.

Every error has the shape `{"code": "...", "message": "..."}`. Common codes: `missing_device_id`, `unknown_language`,
`unsupported_language`, `empty_audio`, `bad_audio`, `audio_too_large` (413), `model_unavailable` (503),
`no_expense_found` / `no_speech` (422, with `event_id` / `asr`), `invalid_request` (422, with `details`),
`unauthorized` (401), `admin_disabled` (503).

## Speech engines (points 1 + 2)

| engine | model | languages | speed |
|---|---|---|---|
| `onnx` | FP16 ONNX pack per language in `LAXMI_ASR_ONNX_DIR/<lang>/`, the same files the apps download | gu, hi | ~1 s |
| `nemo` | multilingual IndicConformer 600M `.nemo` (`LAXMI_ASR_NEMO_PATH`), needs torch + AI4Bharat NeMo | 22 Indic languages (Marathi, Tamil, …) | ~45 s first load, then ~1-2 s |

`engine=auto` (the default) uses the ONNX pack when the language has one, otherwise `.nemo`. The multilingual model
cannot detect the language, so `language` is always required. For voice/parse, the expense's `language` is the
spoken one: Marathi and Hindi share a script, so the transcript alone can't tell them apart.

The NLU model file knows gu / hi / en. Other languages (e.g. Marathi from `.nemo`) go to Gemini, then to the rules.

## Layout

```
app/
  main.py  config.py  db.py  schemas.py  errors.py  deps.py
  routers/   docs health voice nlu models languages admin user_data
  services/  nlu_store asr_store languages model_catalog
  engines/
    audio.py                  decode + resample to 16 kHz
    asr/   onnx_pack nemo_multi stub
    nlu/   model gemini pipeline verifier matching
    rules/ numbers_indic dates categories   (from the old server; lexicons in data/indic_*.json)
scripts/  label_nlu_events.py                        nightly: saved results -> nlu_labels (training labels)
          export_asr_packs.py verify_asr_packs.py   .nemo -> ONNX speech packs, and the parity check
          purge_expired_audio.py                    daily: deletes opted-in audio past retention
          import_from_server_old.py                 one-off: the old server's data -> this database (done)
docs/     the guides above
models/   asr/<lang>/  nlu/<version>/ + latest.json  nemo/*.nemo      (git-ignored)
```

Tables (`data/laxmi.db`, created on first use): `asr_events`, `nlu_events`, `nlu_labels`, `language_requests`, plus
`nlu_training_logs` (the old server's v1 history, imported once, read by `nlu_model/`).

## Tests

```bash
.venv/bin/python -m pytest -q
```

The tests never read `.env`. They use the stub speech engine, a fake NLU model, a fake Gemini and a temporary
database, so they make no network calls and need no model files. The real models were checked by hand (2026-09-21):
- **gu/hi ONNX:** 17 of 17 clips gave exactly the same transcript as the old server, and `verify_asr_packs.py` reports
  the gu pack IDENTICAL to its `.nemo` source.
- **Marathi:** transcribed through `.nemo`.
- **Voice → expense:** correct on the gu and hi clips.

## Training data (nlu_model/)

`nlu_model/` reads this server's `data/laxmi.db` and publishes to `models/nlu/` (its defaults). Run
`scripts/label_nlu_events.py` before building a dataset. Checked on 2026-09-21: after the import, the dataset,
the labels and the Colab zip data are the same as the old server produced. Details:
[NLU training guide](docs/NLU_TRAINING.md).

Left out on purpose: the old `/v1/*` endpoints.