Quick search (Meilisearch)

What it is (and is not)

GET /api/v1/search is a typeahead endpoint backed by Meilisearch. It powers the dashboard's Cmd/Ctrl+K palette and is meant for async USelectMenu lookups and quick user lookup. It does not replace the paginated list endpoints: those stay the source of truth for tables, filters, sorting and exports.

GET /api/v1/search?q=celt&types=accounts&types=subscribers&limit=5
X-Tenant-ID: <tenant>

{
  "query": "celt",
  "groups": [
    {"type": "accounts", "estimated_total": 1,
     "hits": [{"id": "…", "label": "Celtic", "sublabel": "[email protected]"}]}
  ]
}

• limit is per type (1-20, default 5); q is at most 200 characters.
• Groups with no hits are omitted. A type the caller may not search is omitted too, not an error.
• 400 for an unknown types value. The response carries source: meilisearch, or database when Meilisearch is not configured or could not answer (see Database fallback). 503 only if the database fallback itself fails (for example a subscriber scan hitting the statement timeout); the dashboard treats 503 as "no results".
• Hits only carry what is needed to render a row (id, label, sublabel, status, role). Open the entity through its normal endpoint for details.

Database fallback

Search works without Meilisearch. /api/v1/search answers from Postgres when MEILI_URL is empty, when neither MEILI_SEARCH_KEY nor MEILI_MASTER_KEY is set, or when Meilisearch is unreachable or errors (query requests time out after 3 s, then fall back; a warning is logged and the response says "source": "database"). Scoping is identical: both paths are built from the same Scope objects in app/search/scope.py; only the way the filter is applied differs.

It is a degraded mode, implemented in app/search/fallback.py:

• Each word of q (at most 8) must match, as a case-insensitive substring (ILIKE '%word%', wildcards escaped), one of the type's searchable columns. A tail of a number therefore matches without suffix tokens; there is no typo tolerance and no ranking (results are ordered by name / MSISDN).
• Nothing indexes those columns, so on millions of subscribers it is a sequential scan. Subscriber queries run under a 5 s statement_timeout and answer 503 instead of holding a connection. Accounts are few and cheap. For a large deployment treat this as a stop-gap, not a steady state.
• estimated_total is the number of hits returned, not a count of all matches.
• Users come from the users table (filtered to the caller's tenant/accounts).
• Unlike the index, results are always current (no outbox lag, no dead-lettered rows).

Index layout

One index per entity type, each document carrying tenant_id. Separate indexes keep ranking rules and searchable attributes per type, give grouped results through one multi-search call, and let a type be rebuilt on its own.

Index	Document id	Searchable	Filterable (scope)
accounts	account_id	company/contact name, email, phone, VAT, ref, external id	tenant_id, account_id
subscribers	subscriber_id	MSISDN, ICCID, IMEI, serial, names, external id	tenant_id, group_id, status
users	users.id	email, first/last name	tenant_ids, account_ids, role

(sync_epoch is also filterable on every index, for full rebuilds.) Only current, live revisions are indexed (from_date <= now < to_date, not deleted, not archived). Users come from the users table; customer-kind users (RoleKind.CUSTOMER) are never indexed and platform memberships (superadmins) are not mapped to any tenant.

Definitions live in app/search/indexes.py, document builders in app/search/documents.py.

Matching notes

• Meilisearch matches word prefixes only. For identifiers people search by their tail (last digits of an ICCID or MSISDN), so subscriber documents also carry id_suffixes: every tail of 4 or more characters of each identifier.
• Typo tolerance is off for identifier fields, so a number never fuzzy-matches another number.
• At most 200 hits per query are computed (pagination.maxTotalHits), enough for typeahead.

Scoping

Meilisearch is queried through core-api (a proxy), never directly by the browser, because scope lives in the database and changes at any time (memberships, group moves). The filter is rebuilt on every request in app/search/scope.py - the only place scoping is decided. A type with no filter is skipped instead of searched unfiltered.

Access to a type needs its read permission (accounts:read, subscribers:read, users:read); without it the type is skipped. Beyond that, scope depends on whether the caller is account-scoped (is_account_scoped, role kind client):

Caller	accounts	subscribers	users
not account-scoped (admin / operator)	whole tenant	whole tenant	members of the tenant
account-scoped	own accounts (none if no account memberships)	subscribers in groups of own accounts (none if no groups)	whole tenant, or only users sharing one of their accounts if the caller's membership is restricted to accounts

Subscriber groups flagged exclude_from_search are always left out of subscriber results (Scope.exclude_groups).

The tenant is always the one from X-Tenant-ID; a tenant filter is part of every query.

Freshness

Two mechanisms keep the indexes current:

1. Live sync through a transactional outbox (accounts, subscribers). app/search/sync.py listens to SQLAlchemy sessions. Every flush that touches an Account or Subscriber inserts rows into search_outbox in the same transaction, so a committed change is never lost to a crash or a Meilisearch outage (and a rolled-back one leaves nothing behind). After commit the api starts a background drain for low latency (new and edited records are searchable within about a second; deletes and archives disappear just as fast). A drain claims rows with FOR UPDATE SKIP LOCKED (safe with many api replicas and the indexer at once), refreshes the current state of each distinct entity (documents.refresh: upsert if searchable, delete otherwise) and deletes the rows only on success. If Meilisearch is down the rows simply stay queued and the indexer's timer drains them later. Bulk SQL that bypasses the ORM writes no outbox rows (see reconcile below).

   Poison rows. If a page fails while Meilisearch is healthy, the drain bisects it to find the entities that fail on their own; everything else in the page is still indexed. Each failing entity gets attempts bumped and last_error set, and after SEARCH_OUTBOX_MAX_ATTEMPTS (default 10) its rows are dead-lettered: failed_at is set, an error is logged, and the rows are never claimed again, so they cannot block the queue. An outage never counts as an attempt. Inspect with SELECT * FROM search_outbox WHERE failed_at IS NOT NULL; once the cause is fixed re-queue with UPDATE search_outbox SET attempts = 0, failed_at = NULL, last_error = NULL. A later successful refresh of the same entity also clears its dead-lettered rows.

2. search-indexer service (the safety net). scripts/search_indexer.py runs continuously:

   • every SEARCH_OUTBOX_INTERVAL (default 5 s) it drains the outbox (the durable safety net for the api's own post-commit drain);
   • on start, an accounts/subscribers index that is empty gets a full rebuild; otherwise it is reconciled incrementally;
   • every SEARCH_RECONCILE_INTERVAL (default 3600 s) it reconciles accounts and subscribers: documents.reconcile refreshes only entities whose revision started, ended, was deleted or archived since the previous run. This catches writes that bypass the commit hook (bulk SQL, scheduled from_date changes) and costs little even with millions of rows;
   • every SEARCH_USERS_INTERVAL (default 300 s) it rebuilds users (the users table has no commit hook; the set is small).

   A full rebuild is manual: python scripts/search_reindex.py [accounts|subscribers|users]. It streams rows in keyset batches (flat memory) and then drops documents with an older sync_epoch. Measured indexing speed is about 4k docs/s, so 5M subscribers take roughly 20 minutes; do not schedule it frequently.

What a caller is allowed to see is read from the live database on every request (memberships, group ownership); only the content and placement stored on the documents can lag. Moving a subscriber to another group, or changing a user's accounts, therefore takes effect for scope filtering at the next refresh of that document (for users: the next full rebuild).

Configuration

Variable	Purpose
MEILI_URL	Meilisearch base URL. Empty (or no key below) makes /api/v1/search use the database fallback.
MEILI_MASTER_KEY	API key sent as a bearer token. Required by Meilisearch in production mode (16+ bytes). Needed by writers; queries use it only when MEILI_SEARCH_KEY is unset.
MEILI_INDEX_PREFIX	Prepended to index names, for sharing one instance between environments or tests.
MEILI_SEARCH_KEY	Optional search-only key used for /api/v1/search queries; falls back to the master key. See Operating Meilisearch.
SEARCH_SUFFIX_FIELDS	Subscriber identifiers searchable by their last digits (default msisdn,iccid; any of msisdn, iccid, imei, serial). Changing it needs search_reindex.py subscribers.
SEARCH_OUTBOX_INTERVAL	search-indexer: seconds between outbox drains (default 5).
SEARCH_OUTBOX_MAX_ATTEMPTS	Failed refreshes (Meilisearch reachable) before an outbox row is dead-lettered (default 10).
SEARCH_USERS_INTERVAL	search-indexer: seconds between user rebuilds (default 300).
SEARCH_RECONCILE_INTERVAL	search-indexer: seconds between incremental accounts/subscribers reconciles (default 3600).
SEARCH_RECONCILE_LOOKBACK	search-indexer: how far back the first reconcile after a restart looks (default 86400).

Dev compose files start a meilisearch service on localhost:7700 with a throwaway key; api has no depends_on on it, so search stays optional (the database fallback covers it). In deployment/docker-compose.yml Meilisearch is not published on the host and MEILI_MASTER_KEY must be set in .env.local before first boot (changing it later means recreating the meilisearch-data volume).

Operating Meilisearch

Numbers measured on a dev machine with synthetic subscribers (100k and 300k documents), so treat them as estimates:

• Index size: about 4.4 KB per subscriber with the default suffix fields (msisdn,iccid; 5.6 KB with all four identifiers) - roughly 22 GB for 5M subscribers. LMDB keeps free pages and grows during rebuilds, so deployment/docker-compose.yml budgets a ~100 GB meilisearch-data volume.
• Speed: about 4-9k docs/s indexing; a full rebuild of 5M subscribers takes roughly 10-20 min. Queries stay at 6-16 ms with a tenant filter.
• Memory: the container is capped (MEILI_MEM_LIMIT, default 4g) and the indexing budget is set below it (MEILI_MAX_INDEXING_MEMORY, default 2GiB). Raise both together for big imports; the remainder is page cache for the index.
• Snapshots: MEILI_SCHEDULE_SNAPSHOT writes a snapshot every MEILI_SNAPSHOT_INTERVAL (default daily) to the meilisearch-snapshots volume, for a fast restore (--import-snapshot). It is not a backup of record: Postgres is the source of truth and search_reindex.py rebuilds everything. Keep the snapshot volume as large as the data volume (Meilisearch keeps the latest snapshot only).
• Search-only key: run docker compose --env-file .env.local exec search-indexer python scripts/search_key.py once to create the key (it can only search), put it in MEILI_SEARCH_KEY and recreate api. Query requests then use it. The api still holds the master key because its post-commit drain writes documents; to take the master key off the api entirely, drop that drain and leave all writes to search-indexer.

Known limitations / next steps

• The database fallback does not scale like the index (see above); there is no trigram index on the searched columns. Add pg_trgm GIN indexes if it must carry real traffic.
• Users are only refreshed by the periodic rebuild, so a new or changed user can take up to SEARCH_USERS_INTERVAL to appear.
• Dead-lettered outbox rows are only logged; there is no alert on outbox age or dead-letter count yet.
• Reconcile finds changes through Postgres columns, so a row removed with a hard DELETE (the api only soft-deletes) is only cleaned up by a full rebuild.
• Tail search only covers SEARCH_SUFFIX_FIELDS; an IMEI or serial matches by prefix only.
• Only accounts, subscribers and users are indexed; customers, groups, plans and tickets can be added by one IndexDef, one document builder and one scope rule each.
• The api still holds the master key for its post-commit drain (see the search-only key above).