AI observability: what should a small team actually monitor when shipping an LLM feature?

We are three developers adding an AI assistant to our product. Everyone says "you need observability" and then sells a platform. What do we actually need to log and watch so we notice when it goes wrong, without building a monitoring department?

Start with logging every call, because with LLMs you cannot reproduce a problem you did not record. Per request, keep:

• the full prompt (system + user), the model and its version, the parameters;
• the response, token counts in and out, latency, cost;
• an ID that ties it to the user session and to any tool calls it made.

Then the four things worth a dashboard:

1. Cost and tokens per day and per feature. Runaway prompts show up here first.
2. Latency percentiles (p50, p95). Averages hide the requests that time out.
3. Error and refusal rates: API errors, empty answers, the model declining. A jump usually means a provider change or a prompt regression.
4. Quality signals: thumbs up/down in the UI, how often people retry or rephrase, and a small daily sample you actually read.

Two habits that matter more than the tooling: version your prompts like code, and keep a fixed set of 30 to 50 test inputs you run before every prompt change and every model upgrade. Any tracing tool or a plain database table works for the storage; the discipline is the product.

Add an alert for the boring failure: the API key or quota. Half the "AI outages" I have been called about were a billing limit.