How to add AI to your product: a practical guide to LLM features

Short answer: pick one frequent, well-defined task your users already do, where a good draft or answer saves real time and a mistake is easy to spot. Build it behind a small service that calls a language model with the right context, measure it against a set of realistic test cases before launch, design the interface so users can check and correct the output, and roll it out gradually while you monitor quality, cost and abuse. Then repeat for the next task.

Adding AI to an existing product is mostly ordinary product and engineering work with a few new rules. The model call is the easy part. The hard parts are choosing the right job, giving the model the right information, knowing whether it works, keeping it safe, and keeping costs predictable. This guide walks through each, in the order you will face them.

Start from a job, not from AI

The most common failure is starting with 'we need an AI feature' and ending up with a generic chat box nobody uses. Start instead from a task in your product that is frequent, tedious and language-heavy. Look at support tickets, session recordings and the places users copy data out of your product into something else.

Typical strong first features: summarising long records or threads, drafting replies or documents from context the product already holds, extracting structured fields from emails, PDFs or forms, classifying and routing incoming items, searching across a customer's own content in natural language, and turning a vague request into a filled-in form for the user to confirm.

Choose the simplest pattern that works

Anthropic's advice on building effective agents is to find the simplest solution possible and add complexity only when it demonstrably improves outcomes. Often one well-prepared model call is enough. Here are the main patterns, from simplest to most complex.

Choose a model, and keep the choice reversible

You will choose between hosted models from providers such as Anthropic, OpenAI and Google, often also available through your cloud provider, and open-weight models you run yourself. Compare them on your own test cases, not on public leaderboards, and weigh these criteria:

• Quality on your task, measured with your evaluation set.
• Latency: time to first token for interactive features, total time for background jobs.
• Cost per task at your expected volume, including the context you send each time.
• Data handling: whether the provider trains on your data, where it is processed and stored, retention, and whether a data processing agreement is available.
• Features you need: structured output, tool calling, long context, vision for documents and images, prompt caching, batch processing.
• Operational fit: rate limits, regional availability, and support through your existing cloud contract.
• Lifecycle: providers retire models on a published schedule. Anthropic, for example, gives at least 60 days' notice before retiring a publicly released model, after which requests to it fail.

Because models change so often, keep the choice reversible. Put model calls behind one internal interface, store prompts as versioned files rather than strings scattered through the code, and keep your evaluation set ready so switching models becomes a measured change rather than a leap of faith. Many products end up routing requests: a small, fast model for simple tasks and a larger one for hard cases.

Where the AI fits in your architecture

Avoid calling model APIs from all over your codebase or, worse, from the browser. A small AI service or module inside your backend gives you one place to apply the rules that matter.

• Authorisation first: the service only receives data the current user is allowed to see, resolved on the server. The model must never be the thing that decides access.
• Prompt assembly: versioned templates, the user's input clearly separated from instructions, and only the context the task needs.
• Structured outputs: ask for JSON against a schema when the result feeds other code, and validate it before use. Treat model output as untrusted input.
• Timeouts, retries and fallbacks: if the model is slow or unavailable, the product should degrade gracefully, for example by hiding the suggestion rather than blocking the user.
• Streaming for interactive text so users see progress immediately.
• Background jobs for long or bulk tasks such as processing a batch of documents, with progress and results stored like any other job.
• Logging and tracing: record prompt version, model, latency, token counts, cost and outcome for each call, with personal data minimised or redacted.
• Feature flags: switch the feature on per tenant or user, and off instantly if something goes wrong.

If your product is multi-tenant, the AI layer must respect tenancy as strictly as the rest of the system: separate retrieval indexes or strict tenant filters, per-tenant usage limits, and no shared caches of customer content. Our SaaS development guide covers tenant isolation in more depth.

Give the model the right context

Most quality problems are context problems. The model can only be as good as the information you put in front of it. Spend your effort here before reaching for a bigger model or fine-tuning.

• Use what your product already knows: the record the user is looking at, its history, the customer's settings and the user's role are often more valuable than any document search.
• Retrieve carefully: split documents along their natural structure, keep titles and metadata with each chunk, combine keyword and semantic search, and filter by permission before ranking.
• Keep it fresh: decide how quickly changes to source content must show up in answers, and build the indexing pipeline to match.
• Show examples: a few examples of excellent outputs for your domain often improve results more than longer instructions.
• Say what to do when the answer is not there: instruct the model to say it does not know, and test that it does.

We go deeper on retrieval in RAG in production: what decides answer quality. Fine-tuning a model on your data is occasionally worth it for consistent formats, specialised style or cost at very high volume, but it adds data preparation and maintenance, and it does not teach a model facts reliably. Try better context first.

Evaluate before you launch, and after

An evaluation set is to an AI feature what automated tests are to ordinary code. Without one, every prompt change or model upgrade is a guess. Anthropic's guidance on developing tests recommends evaluations that mirror the real distribution of tasks including edge cases, automated grading wherever possible, and more test cases over fewer hand-graded ones.

1. Collect real cases. Start with 50 to 200 realistic inputs from your product, anonymised where needed, including awkward ones: empty input, very long input, wrong language, hostile input, and questions the data cannot answer.
2. Define what good means for each case: an exact expected value for extraction and classification, or a short rubric for drafts and summaries (accurate, complete, cites the source, right tone).
3. Automate scoring: code checks for structured outputs, and a separate grading model with a clear rubric for free text, spot-checked by people until you trust it.
4. Set a launch bar with the product owner, based on the cost of errors, for example the share of drafts accepted with minor or no edits.
5. Run it on every change to prompts, retrieval, models or code, and compare results before merging.
6. Feed it from production: every bad output a user reports becomes a new test case.

Treat the evaluation set as the specification of the feature. Prompts and models will change many times; the definition of a good answer should change only when the business does.

Design the experience for imperfect outputs

Language models are sometimes wrong in fluent, convincing language. The interface decides whether that is a minor annoyance or a serious problem.

• Draft, then confirm: present outputs as suggestions the user can edit, accept or discard, especially before anything is sent, saved or paid.
• Show the sources behind answers, with links to the original document or record, so users can check quickly.
• Make it easy to correct and report: a quick way to fix, retry or flag a bad output, which also feeds your evaluation set.
• Be honest about what it is: label AI-generated content and make clear when users are talking to an AI system. In the EU this is becoming a legal requirement, as described below.
• Avoid the empty chat box when a button will do. 'Summarise this case' in context is more discoverable and easier to test than an open prompt.
• Respect performance: stream text, show progress for longer tasks, and never block a core workflow on an AI call.

Security and privacy

LLM features open new attack paths. The OWASP Top 10 for LLM Applications is the best checklist; these are the risks that most often matter when adding AI to an existing product, with the controls we would expect.

On privacy, check your provider's business terms on training, retention and processing regions, sign a data processing agreement, add the provider to your list of sub-processors, update your privacy notice, and send only the data each task needs. If you serve EU users, the AI Act's Article 50 transparency duties, including telling people when they interact with an AI system, apply from 2 August 2026 according to the implementation timeline. Features that make decisions in high-risk areas such as hiring or credit carry heavier obligations, so take legal advice before building them.

Control cost and latency

Model costs are usage-based, usually per token of input and output, so a feature that works in a demo can become expensive at scale. Design for cost from the start:

• Send less: trim context to what the task needs; retrieval exists partly to avoid sending whole documents.
• Cache repeated context. Providers offer prompt caching for long, stable prompt prefixes. On Anthropic's API, cache reads are billed at a fraction of the normal input price, typically 0.1 times, with a premium on the first write, and the default cache lifetime is five minutes.
• Route by difficulty: use smaller, faster models for classification and simple drafting, and larger ones only where your evaluations show they are needed.
• Batch what is not urgent: overnight summaries and bulk extraction can use asynchronous or batch processing, which is often cheaper.
• Cache results: if many users ask for the same summary of the same record, compute it once and invalidate it when the record changes.
• Set limits and watch them: per-user and per-tenant quotas, maximum input sizes, and alerts on daily spend. Track cost per feature and, in SaaS, per tenant, so pricing can reflect it.

Ship gradually and keep it healthy

1. Internal use first: your own team uses the feature on real data for a week or two and reports problems.
2. Design partners: switch it on for a few willing customers behind a feature flag, with close contact and visible feedback controls.
3. Gradual rollout: widen access in steps while watching acceptance rates, error reports, latency and cost.
4. Monitor in production: sample outputs for review, track user corrections and abandonment, and alert on spikes in errors, refusals or spend.
5. Plan model changes: when a provider releases or retires a model, run your evaluation set on the replacement, compare cost and latency, and switch behind a flag.
6. Assign an owner: someone must own the feature's quality, the evaluation set and the prompt changes, just as someone owns any other part of the product.

A worked example: AI-drafted support replies

Consider an illustrative B2B product with a built-in help desk, where agents spend much of their day writing similar replies. The feature: when an agent opens a ticket, the product suggests a reply based on the ticket, the customer's account and the help centre, with links to the sources used. The agent edits and sends it. Nothing is sent automatically. Timings are examples, not a quote.

• Weeks 1 to 2: a senior engineer and the product owner collect a few hundred past tickets with the replies that were actually sent, define a rubric (correct, complete, right tone, cites the right article), and build a quick prototype scored against it.
• Weeks 3 to 6: two engineers build the AI service, help centre indexing with tenant filters, the reply panel with sources and edit controls, logging, quotas and feature flags. Evaluations run on every change.
• Weeks 7 to 8: security testing with injection attempts hidden in tickets, a privacy review, then release to internal support staff and two design-partner customers.
• After launch: weekly review of acceptance rates and flagged replies, new test cases from failures, and a gradual rollout to all customers.

A small team can take a feature like this from idea to a controlled launch in about two months, because the scope is one job, the output is a draft a person checks, and the data already lives in the product. Adding automatic sending, or letting the assistant issue refunds, would be a separate project with a much higher bar for evaluation and approval. Use our software cost calculator to explore how scope changes effort.

When not to add AI

• When it is there for the press release. An AI feature that solves no specific user problem adds cost, risk and support load.
• When rules or search would do. Clear conditions, a better filter or ordinary full-text search are cheaper, faster and perfectly predictable.
• When errors are costly and nobody can review them. If a wrong output causes real harm and there is no practical way for a person to check it, wait until you can design that check in.
• When the data is not ready. If the information is missing, out of date or not permissioned properly, fix that first; the AI will only expose the problem.
• When you cannot measure it. If you cannot say what a good result looks like, you cannot tell whether the feature works, or whether next month's change broke it.

How Authect approaches this

When you work with Authect, you talk to the founders who will build the feature, never a salesperson. A senior engineer leads every project, starting with the job to be done, your data and an evaluation set, so decisions about models, prompts and architecture rest on evidence. We will tell you if rules, search or an existing tool would serve you better than a language model.

We use AI in our own delivery, with an engineer reviewing every line, and we build small, focused teams. You own the code, prompts, evaluation sets, infrastructure and designs from day one, with no lock-in to us or to one model provider. We work as fixed-price projects for well-defined features, monthly sprints when you are exploring, and support and evolution once features are live. See our generative AI services, read what AI development services include, or tell us which feature you are considering.