Inference

AI & Machine Learning · Intermediate · 4 min read

What is it?

Inference is the process of using a trained AI model to make predictions or generate output, as opposed to training it.

Explain like I'm 5

If training is studying for an exam, inference is actually taking the exam: the model uses what it already learned to answer new questions.

Why was it created?

It's simply the 'using' phase of a model's life. The term distinguishes running a finished model from the separate, costly job of training it.

Where is it used?

Serving AI predictions in apps
Generating LLM responses
Real-time recommendations
Any live use of a trained model

Why should developers care?

Inference is where AI delivers value and where most ongoing cost and latency live, so it matters for anyone deploying models.

How does it work?

A trained model takes new input and runs it through its fixed, learned parameters to produce an output. Unlike training, the model's weights don't change during inference.

Real-world example

When you send a prompt to an AI assistant, the running model performs inference to generate the reply, using weights set during training.

Common use cases

Live predictions and generation
Serving models at scale
Real-time AI features
Batch scoring of data

Advantages

Delivers the model's value
No weight updates needed
Can be optimized for speed
Runs continuously in production

Disadvantages

Ongoing compute cost
Latency to manage
Hardware demands for large models
Scaling to many users is expensive

When should you use it?

Whenever you put a trained model to work on new inputs.

When should you avoid it?

Not avoidable for using a model; the cost-saving levers are optimization and right-sizing.

Alternatives

Precomputed/cached results (for repeated inputs)Smaller or quantized models for cheaper inference

Interview questions

Beginner

What is inference?
How does it differ from training?

Intermediate

Why does inference cost add up?
Do weights change during inference?

Senior

How do you reduce inference latency and cost?
How does quantization help inference?

Common misconceptions

"Inference and training are the same" — training learns the weights; inference just uses them.
"Inference is free once trained" — running a model at scale has real, ongoing compute cost.

Fun facts

For popular models, total inference cost over time can far exceed the one-time training cost.
Techniques like quantization speed up inference by using smaller number formats.

Timeline

2010s — Inference optimization becomes a major focus as models deploy widely

Learning resources

Google: Machine Learning Crash Course

Quick summary

Inference is using a trained model to generate predictions or output, where AI delivers value and where ongoing cost and latency concentrate.

Cheat sheet

Using a trained model
Weights stay fixed
Where value + cost live
Optimize for speed and price

If you remember only one thing

Inference is putting a trained model to work — the 'using' phase, distinct from the 'learning' phase of training.