A large language model (LLM) is a neural network trained on trillions of words of text to do one thing extremely well: predict the next token in a sequence. Everything an LLM appears to do, from answering questions to writing code to summarizing a legal brief, emerges from that single predictive objective applied at massive scale. In practice, that one skill covers a lot of ground: teams use LLMs to generate and review code, summarize documents, answer questions over internal data, translate between languages, and drive agents that call tools and APIs.
That definition is true in the same way "a CPU executes instructions" is true. It won't help you build anything. If you're going to ship something on top of an LLM, the useful knowledge is how the pieces fit together: what happens between the prompt you send and the tokens you get back, why training works the way it does, and where the sharp edges are.
From text to tokens to vectors
An LLM never sees words. Input text is first split into tokens, subword units drawn from a fixed vocabulary of roughly 32,000 to 250,000 entries depending on the model. Common words like "the" get their own token; rarer strings get split. "Tokenization" might become token + ization. This is why LLMs are billed and benchmarked per token, and why a model's context window is measured in tokens rather than words. A useful rule of thumb for English: one token is about three-quarters of a word.
Each token is then mapped to an embedding, a vector of numbers, typically thousands of dimensions long. Embeddings are learned during training, and they encode meaning as geometry: tokens that behave similarly in text end up near each other in vector space. The embedding for "Python" starts as one fixed vector covering every sense of the word; pushing it toward programming or reptiles based on the surrounding text is the job of the layers above.
Attention: how context gets built
The architecture behind nearly every modern LLM is the transformer, introduced in the 2017 paper "Attention Is All You Need." Its core mechanism, self-attention, lets every token in the input look at every other token and decide which ones matter for interpreting it.
Mechanically, each token's embedding is projected into three vectors: a query (what this token is looking for), a key (what this token offers), and a value (the information it passes along). The model scores every query against every key, normalizes those scores into weights, and uses the weights to blend value vectors into an updated representation of each token. Stack that operation across dozens of layers and thousands of attention heads, and the vector for "it" in a long paragraph ends up carrying information about which noun it refers to, forty tokens back.
The parameters you see in model names (8B, 70B, 405B) are the learned weights in these projection and feed-forward matrices. Parameter count correlates with capability, but it's an increasingly loose correlation: architecture choices like mixture-of-experts, training data quality, and post-training now matter as much as raw size.
Attention's catch is cost: compute scales quadratically with sequence length, and the memory traffic used to as well. IO-aware kernels like FlashAttention cut the memory side of that bill, and combined with advances in positional encoding they made today's context windows of hundreds of thousands of tokens practical.
How LLMs are trained
Training happens in stages, and the distinction matters when you're deciding how to adapt a model to your own use case.
- Pretraining is the expensive part. The model ingests a web-scale corpus (for open datasets, think RedPajama-v2 and its 30 trillion tokens) and learns by self-supervision: predict each next token from everything before it, measure the error, update the weights via backpropagation. No human labels anything. The corpus itself is the ground truth. What comes out is a base model: a raw next-token predictor with broad knowledge and no manners. Ask it a question and it may answer, continue your question, or produce three more questions like it, because all of those are plausible continuations of the text.
- Supervised fine-tuning (SFT) turns the base model into something usable. Train on curated prompt-response pairs and the model learns the shape of an assistant: answer the question, follow the instruction, stop when done.
- Preference tuning refines behavior that's hard to specify with examples alone: tone, honesty, refusal boundaries. The classic method here, reinforcement learning from human feedback (RLHF), trains a separate reward model on human rankings, then optimizes against it. Direct Preference Optimization (DPO) gets comparable results by training directly on chosen-vs-rejected response pairs, no reward model needed, and that simplicity has made it the default for teams tuning open models on their own preference data.
- Reasoning training is the newest stage. Models are trained, usually with reinforcement learning on verifiable problems, to produce intermediate reasoning steps before a final answer. This is what separates a "reasoning model" from a standard chat model on math, code, and multi-step planning tasks.
What actually happens at inference
Inference, the process of generating text from a trained model, is where LLMs live or die in production. It's also the part most explainers skip.
Generation is autoregressive: the model produces one token, appends it to the sequence, and runs again. A 500-token response means 500 sequential forward passes. Two phases with very different performance profiles:
- Prefill processes your entire prompt in parallel and is compute-bound. This is what you're waiting on before the first token appears.
- Decode generates tokens one at a time and is memory-bandwidth-bound: each step has to stream the model's weights from GPU memory. This determines the tokens-per-second rate you see during streaming.
The KV cache makes decode tractable: attention keys and values for previous tokens are stored rather than recomputed each step. The tradeoff is memory, because the cache grows with sequence length and batch size. Add the quadratic prefill compute on a long prompt, and long-context requests stay expensive even when the output is short.
Serving systems layer optimizations on top. Continuous batching keeps GPUs saturated across many concurrent requests, and speculative decoding lets a small draft model propose several tokens that the large model verifies in a single pass. Both are exact: they cut latency and cost without changing the output distribution. Quantization, the third common lever, compresses weights to 8-bit or 4-bit precision and is not exact. It buys memory and speed at a small, measurable quality cost, one worth verifying with evals before you rely on a quantized endpoint. Cheap, fast tokens are the whole ballgame for most production workloads.
The knobs you control at the API level operate on this loop. Temperature rescales the model's logits before they become probabilities: low values make the model pick the likeliest token nearly every time, higher values flatten the distribution and admit more variety. Max tokens caps the decode loop. Streaming just returns each token as the loop produces it.
Open vs. closed models
LLMs reach developers in two forms. Closed models (OpenAI's GPT family, Anthropic's Claude, Google's Gemini) are accessible only through the provider's API; the weights stay private. Open models publish their weights: the Llama, DeepSeek, Qwen, and Kimi families, among many others, can be downloaded, inspected, fine-tuned, and served anywhere.
The capability gap between the two has narrowed a lot over the past two years. What remains is a set of deployment questions, and open weights answer several that a closed API leaves open:
- Control: Pin a model version permanently. No silent upgrades changing behavior under your application.
- Customization: Full fine-tuning access: SFT, DPO, continued pretraining on domain data.
- Economics and portability: Serve on shared serverless infrastructure while traffic is small, move to dedicated GPUs when utilization justifies it, or self-host if compliance demands it.
The tradeoff: someone has to run the thing well. Fast, cheap open-model inference is its own engineering discipline, which is roughly the reason Together AI exists.
What LLMs get wrong
A next-token predictor has failure modes that follow directly from its design. Knowing them changes how you build.
- Hallucination: The model always produces a plausible continuation, whether or not a true one is available. It exposes no reliable signal distinguishing "I know this" from "this pattern-matches." The mitigations live in the system around the model: retrieval-augmented generation (RAG) to ground answers in real documents, constrained outputs, verification passes.
- Stale knowledge: Weights encode the world as of the training cutoff. Anything after that has to arrive through the context window.
- Context isn't memory: The model has no state between calls. "Memory" in LLM products is engineering (prior conversation stuffed into the prompt), and attention over very long contexts degrades in quality ("lost in the middle"), so more context isn't automatically better.
- Training data in, training data out: A model reproduces the biases and blind spots of the text it learned from, and post-training reduces this without eliminating it. When outputs face users in consequential domains, bias evaluation belongs in the test suite next to accuracy.
- Arithmetic and precision: Token-by-token prediction is a poor calculator. Production systems route math and structured queries to tools and let the model orchestrate.
Every one of these failure modes is manageable, and managing them is why production systems wrap the model with retrieval, tools, and evaluation layers rather than shipping a bare chat completion.
Where to go from here
The fastest way to build real intuition is to run a few models side by side and watch how they differ on your own prompts. An 8B versus a 70B on the same task teaches you more about parameter scaling than any benchmark table. You can do that in a few minutes against Together AI's serverless inference API, which serves the major open-source model families on demand, with fine-tuning available when prompting stops being enough.
FAQ
What does "LLM" stand for? Large language model: "large" for the parameter count (billions to trillions of weights), "language model" for the objective of modeling the probability of text sequences.
Are LLMs just autocomplete? The training objective is next-token prediction, so mechanically, yes. But predicting text well at scale forces the model to internalize grammar, facts, code semantics, and reasoning patterns, because those are what make text predictable. So "autocomplete" is a fair description of the mechanism and a poor predictor of the ceiling.
What's the difference between an LLM and ChatGPT? An LLM is the model; ChatGPT is a product wrapping one: model plus chat interface, memory engineering, tools, and safety layers. The same distinction applies to any assistant built on an open model.
How is an LLM different from generative AI? Generative AI is the umbrella term for models that produce content of any kind: text, images, audio, video. LLMs are the text-focused branch, though the line is blurring as multimodal models handle several media types in one architecture.
.png)
.png)
.png)
.png)
