AI Concepts Glossary: A Principal Engineer's Reference

Every entry includes:

• What it is — plain-language explanation
• Why it matters — engineering and strategic relevance
• Examples — concrete instances
• vs. — what distinguishes it from related concepts
• Key insight — what this tells you about the direction of the industry or your own decisions

Jump to a section:

• Model Access & Licensing
• Training & Customization
• Architecture & Deployment
• Evaluation & Use
• Representations, Search & Agents

The single most important question when choosing an LLM is: do the weights leave the provider's servers? This determines your data sovereignty, compliance posture, cost model, and customization options.

Open Weight

What it is: The trained model parameters (weights) are publicly downloadable. You can run the model on your own hardware without calling a third-party API. The model's knowledge and capabilities are encoded in these weights.

Why it matters: Enables self-hosting, data sovereignty, privacy, and cost control. Once you have the weights, inference cost is pure compute — no per-token API fees. You can run it behind your firewall, and data never leaves your infrastructure.

Examples: LLaMA 3 (Meta), Mistral 7B/22B/8×22B, Gemma (Google), Qwen2.5 (Alibaba), Phi-3 (Microsoft), Falcon (TII)

vs. Proprietary/Closed Weight models (GPT-4, Claude, Gemini Ultra) — those are accessible only via API; you never touch the weights.

Key insight: The quality gap between open-weight and proprietary frontier models has narrowed dramatically in 2025-26. Open-weight is now a serious choice for production workloads, not just experimentation.

Open Source (True Open Source)

What it is: Goes beyond releasing weights. True open source includes the training code, datasets, evaluation scripts, and a permissive license (Apache 2.0, MIT) that allows commercial use and modification. The entire pipeline is reproducible.

Why it matters: Enables full auditability. Researchers can replicate results from scratch; engineers can study failure modes at the data level; organizations can verify there's no hidden bias or backdoor. This is the academic gold standard.

Examples: Pythia (EleutherAI), OLMo (AI2), BLOOM (BigScience — partial), Falcon (weights + partial code)

vs. Open Weight but not truly open source: LLaMA 3 releases weights but training data and exact code are restricted. Openness exists on a spectrum.

Key insight: Rare — most "open" models are open weight only. As regulatory scrutiny of AI increases (GDPR, EU AI Act), true open source will become a competitive differentiator for enterprise procurement.

Proprietary / Closed Weight

What it is: The model is hosted by the provider and accessed exclusively via API calls. Weights, architecture details, and training data are trade secrets. You pay per token or per subscription tier.

Why it matters: The highest-capability frontier models live here. No infrastructure overhead to manage — but you're dependent on the provider's availability, pricing, and policies.

Examples: GPT-4o (OpenAI), Claude 3.5/3 Opus (Anthropic), Gemini 1.5/2.0 Ultra (Google), Command R+ (Cohere)

vs. Open weight: you run it yourself. Proprietary: you call it over a network. The distinction is fundamental for data privacy and vendor lock-in risk.

Key insight: Fine for prototyping and general applications. Problematic for regulated industries with strict data residency requirements. Increasingly, enterprises require on-premises or VPC-deployed models.

Weights-Available / Source-Available

What it is: A middle ground — the weights are downloadable but the license limits commercial use, specific applications, or redistribution. Common for research models released "for research only."

Why it matters: Allows researchers to study models while protecting creator commercial interests. The catch: you cannot ship a product on a research-only licensed model without violating terms.

Examples: Early LLaMA 1 (non-commercial), some Stable Diffusion variants, certain academic models

vs. True open weight with Apache 2.0 license (fully commercial) vs. research license (academic use only). "Free to download" ≠ "free to deploy in production."

Key insight: Always read the license before building on a model. This is legal risk, not just technical risk. Use SPDX identifiers to track license terms systematically.

Understanding the training pipeline is what separates engineers who use AI from engineers who architect AI systems. You rarely run pre-training, but you need to understand where in the pipeline to intervene for your use case.

Pre-training

What it is: The foundational training phase where a model learns language by predicting the next token across trillions of tokens of raw text — web crawls, books, code, academic papers. This builds the base model's world knowledge, language understanding, and reasoning capabilities.

Why it matters: Pre-training is where 99% of compute and cost lives. It's why frontier models cost $50M–$500M+ to train. The resulting "base model" has broad knowledge but isn't aligned for conversation — it will complete prompts, not follow instructions.

Examples: GPT-4 pre-training (estimated ~$100M), LLaMA 3.1 trained on 15 trillion tokens, Gemma trained on web + code data (~6T tokens)

vs. Fine-tuning works on a pre-trained base. Without pre-training, there's nothing to fine-tune. The pre-trained base is the foundation — you're building on top of it.

Key insight: Choosing the right pre-trained base model is the highest-leverage decision in any LLM project. Everything downstream depends on it.

Fine-tuning / Post-training

What it is: Continuing training on a smaller, curated dataset to steer the model toward specific behavior, domain vocabulary, output formats, or task patterns. "Post-training" often refers specifically to the alignment phase (instruction following, safety) that follows pre-training.

Why it matters: Enables teaching a model domain-specific tone, terminology, output formats, or task patterns — with far less compute than pre-training. A 7B model fine-tuned on 10,000 high-quality domain examples will outperform GPT-4 on narrow in-domain tasks.

Examples: Fine-tuning LLaMA 3 on medical Q&A pairs, fine-tuning Mistral on customer support logs, instruction-tuning a base LLaMA to follow chat formats

vs. RAG retrieves external knowledge at inference time without changing the model. Fine-tuning bakes knowledge and behavior into the weights at training time. Both are complementary: fine-tune for behavior, RAG for knowledge.

Key insight: Fine-tune for consistent format, tone, or task behavior. Use RAG for frequently-updated or proprietary knowledge. The most capable production systems use both.

PEFT — Parameter-Efficient Fine-Tuning

What it is: A family of techniques that update only a small subset of a model's parameters during fine-tuning — or add small trainable adapter modules — rather than updating all billions of parameters. Dramatically reduces memory and compute requirements.

Why it matters: Full fine-tuning of a 70B model requires 8×A100 GPUs (640GB VRAM) and days of training, costing thousands of dollars. PEFT can fine-tune the same model on a single A100 in hours at <1% of the cost.

PEFT Methods (in order of prevalence):

Key insight: LoRA and QLoRA are the standard PEFT methods in 2025-26. If someone says "we fine-tuned a model," they almost certainly used one of these.

LoRA / QLoRA — Low-Rank Adaptation

What it is: LoRA freezes the original model weights and injects small trainable low-rank matrices (A and B) into the attention layers. The weight update is: ΔW = A × B, where A ∈ ℝ^{d×r} and B ∈ ℝ^{r×k} with rank r << d. You train only A and B, not the full weight matrix W. QLoRA adds quantization: the frozen base is loaded in 4-bit (NF4), making it fit in far less VRAM.

Why it matters: A 70B model in BF16 takes ~140GB VRAM. With QLoRA in 4-bit + LoRA adapters, the same model can be fine-tuned on 2×A100 80GB GPUs. Without QLoRA, that requires a cluster of 8-16 GPUs. This single technique democratized LLM fine-tuning.

Key hyperparameters:

• r (rank): 8, 16, 32, 64 — higher = more expressive but more params. Start with 16.
• lora_alpha: scaling factor, typically set to 2×r
• target_modules: which weight matrices to apply LoRA to (q_proj, v_proj, k_proj, etc.)
• lora_dropout: regularization, 0.05–0.1

Tooling: Hugging Face peft library, Axolotl, unsloth (2–5× faster training), LLaMA-Factory

vs. Full fine-tuning modifies all weights (higher quality ceiling, much higher cost). LoRA adds adapters without touching base weights. At inference time, LoRA weights can be merged back into the base for zero-overhead serving, or kept separate for easy swapping.

RLHF — Reinforcement Learning from Human Feedback

What it is: The process that turns a raw pre-trained base model into a useful assistant. Three stages:

1. SFT (Supervised Fine-Tuning): Train on human-written demonstration responses
2. Reward Model Training: Train a separate model to score responses based on human preference comparisons (A vs B)
3. RL Optimization: Use PPO (Proximal Policy Optimization) to optimize the chat model against the reward model

Modern variants have simplified this:

• RLAIF: Replace human annotators with AI feedback (using a capable model as judge)
• DPO (Direct Preference Optimization): Skips the reward model entirely — directly optimizes on preference pairs. Much simpler to train, increasingly preferred.
• Constitutional AI (Anthropic): Uses a written constitution + AI self-critique to guide alignment

Why it matters: Without RLHF or an equivalent, base models don't follow instructions reliably and may reproduce harmful content. RLHF is what makes a raw LLaMA a completion engine vs. LLaMA-Chat a useful assistant.

Key insight: You rarely run RLHF yourself — the infrastructure is complex and annotation is expensive. But understanding it explains why base models and chat models behave completely differently, and why "jailbreaks" work by bypassing the alignment layer.

The technical decisions that determine whether an LLM system is fast, affordable, and production-grade.

Parameters — Weights & Biases

What it is: A neural network is a mathematical function with millions to trillions of learnable numbers (parameters). Weights connect neurons; biases shift activation thresholds. Training adjusts these via gradient descent until the model makes accurate predictions. A "70B model" has 70 billion such numbers.

Why it matters: Parameter count roughly correlates with model capacity, but not linearly. Architecture quality, training data, and training compute all interact. More parameters require more VRAM and more compute per inference call.

Memory requirements (rough):

Key insight: Parameter count ≠ intelligence. Phi-3 Mini (3.8B) outperforms many 13B models on benchmarks due to better training data and architecture choices. Efficiency matters as much as scale.

Quantization — Quant / Q-versions

What it is: Compressing model weights by using fewer bits per parameter. FP32 (full precision) → FP16/BF16 (half precision) → INT8 (8-bit) → INT4 (4-bit). Each step cuts memory roughly in half with increasing (but often manageable) quality trade-off.

Quantization formats:

The GGUF naming convention: Q4_K_M = 4-bit quantization, K-means clustering method, Medium size variant. Q8_0 = 8-bit, simpler method.

Why it matters: Without quantization, a 70B model needs 140GB VRAM. With Q4, it needs ~35GB. Quantization is what makes running large models on accessible hardware possible.

Key insight: For RAG workloads with long contexts, prefer Q8 or Q5 — INT4 can degrade retrieval-grounded reasoning. Always benchmark on your actual task and eval set, not just general benchmarks.

MoE — Mixture of Experts

What it is: An architecture where the feed-forward network in each transformer layer is replaced by N "expert" sub-networks, with a learned router that selects only K of them per token (typically K=2 of N=8). The total parameter count is large, but inference cost is proportional only to activated experts.

The math: Mixtral 8×7B has 8 experts × 7B parameters each ≈ 47B total params. But only 2 experts activate per token → effective activated params ≈ 13B. You get 47B capacity at ~13B compute cost.

Why it matters: MoE models achieve better quality-per-FLOP ratios than dense models at the same compute budget. This is why GPT-4 (reportedly MoE), Mixtral, and DeepSeek-V3 punch above their weight class.

Trade-offs:

• Pro: Higher capacity, lower compute per token, more efficient at scale
• Con: More total VRAM needed (all experts must fit in memory), more complex to serve, expert load balancing is a known training challenge

Examples: Mixtral 8×7B and 8×22B (Mistral AI), Grok-1 (xAI, 314B total), DeepSeek-V2/V3, GPT-4 (rumored MoE)

Key insight: When comparing a 7B dense model and a 13B MoE at similar inference compute cost — the MoE typically wins on quality. Total VRAM requirement is the constraint to watch.

Inference

What it is: Running a trained model to generate output — distinct from training. You send a prompt (input tokens), the model runs a forward pass auto-regressively (one token at a time), and returns the completed sequence.

Two phases of inference:

• Prefill: Processing the entire input prompt. Parallelizable. Cost scales with prompt length.
• Decode: Generating output tokens one at a time. Sequential (each token depends on the previous). Cost scales with output length.

Key inference metrics:

Inference frameworks: vLLM (best GPU throughput, PagedAttention), Ollama (developer-friendly, local), llama.cpp (CPU + GPU, GGUF), SGLang (RadixAttention, great for agents), TGI (Hugging Face)

Key insight: For agentic workloads with long system prompts: prefill cost dominates — optimize with prefix caching. For streaming chat: TTFT matters most for perceived responsiveness. For batch processing: maximize throughput.

Context Window

What it is: The maximum number of tokens a model can process at once — its "working memory." Everything outside the context window is invisible to the model during that inference call. 1 token ≈ 0.75 words in English.

Context window sizes (2025-26):

Why it matters: Context window size determines how much history, retrieved documents, or code a model can reason over simultaneously. Longer context enables more capable agents but also more compute and cost.

The "lost in the middle" problem: Most models perform worse on information placed in the middle of very long contexts vs. the beginning or end. This matters significantly for RAG design.

vs. Context window = working memory (short-term). External memory (RAG, vector DBs, memory layers) = long-term memory. Production agents need both.

KV Cache

What it is: An optimization that stores the computed Key-Value attention tensors for previously processed tokens so they don't need to be recomputed when generating each new output token.

Why it matters: Without KV cache, generating 100 output tokens from a 10K token prompt requires 100 full forward passes over 10K tokens. With KV cache: only the new token is processed per step — roughly 100× speedup in the limit.

Advanced KV cache concepts:

• Prefix caching: Cache the KV state of a shared system prompt across multiple requests. Major savings when all requests share a long system prompt.
• PagedAttention (vLLM): Manages KV cache memory like OS virtual memory — reduces fragmentation, enabling higher batch sizes.
• RadixAttention (SGLang): Caches common prefix trees across requests — powerful for agentic workloads with shared prefixes.

Key insight: Prefix caching on long, repeated system prompts can reduce latency and cost by 30-50% in production deployments. OpenAI, Anthropic, and major inference frameworks all support it.

The vocabulary for measuring and using models — critical for building reliable systems rather than impressive demos.

Benchmarks — MMLU, GSM8K, HumanEval…

What it is: Standardized evaluation datasets with known correct answers, used to measure and compare model capabilities across specific skills.

Key benchmarks by capability:

Benchmark saturation warning: Models are increasingly trained to pass specific benchmarks. MMLU scores above 90% are nearly meaningless as a differentiator now. Always supplement public benchmarks with domain-specific evals built on your actual use case.

Key insight: Build your own eval set of representative questions with verified answers for your domain. A model that scores 85% on your eval beats one that scores 92% on MMLU for your application. Evals are among the most important engineering artifacts in an LLM project.

Perplexity

What it is: A mathematical measure of how well a language model predicts a test corpus. Formally: exp(H(p)), where H(p) is the cross-entropy loss. Conceptually: if perplexity is 10, the model is as uncertain at each step as if choosing uniformly from 10 equally likely options. Lower = better language model.

Why it matters: Perplexity is the intrinsic training metric — language model loss IS log perplexity. Used to:

• Compare base models on the same test set (same tokenizer required)
• Track domain adaptation during fine-tuning
• Evaluate when to stop training (perplexity on a held-out set)

Limitations:

• Not comparable across models with different tokenizers
• Low perplexity ≠ useful model (a model that memorizes training data has low perplexity but may be useless on novel inputs)
• Doesn't measure instruction-following, safety, or task performance

Key insight: Relevant when evaluating base models for fine-tuning. Low perplexity on your target domain text means the model has more prior familiarity with that language register — a better fine-tuning starting point.

Temperature & Sampling

What it is: Parameters that control how the model samples from its output probability distribution at each generation step.

Temperature (T):

• T = 0: Always pick the highest-probability token (greedy decoding). Deterministic. Reproducible.
• T = 1: Sample from the distribution as-is.
• T > 1: Flatten the distribution — more creative, more likely to hallucinate.
• T < 1: Sharpen the distribution — more conservative, more repetitive.

Other sampling parameters:

• Top-p (nucleus sampling): Sample only from the smallest set of tokens whose cumulative probability ≥ p. Typically 0.9. Cuts off long tails.
• Top-k: Sample only from the top K most probable tokens. Cruder than top-p.

Recommended settings by task:

Key insight: For any factual or compliance application — use temperature = 0. You want deterministic, reproducible answers. Temperature > 0 introduces variance that users perceive as inconsistency.

RAG — Retrieval-Augmented Generation

What it is: An architecture pattern where relevant documents are retrieved from an external knowledge base at inference time and injected into the model's context alongside the user query. The model generates a response grounded in those retrieved documents.

Standard RAG pipeline:

Query → Embed query → ANN search in vector DB
     → Retrieve top-K chunks → (Optional) Rerank
     → Inject into context → LLM generates answer → Return with citations

Advanced RAG variants:

Key eval metrics for RAG:

• Retrieval precision: Of the top-K retrieved chunks, what fraction are actually relevant?
• Retrieval recall: Of all relevant chunks in the corpus, what fraction were retrieved?
• Answer faithfulness: Is the answer actually grounded in the retrieved documents — not hallucinated?
• Answer relevance: Does the answer actually address the question asked?

vs. Fine-tuning bakes knowledge into weights (static, expensive to update). RAG retrieves at runtime (dynamic, cheap to update). Use RAG for knowledge, fine-tuning for behavior.

Key insight: Retrieval quality is the bottleneck in most RAG systems — invest in embedding model choice, chunking strategy, and reranking before optimizing the LLM itself.

Hallucination

What it is: The model generates fluent, confident, plausible-sounding text that is factually incorrect, fabricated, or unsupported by the provided context.

Types of hallucination:

• Intrinsic: Contradicts the provided source documents
• Extrinsic: Cannot be verified against any source (fabricated from whole cloth)
• Conflation: Mixes real facts incorrectly (gets two real facts right but combines them wrong)

Why it matters: In compliance, legal, or medical applications, a model that confidently cites a nonexistent regulation is catastrophically worse than one that says "I don't know." Hallucination is the primary reliability risk for high-stakes AI deployments.

Mitigation stack:

1. RAG with citations — ground every claim in retrieved source text
2. High-quality retrieval — if retrieved docs are relevant, hallucination drops dramatically
3. Reranking — ensure top-3 docs are truly relevant before generation
4. Low temperature — deterministic generation reduces creative confabulation
5. Constitutional prompting — instruct the model to use only provided sources
6. Answer faithfulness checking — post-generation verification (NLI models, LLM-as-judge)
7. Human review gates — for high-stakes outputs, keep humans in the loop

Key insight: Never let a high-stakes system output claims without traceable citations. Build a faithfulness checker into your pipeline — a separate model that verifies each claim is supported by a retrieved chunk.

The vocabulary of the systems that surround and extend LLMs.

Embeddings

What it is: Dense vector representations of text in high-dimensional space, produced by embedding models. Semantically similar texts produce vectors that are geometrically close (measured by cosine similarity or dot product). This enables semantic search: finding documents similar in meaning to a query, not just matching exact words.

How embedding models work: A transformer encoder processes input text and produces a fixed-size vector (e.g. 1536 dimensions). The model is trained via contrastive learning — similar texts pulled together, dissimilar texts pushed apart in vector space.

Leading embedding models (2025-26):

vs. BM25 keyword search matches exact terms. Embedding search matches meaning. Hybrid search combines both — typically the best retrieval approach in practice.

Key insight: Chunk documents by token count using the specific tokenizer of your embedding model, not by word count. Model-specific tokenization significantly affects what fits in the embedding window.

Reranking

What it is: A second-pass scoring model (cross-encoder) that reads both the query and each candidate document together in full context, producing a precise relevance score. Used after initial embedding retrieval to re-sort and filter results before passing to the LLM.

Why two stages?

• Stage 1 (bi-encoder / embedding): Fast approximate search. ANN finds ~50-100 candidates. Optimizes recall.
• Stage 2 (cross-encoder / reranker): Slow but precise. Reads query + doc together. Optimizes precision. Pass top 3-5 to LLM.

Why cross-encoders are more accurate: Bi-encoders encode query and document independently then compare vectors. Cross-encoders see both simultaneously — they can model fine-grained lexical, semantic, and syntactic interactions between query and document.

Latency budget: A reranker call over 20 candidates takes 50–200ms depending on model size and hardware. Factor this into your pipeline latency budget.

Leading rerankers (2025-26):

• Cohere Rerank v3.5 — strong multilingual, domain-adaptive, API-based
• Jina Reranker v2 — open, strong English, self-hostable
• bge-reranker-large — open, excellent BEIR benchmark scores
• Flashrank — lightweight, local, good for development

Key insight: Reranking is the single highest-ROI optimization for RAG precision. Before adding architectural complexity elsewhere, add a reranker. The quality jump is consistently measurable across use cases.

Tokens & Tokenization

What it is: LLMs process text as tokens — subword units determined by a vocabulary learned during training via Byte-Pair Encoding (BPE) or similar algorithms. Common words are single tokens; rare or technical words split into multiple tokens.

Rules of thumb:

• 1 token ≈ 0.75 words (English)
• 1 token ≈ 4 characters (English)
• Code tokenizes more efficiently than prose
• Non-Latin scripts (Chinese, Arabic, Hindi) use more tokens per word

Why tokenization matters for engineering:

• API cost is per token — token economics are your cost model
• Context window is measured in tokens — not words
• Chunking for RAG must align to token boundaries (use the model's tokenizer, not word count)
• Different models have different vocabularies — a token in GPT-4 ≠ a token in LLaMA 3

Key insight: Use tiktoken (OpenAI) or tokenizers (Hugging Face) to count tokens programmatically before building any system that depends on staying within context limits. Assume 1.3× your word count estimate as a safety buffer.

Vector Database

What it is: A database specialized for storing high-dimensional embedding vectors and performing efficient approximate nearest-neighbor (ANN) search — finding the K vectors most similar to a query vector, even across billions of stored vectors.

Core operations:

• Upsert: Store a vector + associated metadata (document text, source, date, etc.)
• Query: Find top-K most similar vectors to a query vector
• Filter: Narrow the search space by metadata before vector comparison (e.g., "only search documents from 2024")

ANN index types:

• HNSW (Hierarchical Navigable Small World): Best recall/speed trade-off — used by Qdrant, Weaviate
• IVF (Inverted File Index): Better for very large datasets — used by Faiss
• PQ (Product Quantization): Compresses vectors for memory efficiency

Vector DB comparison:

Key insight: Metadata filtering (narrowing the vector search space by document attributes before ANN search) is critical in production RAG. Not all vector DBs handle this equally — Qdrant and pgvector are strongest here.

Agentic AI / AI Agents

What it is: Systems where an LLM acts as a reasoning engine that takes multi-step actions — calling tools, querying databases, running code, browsing the web — in a loop to complete complex tasks. The model decides what to do, interprets results, and plans next steps.

Agent components:

• LLM core: Reasoning and planning
• Tools: Functions the model can invoke (search, code execution, API calls, DB queries)
• Memory: Short-term (context window), long-term (vector DB, memory layers)
• State management: Tracking what's been done across steps
• Orchestration: LangGraph, LangChain, CrewAI, AutoGen

Agent patterns:

Failure modes to engineer around:

• Infinite loops (add max iterations and exit conditions)
• Tool hallucination (model calls tools that don't exist or with wrong parameters)
• State corruption (poorly managed state across long agent runs)
• Context overflow (long agent runs exhaust the context window)
• Determinism collapse (subtle errors compounding over many steps)

Key insight: Production agents require explicit state management, resumability, and human-in-the-loop injection points. LangGraph with checkpointing is the current standard for production-grade agentic systems. Understand your failure modes before deploying.