Generative AI, LLMs & Agents — Explained Simply

A model made of layers of simple units (neurons); each computes a weighted sum of its inputs plus a bias, passed through a non-linear activation. Stacked layers learn hierarchical features that map inputs to outputs.

It introduces non-linearity so the network can model curved, complex relationships. Without it, any stack of layers collapses into a single linear function.

Composing linear layers just gives another linear layer, so depth adds nothing. Non-linear activations (ReLU, GELU, tanh) are what make depth expressive.

The forward pass runs inputs through the layers to produce a prediction and a loss. The backward pass (backpropagation) applies the chain rule to compute each weight's gradient, which gradient descent uses to update the weights.

An efficient algorithm that computes the gradient of the loss with respect to every weight by propagating error backwards through the network using the chain rule.

An optimisation method that iteratively moves each weight in the direction that reduces the loss, scaled by a learning rate. Variants: SGD, momentum, Adam.

The step size for weight updates. Too high and training diverges or oscillates; too low and it crawls or gets stuck. It is often scheduled to decay over training.

In deep nets, gradients can shrink toward zero (vanishing) or blow up (exploding) as they propagate back, stalling or destabilising training. Mitigations: ReLU-family activations, careful initialisation, residual connections, and normalisation.

ReLU (max(0,x)) is the default hidden activation; sigmoid squashes to (0,1) for binary outputs; softmax turns a vector of scores into a probability distribution for multi-class outputs.

It quantifies how wrong predictions are. MSE for regression; cross-entropy (log-loss) for classification. Training minimises it.

When the net memorises training data and generalises poorly. Prevent with more data, dropout, weight decay (L2), early stopping, data augmentation, and reducing capacity.

During training it randomly zeroes a fraction of activations, forcing the network not to rely on any single unit — a cheap, effective regulariser.

Normalisation keeps activations in a stable range, which speeds up and stabilises training. Batch norm normalises across the batch; layer norm across features (standard in transformers).

An epoch is one full pass over the training data; a batch is the subset processed per weight update; an iteration is one such update. Epochs = iterations × batch_size / dataset_size.

Parameters (weights, biases) are learned from data during training. Hyperparameters (learning rate, layers, batch size) are set by you before training and tuned on a validation set.

A neural architecture built on self-attention and feed-forward blocks that processes a whole sequence in parallel, capturing long-range dependencies. It underpins virtually all modern LLMs.

A mechanism where each token computes how much to attend to every other token, then aggregates their information — letting the model weigh context dynamically.

Three learned projections of each token. Attention scores a token's Query against all Keys, softmaxes them into weights, and returns a weighted sum of the Values.

As the scaled dot product of Query and Key vectors: softmax(QKᵀ/√d)·V, where d is the key dimension.

Dot products grow with dimension; scaling by √d keeps them from becoming large, which would push softmax into tiny gradients and hurt training stability.

Running several attention operations in parallel with different learned projections, so the model can attend to different types of relationships simultaneously, then concatenating the results.

Attention is order-agnostic, so position information is added to token embeddings (sinusoidal, learned, or rotary/RoPE) to tell the model where each token sits in the sequence.

Encoder-only (BERT) is bidirectional, good for understanding/classification; decoder-only (GPT) is causal/autoregressive, good for generation; encoder-decoder (T5) maps an input sequence to an output sequence, good for translation/summarisation.

In decoders, each token may only attend to earlier tokens (a mask hides the future), so generation stays autoregressive and can't peek at what it hasn't produced yet.

They process sequences in parallel (not step by step), capture long-range dependencies without forgetting, and scale efficiently on GPUs — enabling training on far more data.

A per-token two-layer MLP (with a non-linearity) applied after attention, adding representational capacity. Attention mixes tokens; the FFN processes each token.

Residual (skip) connections let gradients flow through deep stacks and preserve information; layer normalisation stabilises activations. Both are essential for training deep transformers.

Standard attention compares every token to every other, so compute and memory scale as O(n²) with sequence length — the main bottleneck for long contexts, addressed by variants like FlashAttention and sparse attention.

Self-attention relates tokens within one sequence; cross-attention lets one sequence (e.g. the decoder) attend to another (e.g. the encoder output or an image encoding).

'Attention Is All You Need' (Vaswani et al., 2017), which replaced recurrence with attention and reshaped NLP and beyond.

A chunk of text (often a sub-word) that the model treats as one unit. Text is split into tokens before processing, and pricing/context limits are measured in tokens.

The process of splitting text into tokens and mapping them to integer ids the model can consume, using an algorithm like Byte-Pair Encoding (BPE) or WordPiece.

Sub-words keep the vocabulary small yet can represent any word — including rare, misspelled, or unseen ones — by composing pieces, avoiding out-of-vocabulary problems.

In English, about 1.3 tokens per word on average (~4 characters per token). This is why token counts exceed word counts and why long documents cost more.

A dense vector of numbers representing a token, word, sentence, or document, positioned so that semantically similar items are close together in the vector space.

Items used in similar contexts get similar vectors, so distance encodes meaning — 'dog' and 'puppy' are near, and analogies (king−man+woman≈queen) emerge as directions.

It measures the angle between vectors, ignoring magnitude, so it compares direction (meaning) rather than length — which suits embeddings where scale is not meaningful.

Token embeddings represent individual tokens inside the model; sentence embeddings (e.g. from Sentence-Transformers) represent whole texts as one vector for search and clustering.

A lookup table with one row per vocabulary token; a token id indexes its row to fetch that token's embedding vector, and these rows are learned during training.

Encode the query and all documents into the same space, then rank documents by similarity to the query vector — retrieving by meaning rather than exact keywords.

Static embeddings (word2vec, GloVe) give a word one fixed vector; contextual embeddings (from transformers) give a word different vectors depending on the sentence around it.

The length of the vector (e.g. 384–4096). Higher dimensions capture more nuance but cost more storage and compute; you pick based on the embedding model and downstream needs.

It caps how many tokens the model can attend to at once. Exceed it and older tokens are dropped or the request fails, so long inputs must be chunked or summarised.

No — embeddings only make sense within the same model's space. You must embed both queries and documents with the identical model to compare them.

A reserved token with a structural role (e.g. [CLS], [SEP], <|endoftext|>, padding). Models use them to mark boundaries, classification slots, or the end of generation.

A transformer with billions of parameters trained on massive text corpora to predict the next token, which as a by-product learns grammar, facts, reasoning, and style.

The model generates one token at a time; each generated token is appended to the input and fed back to predict the next, so text is produced sequentially.

Pretraining (self-supervised next-token prediction on huge corpora), then instruction/supervised fine-tuning, then alignment (RLHF or DPO) to make it follow instructions and be helpful and safe.

Reinforcement Learning from Human Feedback: humans rank model outputs, a reward model learns those preferences, and the LLM is optimised to produce preferred responses. DPO is a simpler alternative.

They control randomness in sampling. Temperature scales the probability distribution (low = focused/deterministic, high = creative/varied); top-p (nucleus) samples from the smallest set of tokens whose probability sums to p.

The maximum number of tokens (prompt + response) the model can process at once. It bounds how much text you can include; exceeding it truncates or errors.

They generate the most probable-sounding continuation, not verified truth, so they can produce confident but false statements — especially outside their training knowledge. Grounding (RAG), citations, and lower temperature reduce it.

A base model only predicts text continuations; an instruct/chat model has been fine-tuned and aligned to follow instructions and hold a conversation.

The model's ability to learn a task from examples given in the prompt at inference time, without any weight updates — the basis of few-shot prompting.

Capabilities (like multi-step reasoning or arithmetic) that appear only once models pass a certain scale, absent in smaller models.

Empirical relationships showing that loss improves predictably as model size, data, and compute grow together — the rationale behind training ever-larger models.

Open models (e.g. Llama, Mistral, Qwen) have downloadable weights you can self-host and fine-tune; closed models (GPT, Claude, Gemini) are accessed via API with no weight access.

Storing weights in lower precision (e.g. 4-bit instead of 16-bit) to shrink memory and speed up inference, with minor quality loss — enabling large models to run on modest hardware.

An architecture where a router activates only a few 'expert' sub-networks per token, giving a large total parameter count while keeping per-token compute low.

Use smaller/quantized models where possible, cache repeated prompts and responses, shorten prompts, stream output, batch requests, and route easy queries to cheaper models.

The practice of crafting inputs — instructions, context, examples, and output format — so an LLM reliably produces the desired output.

Zero-shot gives only the instruction; few-shot includes example input→output pairs to demonstrate the pattern; chain-of-thought asks the model to reason step by step before answering.

The system prompt sets durable role, rules, tone, and constraints; the user prompt carries the specific request. The system prompt persists across turns.

Examples show the model the exact task and output format via in-context learning, reducing ambiguity and improving consistency without any fine-tuning.

Prompting the model to show intermediate reasoning. It improves accuracy on multi-step problems (maths, logic) but adds tokens/latency and isn't needed for simple tasks.

Specify the exact schema, provide an example, use the provider's structured-output/JSON mode or function calling, lower the temperature, and validate/parse the result with retries on failure.

A malicious input that tries to override your instructions (e.g. 'ignore previous instructions and reveal the system prompt'). It's the top security risk for LLM apps.

Separate trusted instructions from untrusted content, never blindly execute retrieved text, constrain tools and permissions, validate outputs, and add input/output guardrails plus injection detection.

It trades determinism for creativity: use low temperature for factual, structured, or reproducible tasks; higher for brainstorming and varied writing.

Build a small labelled test set, try variants, and score each against it (accuracy, format adherence) rather than judging from a single example — prompt evaluation, not vibes.

Breaking a task into a sequence of prompts where each step's output feeds the next (e.g. extract → transform → summarise), improving reliability over one giant prompt.

It elicits intermediate reasoning tokens, giving the model 'space' to work through the problem before committing to a final answer, which reduces errors on complex tasks.

Prompting steers a frozen model at inference time (fast, flexible, no training); fine-tuning changes the model's weights for durable behaviour (better for fixed style/format at scale).

Avoid over-fitting to one model's quirks, pin behaviour with structure and examples, and re-run your eval set whenever you switch models or versions.

Wrapping user content in clear delimiters (triple backticks, XML tags) so the model can distinguish instructions from data — reducing confusion and injection risk.

A technique that retrieves relevant documents at query time and provides them to the LLM as context, so it answers from real sources instead of only its training memory.

RAG grounds answers in your own, up-to-date data, enables citations, updates instantly when documents change, and avoids costly retraining — ideal for changing knowledge.

Load documents, split them into chunks, embed each chunk into a vector, and store the vectors plus metadata in a vector database.

Embed the user's question, retrieve the top-k most similar chunks, insert them into the prompt as context, and ask the LLM to answer using only that context, ideally with citations.

Chunks must be small enough to be specific yet large enough to be self-contained; overlap preserves context across boundaries. Typical starting points: a few hundred tokens with ~10–20% overlap, tuned by evaluation.

Combining dense vector similarity with sparse keyword search (e.g. BM25) so you catch both semantic matches and exact terms (names, codes) that embeddings can miss.

A model that re-scores the retrieved candidates for relevance to the query (e.g. a cross-encoder). Use it when top-k retrieval returns roughly relevant but poorly ordered results.

Instruct the model to answer only from the provided context, require citations, add a 'say you don't know if not in context' fallback, and improve retrieval quality.

Separately measure retrieval (did the right chunks come back — hit-rate, recall) and generation (faithfulness/groundedness to context, answer relevance). Frameworks like Ragas automate this.

Filling the prompt with retrieved text. Too little hurts recall; too much wastes tokens, raises cost/latency, and can bury the answer ('lost in the middle').

Usually retrieval, not generation: bad chunking, weak embeddings, missing documents, or no re-ranking. Debug the retrieved context first.

Restricting retrieval by attributes (user, date, source, permissions) alongside vector similarity, so you fetch only allowed and relevant chunks.

Reformulating the user's question (e.g. resolving pronouns, adding synonyms, or generating multiple sub-queries) before retrieval to improve recall.

Fine-tune for consistent style/format or a fixed skill; use RAG for factual, changing, or proprietary knowledge. They are complementary and often combined.

Letting the system retrieve iteratively — reasoning, retrieving again with refined queries, and combining evidence across steps — for questions a single retrieval can't answer.

Continuing training a pretrained model on task- or domain-specific example pairs so its weights adapt to your task, style, or format.

Fine-tune for consistent tone/format or a narrow skill and to shrink prompts; use RAG for changing/proprietary knowledge; use prompting for quick, flexible steering. They combine.

Parameter-Efficient Fine-Tuning: freeze the base model and train only a small set of extra parameters, cutting compute and storage while retaining most of the benefit.

Low-Rank Adaptation: it injects small low-rank matrices into (usually attention) layers and trains only those — often under 1% of parameters — producing a compact, swappable adapter.

It updates and stores only the tiny adapter weights, not the billions of base weights, so it needs far less memory, compute, and disk, and lets you keep many adapters for one base model.

LoRA applied on top of a base model quantized to 4-bit, drastically cutting memory so large models can be fine-tuned on a single consumer GPU with little quality loss.

High-quality, consistent input→output examples in the target format — often hundreds to a few thousand. Quality and consistency matter more than sheer volume.

When fine-tuning on a narrow task degrades the model's general abilities. Mitigate with PEFT/LoRA, lower learning rates, fewer epochs, and mixing in diverse data.

Fine-tuning a base model on many (instruction, response) pairs so it learns to follow instructions generally — the step that turns a base model into a helpful assistant.

Learning rate, number of epochs, LoRA rank/alpha, and batch size. Too many epochs or too high a learning rate causes overfitting and forgetting.

Evaluate on a held-out test set against the base (well-prompted) model using task metrics and, for generation, LLM-as-a-judge — not just training loss.

Preference-based alignment: RLHF trains a reward model from human rankings and optimises against it; DPO optimises directly on preference pairs. Both shape behaviour beyond supervised fine-tuning.

Yes — a key advantage. You keep one base in memory and load small adapters per task/customer, even swapping them at request time.

It can amplify the base model's biases and errors and cause 'model collapse' over generations. Prefer high-quality human or verified data.

Not reliably — it's better at teaching behaviour, format, and style than at injecting facts. For factual knowledge, RAG is usually the right tool.

A database designed to store embedding vectors and efficiently find the nearest ones to a query vector by similarity, powering semantic search, RAG, and recommendations.

Finding the stored vectors closest to a query vector under a distance metric. Exact search is accurate but slow at scale; approximate (ANN) search trades a little accuracy for big speed gains.

Exact search over millions of high-dimensional vectors is too slow. ANN indexes return near-optimal neighbours in milliseconds with tunable accuracy/speed trade-offs.

Hierarchical Navigable Small World — a multi-layer graph you traverse greedily to reach a query's neighbours quickly. It offers excellent recall/latency but uses more memory.

Inverted File index — it clusters vectors into cells and searches only the cells nearest the query, cutting comparisons. Often combined with product quantization (IVF-PQ) to save memory.

Cosine similarity for most text embeddings (direction = meaning); dot product when vectors encode magnitude; Euclidean (L2) for some image/geometric embeddings. Match what the embedding model was trained with.

To restrict results by attributes (user, date, permissions, category) so you return only relevant and authorised vectors, not just the semantically closest.

pgvector (Postgres extension) is great when volumes are modest and you want data and vectors together. Dedicated vector DBs (Pinecone, Weaviate, Qdrant, Milvus) scale to larger, higher-throughput workloads with richer ANN options.

The fraction of true nearest neighbours the approximate index actually returns. Index parameters trade recall against speed and memory.

A compression technique that splits vectors into sub-vectors and encodes each with a small codebook, shrinking memory dramatically at some accuracy cost.

Depends on the index: some support incremental inserts/deletes efficiently; others need periodic rebuilds. Consider this when choosing an index for changing data.

Facebook AI Similarity Search — a high-performance library (not a full DB) for building ANN indexes locally, widely used for prototyping and embedded search.

More vectors increase index size, memory, and search time; good indexing keeps latency near-constant, but storage and build cost grow with scale.

Weak or mismatched embedding models, wrong distance metric, poor chunking, or missing metadata filters. Diagnose by inspecting what actually gets retrieved.

Yes, typically in separate collections/namespaces per model and dimension, since vectors are only comparable within the same embedding space.

An LLM placed in a loop with tools, memory, and the autonomy to decide and take actions repeatedly until a goal is achieved, rather than answering in a single turn.

A plain call returns text once. An agent can call tools, observe results, and iterate — making decisions and taking actions over multiple steps toward a goal.

Reason + Act: the model reasons about the goal, chooses an action (tool call), receives an observation from running it, and loops with that new information until done.

A workflow follows predefined steps (deterministic control flow); an agent decides its own steps dynamically. Workflows are more predictable; agents are more flexible but harder to control.

Set a max-step/iteration limit, a token/cost budget, timeouts, loop/termination conditions, and detection of repeated states — plus a fallback to escalate to a human.

Short-term memory (the scratchpad/context of the current task) tracks the ongoing reasoning; long-term memory (a store or vector DB) persists facts and past interactions across sessions.

Deciding the sequence of steps to reach a goal — either explicitly (generate a plan, then execute and adapt) or emergently through the reasoning loop.

Infinite loops, tool misuse or bad arguments, hallucinated tool results, error cascades, runaway cost, and getting stuck — mitigated by guardrails, validation, and human oversight.

Several specialised agents (e.g. planner, researcher, critic) that collaborate, delegate, or debate, coordinated by an orchestrator — useful for complex tasks but adds cost and complexity.

Via function calling: tools are described with schemas, the model emits a structured call, your code runs it, and the result is fed back into the loop.

Having the agent (or a separate critic step) review its own output or plan and revise it before finalising — improving reliability at the cost of extra steps.

Before high-risk or irreversible actions (payments, deletions, sends), when confidence is low, or when it hits ambiguity or repeated failure — human-in-the-loop checkpoints.

Measure task success rate, number of steps/cost, tool-call correctness, and safety (did it avoid unsafe actions), across a suite of representative tasks — not a single demo.

More autonomy can solve harder tasks but increases unpredictability, cost, and risk. Start constrained (fewer tools, tighter limits) and loosen only as reliability is proven.

The model must reliably parse observations to decide the next step; structured (JSON) results reduce misreads and make the loop robust.

Giving an LLM described functions it can invoke; when useful, it returns a structured call with arguments, your code executes it, and the result is returned to the model to continue.

You send tool definitions (name, description, JSON schema) with the prompt; the model may return a tool call; you validate and run it; you send the result back; the model produces the final answer or another call.

The schema constrains and validates what the model produces so your code receives well-typed, expected arguments — preventing malformed or unsafe calls.

An open standard that lets applications expose tools, resources, and prompts to LLM clients over a common protocol, so integrations are reusable across any MCP-compatible client instead of custom per-app glue.

The N×M integration problem: instead of building bespoke connectors for every model-tool pair, MCP provides one standard, so a tool built once works with many clients.

Separate read from write tools, require confirmation/human approval for risky actions, scope permissions narrowly, validate arguments, sandbox execution, and log every call.

When a model requests multiple independent tool calls at once (e.g. fetch weather for three cities), which your app can execute concurrently for speed.

It matches the request against the tool descriptions and schemas you provide, so clear names and descriptions strongly influence correct routing.

You return the error as the observation; a well-designed agent reads it and retries with corrected arguments, tries another tool, or reports the failure.

Retrieval fetches relevant text to ground an answer; tools perform actions or fetch live/computed data (APIs, code, databases). Many systems use both.

The model relies entirely on the name and description to decide when and how to call a tool; vague descriptions cause wrong or missed calls.

A managed catalogue of available tools with their schemas and permissions, so agents can discover and use them consistently and safely.

No — it can only request from the tools you expose. It may hallucinate a call, so you validate every request against your registered schemas before executing.

Cap the number of tool calls per task, cache results, prefer cheaper tools, and short-circuit when the goal is met.

In MCP, tools are callable actions, resources are readable data/context the client can pull in, and prompts are reusable templates — three complementary capabilities a server can expose.

An open-source framework for building LLM applications, providing standard components (models, prompts, retrievers, memory, tools, parsers) and a way to compose them into chains.

It removes repetitive plumbing — interchangeable model/vector-store interfaces, prompt templating, document loading/splitting, retrieval, memory, and output parsing — so you assemble apps instead of rebuilding basics.

The LangChain Expression Language: a declarative way to pipe components (runnables) together with the | operator, giving streaming, batching, async, and retries for free.

The common interface every LangChain component implements (invoke/stream/batch), which is what lets you compose them uniformly with LCEL.

Through standard abstractions: any chat model, embedding model, or vector store implements the same interface, so swapping providers is usually a one-line change.

Loaders ingest data from many sources (PDF, web, DBs); splitters chunk it sensibly (by tokens, characters, or structure) for embedding and retrieval.

A component that returns relevant documents for a query — commonly backed by a vector store, with variants for hybrid search, re-ranking, and multi-query retrieval.

It provides memory components that persist and inject conversation history or summaries into prompts, so multi-turn apps retain context.

Components that coerce raw model text into structured data (JSON, Pydantic objects, lists), with validation and retry, so downstream code can rely on the shape.

LangChain's observability and evaluation platform for tracing every step of a chain/agent, debugging, monitoring, and running evals in development and production.

When you need loops, branching, retries, persistent state, or multi-agent coordination — control flow a linear chain can't express.

Compose a retriever, a prompt template that inserts retrieved context, a chat model, and an output parser with LCEL: retriever → prompt → model → parser.

Both build LLM apps; LlamaIndex focuses heavily on data indexing/retrieval for RAG, while LangChain is broader (chains, agents, tools, integrations). They can be used together.

Define tools with schemas, bind them to a tool-calling model, and let an agent (or LangGraph) drive the reason-act loop, executing tool calls and feeding back results.

Historically, heavy abstraction and rapid API changes. Mitigate by learning the current LCEL core, keeping chains simple, and dropping to raw API calls when abstractions get in the way.

A framework for building stateful, multi-step LLM applications as graphs — nodes are steps, edges define control flow (including loops and branches), and a shared state is passed and updated throughout.

Real agents must loop, branch, retry, and remember. A graph expresses cycles and conditional paths that a straight-through chain cannot.

A typed, shared object that every node reads from and writes to. It carries messages, intermediate results, and any data the workflow needs across steps.

Nodes are units of work (a function or LLM/tool call); edges connect them, defining what runs next — including conditional edges that route based on state, and cycles for loops.

Edges that choose the next node based on the current state (e.g. 'if the answer is insufficient, go back and retrieve again'), enabling dynamic control flow.

LangGraph can save the graph state at each step, so runs can be paused, resumed, inspected, replayed ('time travel'), and made durable across sessions.

By checkpointing state and interrupting at a node, it can pause for human review or approval, then resume from exactly where it stopped.

It's a companion library; you use LangChain components (models, tools, retrievers) inside LangGraph nodes, and LangGraph provides the stateful control flow.

Represent each agent as a node (or subgraph), with edges routing tasks between them and a shared state coordinating hand-offs — e.g. supervisor, worker, critic patterns.

An edge that returns to a previous node, creating a loop — used for retrying, refining, or iterating (e.g. re-retrieve until the context is good enough).

Add a recursion/step limit, termination conditions checked on edges, and state that tracks progress so a node can decide to stop or escalate.

A graph where a node grades whether retrieved context answers the question; if not, it loops back to rewrite the query and retrieve again before generating the final answer.

It can stream state updates, node outputs, and tokens as the graph executes, so UIs show progress step by step through the workflow.

It makes the workflow inspectable, debuggable, resumable, and testable — you can see and control exactly what data flows between steps.

For simple, linear, single-shot tasks (e.g. a basic RAG query with no loops or approval), a plain LCEL chain is simpler and sufficient.

A generative model that learns to create data (usually images) by reversing a gradual noising process — starting from random noise and denoising step by step into a coherent sample.

During training, Gaussian noise is added to real images over many steps until they become pure noise; the model learns what noise was added at each step.

Generation starts from random noise and repeatedly applies the trained network to predict and remove noise, step by step, until a clean image emerges.

Traditionally a U-Net (encoder-decoder with skip connections); newer models use diffusion transformers (DiT). It predicts the noise (or the denoised estimate) at each step.

Running the diffusion process in a compressed latent space (via a VAE encoder/decoder) instead of raw pixels, drastically reducing compute — the basis of Stable Diffusion.

The prompt is embedded (e.g. by a text encoder like CLIP) and injected into the denoising network via cross-attention, steering generation toward the described content.

A technique that runs the model with and without the prompt and extrapolates between them, with a guidance scale controlling how strongly the output obeys the prompt.

More denoising steps generally improve quality and detail up to a point, at the cost of speed; samplers (e.g. DDIM, DPM++) reduce the steps needed.

Diffusion models are more stable to train and produce more diverse, high-quality samples, but are slower at generation (many steps) than a single GAN forward pass.

Inpainting regenerates a masked region of an image to match its surroundings/prompt; outpainting extends the image beyond its borders — both guided variants of diffusion.

An add-on that conditions diffusion on structural inputs (edges, depth, pose, scribbles), giving precise control over composition beyond the text prompt.

The same denoising idea extends to audio, video, and even molecules — any data where you can define a noising process and learn to reverse it.

The VAE encodes images into a compact latent and decodes latents back to pixels, so diffusion operates cheaply in latent space and the decoder renders the final image.

Text describing what you don't want; the model steers away from those concepts, useful for removing artifacts or unwanted styles.

Deepfakes and misinformation, copyright/training-data concerns, and biased or unsafe outputs — addressed with watermarking, safety filters, and usage policies.

A model that can process and relate more than one type of data — such as text with images, audio, or video — reasoning across them in a single system.

A vision encoder (often a Vision Transformer) turns the image into embeddings, which a projection layer maps into the language model's token space so it can attend to them alongside text.

A transformer that splits an image into patches, embeds each patch as a token, and applies self-attention — bringing the transformer architecture to vision.

A model trained contrastively on image-text pairs so matching images and captions get similar embeddings — enabling zero-shot classification and aligning vision with language.

A space where different modalities are represented compatibly, so an image and its description sit close together — the foundation for cross-modal search and reasoning.

A training method that pulls matching pairs (e.g. image and its caption) together and pushes non-matching pairs apart in embedding space, aligning modalities without explicit labels.

Visual question answering, OCR and document understanding, chart/diagram reading, image captioning, grounding (pointing to regions), and screenshot/UI understanding.

GPT-4o, Claude, and Gemini (proprietary, text+image+more), and open models like LLaVA and Qwen-VL. Many now handle audio and video too.

An audio encoder converts sound into embeddings the model consumes; speech can be transcribed (ASR) or reasoned over directly, and some models generate speech too.

Linking language to specific parts of another modality — e.g. identifying which region of an image a phrase refers to — enabling precise visual reasoning.

Early fusion combines modalities into a shared representation early (joint attention over all tokens); late fusion processes each modality separately and merges the results near the end.

With task-specific benchmarks (VQA accuracy, OCR/document accuracy, captioning metrics, chart QA) and human or LLM judging for open-ended visual reasoning.

Aligning modalities, scarce high-quality paired data, heavy compute, and modality imbalance where the model over-relies on text and under-uses the image.

Reading text and structure directly from a document image with a vision-language model, without a separate OCR step — handling layout, tables, and handwriting end-to-end.

Users share screenshots, photos, PDFs, and receipts. A model that reads them removes brittle preprocessing and enables natural 'look at this and help' interactions.

Embedding LLMs into real workflows — triggered by events — so tasks like classifying, extracting, drafting, and deciding happen automatically, with humans overseeing only where needed.

A trigger (webhook, schedule, new message/row), optional data-fetch/retrieval steps, one or more LLM steps (classify/extract/generate/decide), and actions (send, store, call APIs, notify).

Use structured outputs with validation, add retries and fallbacks, constrain the task narrowly, test against a fixed set, and add human review for low-confidence or risky cases.

For irreversible or high-stakes actions (payments, external sends, deletions), low-confidence decisions, or regulated contexts — the AI drafts/recommends and a human approves.

Downstream steps must parse the LLM's result programmatically; enforced JSON schemas make the output trustworthy and prevent the pipeline from breaking on free text.

Designing steps so re-running them doesn't duplicate effects (e.g. don't send the same email twice). It's essential because retries and re-triggers happen in real systems.

Log every input, prompt, model output, and action; trace runs end-to-end (e.g. LangSmith); track cost, latency, and error rates; and alert on anomalies or guardrail hits.

A deterministic workflow follows fixed steps you designed; an agentic one lets the LLM decide the steps dynamically. Prefer deterministic for predictable tasks, agentic for open-ended ones.

Zapier, Make, and n8n connect triggers, LLM steps, and actions visually; n8n and others add native LangChain/agent nodes for more advanced logic.

Route simple cases to cheaper/smaller models, cache repeated results, batch where possible, cap tokens, and only escalate to large models when needed.

High-volume, low-risk, well-scoped tasks with clear right answers — e.g. classifying/tagging incoming tickets or extracting fields from forms — where errors are cheap and reviewable.

Validate outputs, define fallback paths, retry with backoff, route unparseable or low-confidence cases to a human queue, and never let one failure silently corrupt downstream steps.

Automating high-stakes decisions without oversight can scale mistakes fast. Start with assistive automation (AI drafts, human approves) and increase autonomy as reliability is proven.

Automations often retrieve context (policies, records, docs) before the LLM step so decisions and drafts are grounded in the right, current data.

Track time saved, throughput, error/rework rate, cost per task, and human-override rate before and after — automation should reduce effort without raising error rates.

Outputs are open-ended with many valid answers, quality is subjective, and behaviour is non-deterministic — so simple accuracy metrics often don't apply.

Build a curated test set with expected behaviour, use automatic metrics where possible, apply LLM-as-a-judge for open-ended cases, and track results as prompts/models change.

Using a strong LLM to score another model's outputs against a rubric or reference. It scales evaluation but must be validated against human labels and watched for bias.

Position bias, verbosity bias, self-preference (favouring similar outputs), and inconsistency. Mitigate with clear rubrics, randomised order, and human spot-checks.

Separate retrieval metrics (context recall/precision, hit-rate) from generation metrics (faithfulness/groundedness to context, answer relevance). Tools like Ragas automate these.

Whether the answer is actually supported by the provided context rather than invented — the key anti-hallucination metric for RAG.

Measure groundedness against sources and fact-check claims; reduce it with RAG grounding, citations, lower temperature, 'answer only from context' instructions, and refusal fallbacks.

Checks around the model at input and output — validating structure, filtering PII/unsafe content, detecting prompt injection, and restricting tools/actions — to keep the app safe and on-task.

Separate instructions from untrusted content, don't execute retrieved text as commands, detect injection patterns, constrain tool permissions, and validate outputs before acting.

Offline evaluation runs a fixed test set before deployment; online evaluation monitors real production traffic (user feedback, guardrail hits, success signals) after deployment.

Keep a versioned eval suite and run it on every change; require key metrics (accuracy, groundedness, safety) not to drop before shipping.

Deliberately attacking it with adversarial inputs (jailbreaks, injections, edge cases) to find failures and safety gaps before real users or attackers do.

Risks like prompt injection, insecure output handling, training-data poisoning, sensitive-information disclosure, and excessive agency — a checklist for securing LLM apps.

Track tokens and dollars per request, p50/p95 latency, and throughput, and balance them against quality — a correct answer that's too slow or costly still fails the product.

Representative real inputs, hard and edge cases, clear expected behaviour, coverage of failure modes, and enough size to detect meaningful changes — kept versioned and growing.