Understanding GPU Architecture
for AI
The machine, the metrics, and the bottleneck.
GPUs don’t just run AI — they define its speed limit. But peak TFLOPS on a spec sheet is not what your model actually sees. To close the gap between theoretical performance and real throughput, you need to understand how a GPU actually executes work: its hierarchy of parallelism, its scheduling model, and the three metrics most engineers misread.
A CPU has tens of cores optimised to execute a single thread as fast as possible. A modern AI GPU has tens of thousands of simpler cores designed to execute hundreds of thousands of threads simultaneously. The H100 contains 16,896 CUDA cores across 132 Streaming Multiprocessors, plus 528 fourth-generation Tensor Cores that perform matrix multiply-accumulate operations at a scale no FP32 pipeline can match. The A100 doubled the H100’s predecessor’s HBM memory bandwidth to 2 TB/s; the H100 pushed further to 3.35 TB/s.
That raw scale creates a system with a unique programming model — one where performance is determined not by raw clock speed but by how effectively your workload keeps thousands of parallel execution units busy simultaneously. Understanding this model is what separates engineers who get 40% of theoretical peak performance from those who get 85%. This article walks through three layers: the machine itself, the metrics used to evaluate it, and the three bottleneck misconceptions that cause the most performance loss in production AI systems.
NVIDIA H100 Architecture 2023
NVIDIA Hopper Architecture
CUDA Programming Guide 2025
NVIDIA Datasheet 2023
“Tensor Cores — not CUDA cores — carry the primary workload for AI training and inference in modern NVIDIA GPUs. CUDA cores handle the surrounding general-purpose compute. Understanding which operations hit which cores is the first step toward real performance.”
— Wevolver, Understanding NVIDIA CUDA Cores: A Comprehensive Guide for 2026
A GPU is not a single fast processor — it is a massively parallel machine built from a hierarchy of execution units, each level designed to keep the level below it saturated with work. Understanding this hierarchy is the prerequisite for understanding every performance metric and every bottleneck.
GPU — The Top of the Hierarchy
At the top level, the GPU is a collection of Streaming Multiprocessors connected by a high-speed crossbar to shared L2 cache and HBM (High Bandwidth Memory). The H100 SXM5 contains 132 SMs, 50 MB of L2 cache, and 80 GB of HBM3 with 3.35 TB/s of bandwidth. Thread blocks from your CUDA kernel are distributed across SMs by the hardware scheduler — you write the code that describes what threads do; the hardware decides which SM they run on.
This hardware scheduling is the foundational design choice that makes GPU programming different from CPU programming. You do not pin work to specific execution units. You write massively parallel code and trust the hardware to fill the machine. The GPU architecture is designed to keep as many SMs busy simultaneously as possible — which is why launching enough work to fill all SMs is one of the most fundamental performance requirements.
Streaming Multiprocessor (SM) — The Worker
Each SM is a self-contained compute unit that can have many warps resident simultaneously. The H100 SM can hold up to 64 active warps per processing block and manages scheduling, register files, shared memory, and execution units independently from other SMs. The ability to hold many resident warps is the key to latency hiding — when one warp stalls waiting for memory, the SM’s warp scheduler immediately switches to another ready warp, keeping execution units busy.
Warp — The Scheduling Unit
A warp is a group of 32 threads that execute the same instruction in lockstep — this is the SIMT (Single Instruction, Multiple Threads) model. All 32 threads run the same instruction each cycle, but each thread operates on different data. This design maximises hardware utilisation when all 32 threads take the same code path, but creates a critical problem when they diverge.
Warp Divergence — the hidden serialisation cost
When threads within a warp take different branches (an if/else where some threads go one way and others go another), the GPU must serialise both paths — executing path A with path-B threads masked off, then executing path B with path-A threads masked off. A warp that diverges 50/50 effectively loses half its throughput. This is why neural network kernels are written to minimise branching, and why operations like softmax and layer norm require special attention in CUDA implementations.
Pipelines — What Actually Executes
Each SM contains several execution pipelines that run in parallel, each handling a different instruction type. Keeping all pipelines busy simultaneously is the goal of high-performance kernel design.
The aim is to keep all pipelines busy — a kernel that only uses Tensor Cores leaves CUDA cores idle; a kernel that only does memory operations leaves compute pipelines idle. Real AI workloads mix instruction types, and the best kernels schedule them to overlap their execution.
Three numbers dominate GPU performance conversations — occupancy, utilisation, and throughput. All three are real and useful. All three are routinely misread in ways that lead to wrong optimisation decisions. Understanding what each actually measures is the prerequisite for using profiler output correctly.
The ratio of active warps to the maximum number of warps an SM can hold. More warps resident means more opportunities to hide memory latency by switching to a ready warp when another stalls.
Determined by: register usage per thread, shared memory per block, and thread block size — all of which limit how many warps fit simultaneously.
The fraction of time the GPU is doing useful work across all SMs. Measured by nvidia-smi, Nsight, or Prometheus GPU exporters. Sampled as an average over a time window.
High utilisation means the GPU stays busy overall — not that every execution unit is running at full capacity. A GPU busy with memory operations and idle compute units still shows high utilisation.
The amount of useful work completed per unit time. The right metric depends on your workload: TFLOPS for compute, GB/s for memory, tokens/sec or images/sec for end-to-end model performance.
The roofline model: know which bound you are hitting
Every kernel is either compute-bound (the GPU’s arithmetic throughput is the limit) or memory-bound (the bandwidth between HBM and the compute units is the limit). The roofline model plots your kernel’s arithmetic intensity (FLOPs per byte of memory traffic) against these two ceilings. Kernels to the left of the ridge point are memory-bound — adding more compute won’t help. Kernels to the right are compute-bound — reducing memory traffic won’t help. Most LLM attention kernels are memory-bound at inference time, which is why FlashAttention’s reduction in HBM reads is so impactful regardless of compute utilisation.
Three metrics dominate GPU performance conversations: occupancy, utilisation, and Tensor Core throughput. All three are real and useful. All three are routinely misread in ways that cause engineers to optimise for the wrong thing. Here are the three most common bottleneck misconceptions in production AI systems.
Occupancy measures how many warps are resident on an SM relative to the maximum. The intuition is sound: more resident warps give the scheduler more options to hide latency by switching to a ready warp when one stalls on memory. But occupancy is a means, not an end — and 100% occupancy is not the goal.
A kernel can have 100% occupancy and still be severely memory-bound. If all warps are stalling on HBM reads simultaneously, having many warps doesn’t help — they’re all waiting for the same bottleneck. Conversely, for a compute-bound workload where latency is low and the compute pipelines are the limit, fewer warps with more resources each (more registers, more shared memory) can be faster because each warp runs its arithmetic more efficiently.
The practical implication: target the occupancy level that keeps the execution units that matter for your workload saturated — not necessarily the maximum possible occupancy. NVIDIA’s Occupancy Calculator helps model the tradeoff between warp count and resource allocation, but the final answer always comes from profiling, not from maximising a single number.
Occupancy is particularly relevant for memory-bound kernels like attention at inference time: more resident warps mean more in-flight memory transactions, which can hide the latency of HBM reads. For compute-bound kernels like large GEMM operations, occupancy matters less than ensuring the Tensor Cores are kept continuously fed with data.
- 100% occupancy + memory-bound = still memory-bound
- High occupancy uses more registers → fewer resources per thread
- Fewer warps, more registers per thread can win in compute-bound scenarios
- Profile: check SM efficiency, not just occupancy percentage
- Use Nsight Compute’s Warp State Statistics to see where warps actually stall
GPU utilisation — the number you see in nvidia-smi or a monitoring dashboard — reports the percentage of time the GPU was executing a kernel, averaged across all SMs and over a sampling window (typically 1–100ms). “100% utilisation” means the GPU was executing a kernel the entire time. It says nothing about what the kernel was doing, or whether the right parts of the GPU were busy.
A kernel that is entirely memory-bound — where the HBM bandwidth is the bottleneck and compute units are sitting idle waiting for data — will show 100% GPU utilisation. The Tensor Cores are idle. The CUDA cores are idle. The LD/ST units are saturated. But the top-level metric reads as fully utilised, giving the false impression that the GPU is being used efficiently.
The practical implication: never use nvidia-smi utilisation as your primary performance metric for AI kernels. It tells you whether the GPU is busy — not whether it is busy doing the right things. Use Nsight Compute or Nsight Systems to break down utilisation by execution unit type. A well-optimised inference kernel shows high Tensor Core utilisation, moderate LD/ST unit activity, and a healthy ratio of compute instructions to memory instructions.
Kernel launch overhead, CPU-GPU synchronisation points, and data transfers between host and device all appear as utilisation gaps — the GPU is genuinely idle during these periods. Asynchronous data loading and CUDA streams reduce these gaps by overlapping data movement with compute.
- nvidia-smi reports kernel presence, not unit saturation
- Use Nsight Compute for per-unit breakdown
- Key metric: Tensor Core utilisation for matrix-heavy AI kernels
- Gaps: kernel launches, H2D/D2H transfers, sync points
- CUDA streams overlap data transfer with compute to fill gaps
Tensor Cores are the reason H100s cost what they do. A 4th-generation Tensor Core performs a 16×8×16 matrix multiply-accumulate (MMA) operation in a single clock cycle — an operation that would require thousands of FP32 multiply-add instructions on standard CUDA cores. For the matrix multiplications at the heart of transformer training and inference, Tensor Cores provide a throughput advantage measured in orders of magnitude.
But Tensor Cores are not a general-purpose accelerator — they accelerate specific matrix operations in supported precision formats (FP16, BF16, TF32, FP8 on H100, FP4 on Blackwell). Everything else in a neural network — elementwise operations (ReLU, GELU, dropout), layer normalisation, softmax, the QK^T / V operations in attention when matrices are small, embedding lookups — runs on standard CUDA cores. And for many inference workloads, these non-matmul operations account for a substantial fraction of total execution time.
The practical implication: the speedup from Tensor Cores depends heavily on how matmul-dominated your workload is. A transformer with large batch sizes and long sequences — where GEMMs dominate — can approach 70–80% of peak Tensor Core throughput. A transformer at batch size 1 — where sequence parallelism is limited and many operations are small — may see Tensor Core utilisation below 30%, with the remaining time on CUDA cores and the memory subsystem. Fused kernels (FlashAttention, fused layer norm + dropout) exist precisely to reduce the fraction of time spent in CUDA-core-bound operations by computing them alongside the matmul.
- Tensor Cores: FP16/BF16/TF32/FP8 MMA only
- Not covered: elementwise ops, softmax, LayerNorm, small matrices
- Batch size matters: large batches → more matmul-dominated
- Fused kernels reduce the CUDA-core fraction
- FlashAttention reduces HBM reads, not just compute
- Profile: check Tensor Core throughput vs. CUDA core throughput split
The profiling workflow that closes the gap
The sequence: (1) run nvidia-smi dmon or Prometheus GPU exporter to confirm the GPU is busy. (2) Run nsys profile to identify which kernels consume the most time and where the CPU/GPU gaps are. (3) Run ncu (Nsight Compute) on the top offending kernels to get per-unit breakdown: check Tensor Core throughput, memory throughput vs. bandwidth ceiling, and warp stall reasons. (4) Apply the right optimisation: for memory-bound kernels — reduce HBM traffic (FlashAttention, caching, quantisation); for compute-bound — maximise Tensor Core feed (larger tiles, async prefetch, tensor parallelism). (5) Repeat. The gap between what you see in nvidia-smi and what Nsight Compute shows is where the real performance work lives.
NVIDIA GPU Architecture Generations — AI Context
| Architecture | GPU | Tensor Core Gen | FP16 TFLOPS | HBM Bandwidth | Key AI Innovation |
|---|---|---|---|---|---|
| Volta (2017) | V100 | 1st gen | 125 | 900 GB/s | First Tensor Cores — FP16 MMA, 12× over FP32 on P100 |
| Turing (2018) | T4 | 2nd gen | 130 | 320 GB/s | INT8/INT4 Tensor Cores; efficient inference; DLSS |
| Ampere (2020) | A100 | 3rd gen | 312 | 2 TB/s (HBM2e) | TF32, BF16, structured sparsity (2:4), MIG, 80GB variant |
| Hopper (2022) | H100 | 4th gen | 989 | 3.35 TB/s (HBM3) | FP8 Transformer Engine, NVLink 4, Thread Block Clusters |
| Blackwell (2024) | B200 | 5th gen | 2,250+ | 8 TB/s (HBM3e) | FP4, AI-managed caching, GB200 NVL72 multi-chip scale-up |
GPUs Turn Matrices Into Progress — When You Understand the Machine
The H100’s 989 TFLOPS of FP16 throughput is a ceiling, not a guarantee. The gap between theoretical peak and real model performance is filled by the architectural realities covered in this article: the SM’s warp scheduling model and its latency-hiding strategy, the SIMT execution model and its divergence cost, the three misread metrics that cause engineers to optimise for the wrong thing, and the Tensor Core’s specialisation that makes it transformative for matmul-heavy workloads and irrelevant for everything else.
The engineers closing the largest performance gaps in 2026 are not the ones with the most GPU hours. They are the ones who profile at the right level of granularity — who know the difference between what nvidia-smi tells them and what Nsight Compute reveals, who know whether their kernel is compute-bound or memory-bound before choosing an optimisation strategy, and who understand that occupancy, utilisation, and throughput are each measuring different things and each pointing to different fixes.
Same hardware. Same model. Clearer understanding of the architecture — meaningfully better AI.
GPUs turn matrices into progress. Knowing how they do it is what separates benchmarks from production.