Latency vs Throughput vs Bandwidth vs Concurrency vs Parallelism

Added: 2026-05-20

[00:00:00]One latency downwards arrow tip rightwards time for one request to go and come back downwards arrow tip rightwards measured in milliseconds.

[00:00:07]Two, throughput downwards arrow tip rightwards how many requests the system processes per unit of time downwards arrow tip rightwards measured in requests per second. Three, bandwidth downwards arrow tip rightwards the max capacity of the pipe itself downwards arrow tip rightwards measured in bits per second bps. Four, concurrency downwards arrow tip rightwards how many requests are being handled at the same time downwards arrow tip rightwards measured in concurrent requests. Five, parallelism downwards arrow tip rightwards how many tasks are actually executing at the same time downwards arrow tip rightwards measured in parallel workers, cores, nodes downwards arrow tip rightwards you care when AI inference batch.