Inferencja i infrastruktura

Inference and ML systems · id systems

Uruchamianie, przyspieszanie i utrzymywanie modeli.

Running, accelerating and maintaining models.

In one sentence

To sposoby uruchamiania, przyspieszania i utrzymywania modeli na komputerach, serwerach oraz małych urządzeniach.

These methods run, accelerate and maintain models on computers, servers and small devices.

Problem it solves

Model odpowiada zbyt wolno, nie mieści się w pamięci albo trudno go niezawodnie udostępnić wielu osobom.

A model responds too slowly, does not fit in memory or is hard to serve reliably to many people.

What it does

Łączy dobór sprzętu, oszczędny zapis, planowanie zapytań, pomiary i kontrolę wersji, dopasowując działanie do zasobów oraz potrzeb użytkowników.

It combines hardware choice, compact representations, request scheduling, measurement and version control to match resources and user needs.

Everyday example

Uruchamiasz asystenta dla zespołu i sprawdzasz, jak zmieniają się czas odpowiedzi, pamięć oraz koszt przy większym ruchu.

You launch a team assistant and check how response time, memory and cost change as traffic grows.

This concerns you if you…

• chcesz uruchomić model na własnym sprzęcie
• odpowiedzi muszą działać szybko i powtarzalnie
• want to run a model on your own hardware
• need fast and reliable responses

Subtopics

• AI lokalnie i na urządzeniach (Edge and on-device AI): Running on limited hardware and without a permanent cloud.
  • Eksport modeli i środowiska uruchomieniowe na urządzeniach (Model export and on-device runtimes): Moving a trained model from a training framework into a lightweight runtime on a phone, browser or embedded device: exporting or converting it to a supported format and delegating compute to the CPU, GPU or NPU.
  • Modele językowe na telefonie i laptopie (On-device language models): Language models optimised to run locally on a phone or laptop, often using an NPU, quantisation and other memory-saving techniques. On-device processing can keep data local and support offline use; hybrid architectures may route selected requests to a cloud model.
  • TinyML — modele na mikrokontrolerach (TinyML on microcontrollers): Running neural networks on microcontrollers with kilobytes of memory and milliwatts of power: co-designing tiny architectures with a lean inference engine, and quantising to 8 bits or fewer.
• Akceleratory (AI hardware): Hardware for inference and training.
  • Model roofline: obliczenia czy przepustowość pamięci (Roofline model and arithmetic intensity): A simple performance model: attainable throughput is bounded by the minimum of peak compute and memory bandwidth multiplied by arithmetic intensity (operations per byte). Within the model's assumptions, it helps determine whether an operation is compute-bound or memory-bandwidth-bound.
  • Pamięć HBM i ściana pamięci (High-bandwidth memory and the memory wall): Accelerator compute grows faster than memory bandwidth, so moving weights and KV cache from HBM can become a bottleneck. This is especially important during LLM token generation, particularly at small batch sizes.
  • Połączenia między układami i sieć klastra (Accelerator interconnects and cluster networking): Ways accelerators exchange data: fast links within a server or pod (scale-up) and networking between servers (scale-out), using topologies such as torus or fat-tree. Their bandwidth and latency strongly affect how training and inference scale across many chips.
  • Wyspecjalizowane akceleratory AI (TPU, NPU) (Specialised AI accelerators (ASICs and NPUs)): Specialised chips or compute blocks optimised for neural-network operations, often using multiply-accumulate arrays and low-precision arithmetic. They include datacenter accelerators such as TPUs and NPUs integrated into phones and laptops.
• GPU i obliczenia rozproszone (GPU and distributed computing): Using accelerators and many devices.
  • Języki i kompilatory kerneli (Kernel languages and compilers): Writing kernels one level above raw CUDA: tiles, templates, compilers.
  • Kernele mnożenia macierzy (GEMM) (Matrix multiplication kernels (GEMM)): The heart of AI compute: squeezing near-peak hardware throughput out of matrix multiplication.
  • Komunikacja między GPU (Multi-GPU communication): Moving data between many GPUs: collective operations and topology.
  • Profilowanie i optymalizacja kerneli (Kernel profiling and optimisation): Measuring where time goes, and ways to win it back.
  • Programowanie GPU (CUDA) (GPU programming (CUDA)): The GPU execution model: threads, blocks, the memory hierarchy.
  • Rdzenie tensorowe i niska precyzja (Tensor cores and low precision): Matrix-multiply units and the number formats that feed them.
  • Strategie zrównoleglania treningu (Training parallelism strategies): How to split a model and its data across devices.
• Kwantyzacja i kompresja (Quantization and compression): Cutting memory and compute cost.
  • Kwantyzacja samych wag (Weight-only quantization): Storing weights in low precision with layer-by-layer calibration to keep quality at 4 bits and below.
  • Kwantyzacja wag i aktywacji (Weight and activation quantization): Lowering precision for activations too. Requires taming outlier values that break the scale.
  • Przycinanie wag i rzadkość (pruning) (Pruning and sparsity): Setting selected weights to zero to obtain a sparse model; pruning may be unstructured or follow a hardware-friendly pattern such as 2:4. The quality-sparsity trade-off depends on the model, calibration data and method; SparseGPT and Wanda are examples of one-shot LLM pruning without retraining.
• MLOps i LLMOps (MLOps and LLMOps): Keeping deployments repeatable and observable.
  • Automatyczne retrenowanie i CI/CD dla ML (Continuous training and CI/CD for ML): Automating an ML pipeline: retraining and redeploying a model after a defined trigger such as a schedule, new data, quality degradation or detected drift, while testing and deploying the pipeline itself like code.
  • Bramka LLM: jedno API do wielu modeli (LLM gateway (AI gateway)): An intermediate layer in front of many providers and self-hosted models: one unified API, central keys and access control, rate limits and budgets, fallbacks, load balancing and cost logging.
  • Magazyn cech (feature store) (Feature store): A shared feature platform for training and production: an offline store with feature history for building training sets and an online store serving current features at low latency. Shared definitions and point-in-time joins help limit training-serving skew and leakage from the future.
  • Rejestr i wersjonowanie modeli (Model registry and versioning): A central registry of model versions with metadata and lineage to experiment runs. Mutable aliases such as champion or production can decouple production-version selection from deployment code and make promotion or rollback easier.
  • Wersjonowanie i rejestr promptów (Prompt management and versioning): Storing prompts as immutable, numbered versions with metadata and a production alias. Linking versions to traces and evaluations supports comparison; when an application loads prompts dynamically through an alias, rollback is possible without redeploying code.
  • Śledzenie eksperymentów (parametry, metryki, artefakty) (Experiment tracking): Logging training runs — hyperparameters, metrics, artefacts, and code and data versions — to support result comparison, lineage and reconstruction of experiment conditions.
• Serwowanie i dekodowanie (Serving and decoding): Handling requests and generating responses.
  • Ciągłe batchowanie (Continuous batching): Adding requests to a running batch instead of waiting for it to finish.
  • Dekodowanie spekulatywne (Speculative decoding): A cheap model guesses several tokens ahead; the big one only verifies them.
  • Metryki wydajności inferencji LLM (TTFT, TPOT, przepustowość) (Inference performance metrics (TTFT, TPOT, throughput, goodput)): Core measures of LLM serving speed: time to first token (TTFT), average time for subsequent output tokens (TPOT/ITL), throughput in tokens or requests per second, and goodput — requests served within defined SLOs.
  • Rozdzielenie fazy prefill i decode (Disaggregated prefill and decode): Running prompt processing (prefill, compute-bound) and token generation (decode, memory-bound) on separate GPU pools, transferring the KV cache between them, so TTFT and per-token latency can be tuned independently.
  • Serwowanie wielu adapterów LoRA na jednym modelu (Multi-LoRA serving): One base model in GPU memory serves requests for many fine-tuned variants at once: a LoRA adapter is selected per request, and adapters are loaded and swapped on the fly.
  • Strategie dekodowania (Decoding strategies): How the next token is picked from the model's distribution.
  • Zarządzanie KV cache (KV cache management): The attention cache during generation: paging, sharing, compression.
• Uruchamianie własnych modeli (Self-hosted model inference): Serving a model with your own weights from a local runtime or server.
  • Autoskalowanie i zimny start wdrożeń (Autoscaling model deployments and cold starts): Matching the number of model replicas to bursty traffic, including scale-to-zero. For GPU deployments, one major challenge is cold start: provisioning a machine, pulling the image and loading model weights.
  • Lokalne uruchamianie LLM na własnym komputerze (Running LLMs locally on consumer hardware): Running open models on a laptop or desktop computer. One common path uses quantised GGUF files and selects a quantisation level that fits available memory; a local API server is an optional way to expose the model to applications.
  • Silniki serwowania modeli językowych (LLM) (LLM inference engines and servers): Server software for serving LLMs efficiently to many users. Depending on the implementation, it may combine continuous batching, paged KV cache and shared-prefix reuse, and often exposes APIs compatible with popular interfaces plus metrics.
  • Szacowanie pamięci GPU (wagi i KV cache) (Estimating GPU memory for deployment): Working out how much GPU memory a deployment needs: weights (parameters × bytes per parameter at a given precision) plus a KV cache that grows with context length and concurrent requests, plus headroom for overhead.

Sources

• Stanford CS336 · Language Modeling from Scratch (2025) · reference: Budowa modeli, dane, trening, systemy, inferencja i post-training.
• Full Stack Deep Learning (2022, archiwum) · reference: Pełny cykl systemu ML: dane, testy, wdrażanie, monitoring, zespoły i etyka.
• FSDL · LLM Bootcamp (2023, archiwum) · reference: Aplikacje LLM, prompting, retrieval, UX i LLMOps.