Trenowanie i dostrajanie

Training and adaptation · id training

Jak powstają i zmieniają się wagi modelu.

How model weights come into being and change.

In one sentence

To uczenie modeli na przykładach i dostosowywanie ich do nowych zadań lub oczekiwanego zachowania.

This trains models on examples and adapts them to new tasks or desired behaviour.

Problem it solves

Gotowy model nie zawsze zna twoją dziedzinę, odpowiada w potrzebny sposób lub mieści się w dostępnym budżecie.

A pretrained model may lack your domain knowledge, respond in the wrong way or exceed your budget.

What it does

Dane, przykłady, oceny i sygnały innych modeli zmieniają ustawienia sieci; różne metody obejmują całość, małe części lub łączenie wariantów.

Data, examples, evaluations and other models' signals change network parameters; methods update the whole network, small parts or combine variants.

Everyday example

Uczysz asystenta na przykładach firmowych odpowiedzi i sprawdzasz, czy nowe rozmowy prowadzi zgodnie z potrzebami zespołu.

You train an assistant on company response examples and check whether new conversations meet the team's needs.

This concerns you if you…

• chcesz zmienić zachowanie lub umiejętności modelu
• masz dane, przykłady albo sposób oceny prób
• want to change model behaviour or abilities
• have data, examples or a way to evaluate attempts

Subtopics

• Destylacja (Knowledge distillation): Training a model on signal from another model.
  • Destylacja kontekstu (wpisanie instrukcji w wagi modelu) (Context distillation): Fine-tuning a model so that without a long prompt it answers as it did with one.
  • Destylacja na miękkich etykietach (logity nauczyciela) (Soft-label (logit) distillation): The student learns to reproduce the teacher's full probability distribution, not just its top answer. Requires access to the teacher's outputs (logits).
  • Destylacja on-policy (uczeń generuje, nauczyciel ocenia) (On-policy distillation): The student generates its own responses and the teacher provides a dense token-level signal for them. This trains the student on states it visits itself rather than only on finished teacher-generated answers.
    • Destylacja on-policy dla hierarchicznych zespołów agentów (On-policy distillation for hierarchical multi-agent teams): Combines on-policy distillation with reinforcement learning to train a shared-parameter Leader-Searcher-Answerer team end to end.
    • Eksploracja w kontekście przez asymetryczną destylację on-policy (In-context exploration through asymmetric on-policy distillation): A student-teacher framework in which the student's on-policy history receives action labels from a privileged expert, teaching test-time exploration.
  • Destylacja polityki ze względną przewagą krokową (Step-Advantage Policy Distillation): An on-policy distillation method over recorded prefixes of multi-turn trajectories, using relative advantages among responses at a single decision step.
  • Destylacja przez odpowiedzi nauczyciela (dane syntetyczne) (Sequence-level distillation from teacher outputs): The teacher generates answers and the student is fine-tuned on them like ordinary data (SFT). Works even with an API-only teacher; licence terms need checking.
  • Rekurencyjna samodestylacja (Recursive self-distillation): Self-distillation repeated across rounds, with the updated model becoming the teacher for the next iteration.
• Efektywne dostrajanie (Parameter-efficient fine-tuning): Training selected parameters or small adapters.
  • Adaptery (małe warstwy wstawiane do modelu) (Adapter modules): Small trainable layers are inserted into a frozen model and only they are trained. One base model can serve many tasks with separate, swappable adapters.
  • Dostrajanie przez interwencje w reprezentacje (Fine-tuning through representation interventions): Instead of changing weights, it learns small interventions on hidden representations at chosen layers and positions of a frozen model.
  • Dostrajanie wyłącznie wektorów biasu (Bias-only fine-tuning): Fine-tuning a model by updating only its bias terms or a subset of them while leaving all other weights frozen.
  • LoRA (Low-rank adaptation): Fine-tuning with low-rank adapters. Used for language and image models alike.
    • QLoRA (Quantized low-rank adaptation (QLoRA)): LoRA on top of a base model stored in 4 bits. Cuts the GPU memory needed for fine-tuning.
    • Warianty LoRA (DoRA, LoRA+, AdaLoRA) (LoRA variants): Refinements of basic LoRA: separating weight magnitude and direction, different learning rates for the two adapter matrices, adaptive rank allocation. They aim to close the gap to full fine-tuning.
  • Miękkie prompty (prompt tuning, prefix tuning) (Soft prompts: prompt and prefix tuning): The model stays frozen while a small set of continuous vectors is learned. Prompt tuning adds them as virtual input tokens, whereas prefix tuning supplies learned prefixes to successive attention layers.
  • Skalowanie aktywacji wyuczonymi wektorami (Learned-vector activation rescaling): A PEFT method that multiplies attention keys, values and feed-forward activations by learned vectors.
• Post-training: Shaping a pretrained model toward the behaviour you want.
  • Preferencje i RL dla LLM (Preference optimization and RL for LLMs): Shaping behaviour with preferences or rewards.
    • DPO i optymalizacja offline (Direct preference optimization and offline variants): Training directly on chosen–rejected answer pairs, with no separate reward model. Computationally cheaper than RL.
      • Iteracyjne DPO na świeżych danych preferencji (Iterative DPO with newly generated preference data): Repeated rounds of DPO in which the current model generates answer pairs and a human, reward model or judge rates them.
      • Optymalizacja preferencji bez modelu referencyjnego (Reference-free preference optimization): DPO variants that need no frozen copy of the model as a reference point.
    • Modele nagrody (Reward models): A model trained on human judgements that scores answers. Used for RL, data filtering and evaluation.
    • Optymalizacja preferencji z ocen binarnych (Preference optimization from binary desirability feedback): Aligning a model on single responses labelled desirable or undesirable, using a utility function from prospect theory.
    • RL z nagrodą (Reinforcement learning with rewards): Iteratively improving the model's policy against a reward while staying close to its starting behaviour. Costlier and sensitive to hyperparameters.
      • Destylacja posterioru świadoma drzewa dla długich agentów (Tree-aware posterior self-distillation for long-horizon agentic RL): Critic-free tree-aware posterior self-distillation for credit assignment in agentic RL.
      • GRPO i warianty bez krytyka (GRPO and critic-free policy optimization): A PPO variant without a separate value model: the model samples a group of answers to the same prompt and estimates advantage relative to rewards within that group. This reduces memory requirements and is used in several newer reasoning-training recipes.
      • Klasyczny RLHF: SFT → model nagrody → PPO (Classic RLHF pipeline with PPO): The three-stage RLHF recipe popularised by InstructGPT: supervised fine-tuning on demonstrations, training a reward model from human rankings, then policy optimisation with PPO while constraining divergence from a reference model.
      • Uczenie agentów na trajektoriach wielosesyjnych (Multi-session agentic reinforcement learning): Critic-free hierarchical advantage optimization for agent trajectories composed of multiple sessions.
  • SFT / instruction tuning (Supervised fine-tuning): Fine-tuning on examples of the expected answer.
    • Dostrajanie do wywoływania funkcji i narzędzi (Function-calling and tool-use fine-tuning): Fine-tuning a model on examples whose answer is a correct, structured tool call with arguments.
  • Szablony czatu i formaty rozmów (Chat templates and conversation formats): How a conversation is stored in a dataset and turned into text for the model. The training template has to match at inference.
• Proces uczenia (Training optimization): Steering how the weights get updated.
  • Harmonogram learning rate (rozgrzewka, cosinus, WSD) (Learning-rate schedules): A plan for changing the learning rate during training. Common patterns include warm-up followed by decay; WSD adds a longer stable-rate phase before the final decay, making training runs easier to extend.
  • Optymalizatory (SGD, Adam/AdamW, Muon) (Optimizers for neural network training): Rules that turn a gradient into a weight update: from SGD with momentum, through adaptive Adam and AdamW, to newer matrix-aware optimisers such as Muon.
  • Oszczędzanie pamięci podczas treningu (Memory-efficient training): Techniques that shrink memory used by activations, gradients and optimiser states, so larger models train on the same hardware.
    • 8-bitowe stany optymalizatora (8-bit optimizer states): Storing optimiser statistics (e.g. Adam moments) in 8 bits with block-wise quantisation instead of 32 bits.
    • Niskorzędowe rzutowanie gradientu (Low-rank gradient projection): Full-parameter training in which gradients and optimiser states are kept in a low-dimensional projection.
    • Ponowne liczenie aktywacji (gradient checkpointing) (Gradient (activation) checkpointing): Only selected activations are stored; the missing ones are recomputed during backpropagation.
  • Pakowanie sekwencji w treningu (sequence packing) (Sequence packing for training): Concatenating several short examples into one context-length sequence to reduce padding in a batch.
  • Przenoszenie hiperparametrów z małego modelu na duży (μP) (Maximal update parametrization (μP) and hyperparameter transfer): A network parametrisation under which optimal hyperparameters are tuned on a small model and transferred unchanged to a large one.
  • Rozmiar partii treningowej i akumulacja gradientu (Training batch size and gradient accumulation): Choosing how many examples go into each update step: gradient accumulation over micro-batches and the critical batch size.
  • Stabilność treningu i skoki straty (Training stability): Diagnosing and mitigating instabilities such as sudden spikes or divergence in loss during long training runs of large models. The problems can be reproduced in smaller models to test safeguards in the training recipe.
  • Trening w mieszanej precyzji (FP16, BF16, FP8) (Mixed-precision training): Running most operations in shorter number formats while keeping key values in full precision. It cuts memory use and speeds up training on modern GPUs.
• Trening od zera / pretraining (Training from scratch and pretraining): Teaching a model from initial weights. Pretraining is the stage that prepares it for later use or adaptation.
  • Maskowane modelowanie języka (MLM) (Masked language modelling (MLM)): A pretraining objective for encoders: the model restores masked tokens from context on both sides.
  • Mid-training (późna faza pretrainingu) (Mid-training): A late pretraining phase on a smaller, carefully chosen data mix (e.g. maths, code, reasoning, long documents), usually while the learning rate is annealed. It strengthens selected capabilities before post-training.
  • Prawa skalowania i budżet obliczeń (Scaling laws and compute-optimal training): Predictable relationships between model size, data, compute and quality. They let you plan a large training run from small ones and choose the ratio of parameters to tokens.
  • Przewidywanie następnego tokenu (modele autoregresyjne) (Next-token prediction objective): A common pretraining objective for autoregressive language models: predict the next token from the preceding ones. The error is typically optimised as cross-entropy.
  • Przewidywanie wielu tokenów naraz (multi-token prediction) (Multi-token prediction): An auxiliary pretraining objective: from each position the model predicts several upcoming tokens with separate heads.
  • Wydłużanie okna kontekstu (Context window extension): Further training intended to improve performance on sequences longer than those used previously — often by modifying positional encoding and training on long examples.
• Uczenie ciągłe i transfer (Continual and transfer learning): Adapting to new data and new tasks.
  • Dotrenowanie na tekstach z dziedziny (continued pretraining) (Continued (domain-adaptive) pretraining): An additional pretraining phase of an existing model on unlabelled data from a selected domain, task or language, usually before downstream fine-tuning. It aims to adapt the model to a new data distribution and vocabulary.
  • Edycja wiedzy w modelu (ROME, MEMIT) (Knowledge editing in language models): A targeted change to facts or behaviours stored in a model without full retraining — by editing weights or using external memory. The goal is to limit side effects outside the edit's scope, but this is not guaranteed.
  • Katastrofalne zapominanie i jego ograniczanie (Catastrophic forgetting and its mitigation): While learning new data or tasks, a network may lose earlier capabilities. Mitigation methods include replaying earlier data, protecting important weights and reducing the scope of parameter updates.
  • Selektywny nadzór eksperta w ciągłym treningu agentów (Selective expert supervision in continual agent reinforcement learning): RL as backbone with selective SFT activated on hard states for continual agent training.
• Uczenie w czasie testu (Test-time training): Updating part of the parameters on an unlabeled test sample before the actual prediction.
  • Maskowanie obrazu w czasie testu (Masked autoencoder test-time training): One-sample test-time training whose auxiliary objective is reconstructing a masked image.
  • Warstwy z modelem w stanie ukrytym (Test-time training layers): A sequence layer whose hidden state is a model updated by a self-supervised step also at test time.
  • Zadanie pomocnicze na jednej próbce (One-sample self-supervised test-time training): A short self-supervised update on one sample or a stream, before the main-task prediction.
• Łączenie modeli (Model merging): Building a new model by arithmetic on the weights of already-trained models, with no further training.
  • Arytmetyka zadań (task vectors, TIES, DARE) (Task arithmetic and task vectors): The difference between fine-tuned and base-model weights forms a 'task vector' that can be added, subtracted and combined. TIES resolves sign conflicts between vectors, while DARE sparsifies delta parameters before merging.
  • Ewolucyjne łączenie modeli (Evolutionary model merging): An evolutionary algorithm searches weight mixtures or layer compositions instead of relying on one hand-designed recipe. The source paper also demonstrates merging models specialised in different domains.
  • Uśrednianie i interpolacja wag (model soups, SLERP) (Weight averaging and interpolation): Combining several fine-tunes of the same base model by averaging or smoothly interpolating their weights. Often improves quality and robustness at no extra inference cost.

Sources

• Stanford CS336 · Language Modeling from Scratch (2025) · reference: Budowa modeli, dane, trening, systemy, inferencja i post-training.
• Berkeley CS185/285 · Deep Reinforcement Learning (2026) · reference: Uczenie przez wzmacnianie, imitacja, sterowanie i RL dla LLM.