Post-training

id posttraining

Dostosowanie wcześniej wytrenowanego modelu do oczekiwanego zachowania.

Shaping a pretrained model toward the behaviour you want.

In one sentence

To dostosowywanie już wyuczonego modelu do oczekiwanego sposobu odpowiadania i wykonywania zadań.

This adapts an already pretrained model to the desired way of responding and completing tasks.

Problem it solves

Model zna wzory języka, ale niekoniecznie potrafi prowadzić rozmowę, wykonywać instrukcje lub odpowiadać zgodnie z twoimi wymaganiami.

A model knows language patterns but may not converse, follow instructions or respond according to your requirements.

What it does

Przykłady rozmów, preferowane odpowiedzi i nagrody pomagają zmienić zachowanie, a zgodny format wejścia pozwala wykorzystać to podczas działania.

Conversation examples, preferred responses and rewards shape behaviour, while matching input formats lets the application use it during operation.

Everyday example

Dostosowujesz model do krótkich odpowiedzi z prośbą o doprecyzowanie, gdy klient nie poda potrzebnych danych.

You adapt a model to give brief responses and request clarification when a customer omits necessary details.

This concerns you if you…

model bazowy nie odpowiada w potrzebny sposób
masz przykłady lub oceny właściwego zachowania
find a base model responds in the wrong way
have examples or evaluations of desired behaviour

Examples

Instruction tuning · preferencje · reasoning
Instruction tuning · preferences · reasoning

Subtopics

Preferencje i RL dla LLM (Preference optimization and RL for LLMs): Shaping behaviour with preferences or rewards.
- DPO i optymalizacja offline (Direct preference optimization and offline variants): Training directly on chosen–rejected answer pairs, with no separate reward model. Computationally cheaper than RL.
  - Iteracyjne DPO na świeżych danych preferencji (Iterative DPO with newly generated preference data): Repeated rounds of DPO in which the current model generates answer pairs and a human, reward model or judge rates them.
  - Optymalizacja preferencji bez modelu referencyjnego (Reference-free preference optimization): DPO variants that need no frozen copy of the model as a reference point.
- Modele nagrody (Reward models): A model trained on human judgements that scores answers. Used for RL, data filtering and evaluation.
- Optymalizacja preferencji z ocen binarnych (Preference optimization from binary desirability feedback): Aligning a model on single responses labelled desirable or undesirable, using a utility function from prospect theory.
- RL z nagrodą (Reinforcement learning with rewards): Iteratively improving the model's policy against a reward while staying close to its starting behaviour. Costlier and sensitive to hyperparameters.
  - Destylacja posterioru świadoma drzewa dla długich agentów (Tree-aware posterior self-distillation for long-horizon agentic RL): Critic-free tree-aware posterior self-distillation for credit assignment in agentic RL.
  - GRPO i warianty bez krytyka (GRPO and critic-free policy optimization): A PPO variant without a separate value model: the model samples a group of answers to the same prompt and estimates advantage relative to rewards within that group. This reduces memory requirements and is used in several newer reasoning-training recipes.
  - Klasyczny RLHF: SFT → model nagrody → PPO (Classic RLHF pipeline with PPO): The three-stage RLHF recipe popularised by InstructGPT: supervised fine-tuning on demonstrations, training a reward model from human rankings, then policy optimisation with PPO while constraining divergence from a reference model.
  - Uczenie agentów na trajektoriach wielosesyjnych (Multi-session agentic reinforcement learning): Critic-free hierarchical advantage optimization for agent trajectories composed of multiple sessions.
SFT / instruction tuning (Supervised fine-tuning): Fine-tuning on examples of the expected answer.
- Dostrajanie do wywoływania funkcji i narzędzi (Function-calling and tool-use fine-tuning): Fine-tuning a model on examples whose answer is a correct, structured tool call with arguments.
Szablony czatu i formaty rozmów (Chat templates and conversation formats): How a conversation is stored in a dataset and turned into text for the model. The training template has to match at inference.

Courses

Sources

Stanford CS336 · Language Modeling from Scratch (2025) · reference: Budowa modeli, dane, trening, systemy, inferencja i post-training.
Berkeley CS185/285 · Deep Reinforcement Learning (2026) · reference: Uczenie przez wzmacnianie, imitacja, sterowanie i RL dla LLM.
DeepLearning.AI — course catalogue · course catalogue: 42 topic labels with counts, about 130 courses.