# Post-training

id `posttraining`

Path: [AI Atlas](https://aiatlasgraph.com/llms.txt) › [Trenowanie i dostrajanie](https://aiatlasgraph.com/t/training.md) › Post-training

## Description

- PL: Dostosowanie wcześniej wytrenowanego modelu do oczekiwanego zachowania.
- EN: Shaping a pretrained model toward the behaviour you want.

## In one sentence

- PL: To dostosowywanie już wyuczonego modelu do oczekiwanego sposobu odpowiadania i wykonywania zadań.
- EN: This adapts an already pretrained model to the desired way of responding and completing tasks.

## Problem it solves

- PL: Model zna wzory języka, ale niekoniecznie potrafi prowadzić rozmowę, wykonywać instrukcje lub odpowiadać zgodnie z twoimi wymaganiami.
- EN: A model knows language patterns but may not converse, follow instructions or respond according to your requirements.

## What it does

- PL: Przykłady rozmów, preferowane odpowiedzi i nagrody pomagają zmienić zachowanie, a zgodny format wejścia pozwala wykorzystać to podczas działania.
- EN: Conversation examples, preferred responses and rewards shape behaviour, while matching input formats lets the application use it during operation.

## Everyday example

- PL: Dostosowujesz model do krótkich odpowiedzi z prośbą o doprecyzowanie, gdy klient nie poda potrzebnych danych.
- EN: You adapt a model to give brief responses and request clarification when a customer omits necessary details.

## This concerns you if you…

- PL: model bazowy nie odpowiada w potrzebny sposób
- PL: masz przykłady lub oceny właściwego zachowania
- EN: find a base model responds in the wrong way
- EN: have examples or evaluations of desired behaviour

## Keywords

instruction tuning · SFT · RLHF · DPO · GRPO · preference data · verifiable rewards · chat templates · response masking · rejection sampling

## How it works

- PL: Model bazowy adaptuje się przez demonstracje, porównania odpowiedzi lub nagrody za generowane wyniki. Cele i mieszanki danych zmieniają zachowanie, a testy sprawdzają zgodność z instrukcjami oraz zachowanie wcześniejszych zdolności.
- EN: A base model is adapted through demonstrations, response comparisons or rewards for generated outputs. Objectives and data mixtures change behaviour, while tests assess instruction following and retention of earlier capabilities.

## Limits

- PL: Nadzór może premiować pozorną poprawność lub styl zamiast jakości zadania. Wąska adaptacja osłabia inne zdolności; ocena musi obejmować nowe zachowanie i wcześniejsze zastosowania.
- EN: Supervision can reward apparent correctness or style over task quality. Narrow adaptation can weaken other capabilities; evaluation must cover both new behaviour and earlier applications.

## Key terms

- PL: **demonstracja**: Przykład oczekiwanego zachowania używany jako cel treningowy.
- PL: **sygnał preferencji**: Informacja, która odpowiedź jest lepsza według wybranego kryterium.
- EN: **demonstration**: An example of desired behaviour used as a training target.
- EN: **preference signal**: Information identifying the better response under a chosen criterion.

## Know first

- [Trening od zera / pretraining (Training from scratch and pretraining)](https://aiatlasgraph.com/t/pretraining.md)
- [Zbiory testowe i regresje (Evaluation datasets and regression tests)](https://aiatlasgraph.com/t/evalsets.md)

## Examples

- PL: Instruction tuning · preferencje · reasoning
- EN: Instruction tuning · preferences · reasoning

## Subtopics (3 direct, 14 in total)

- [Preferencje i RL dla LLM (Preference optimization and RL for LLMs)](https://aiatlasgraph.com/t/preferences.md): Shaping behaviour with preferences or rewards.
  - [DPO i optymalizacja offline (Direct preference optimization and offline variants)](https://aiatlasgraph.com/t/dpo.md): Training directly on chosen–rejected answer pairs, with no separate reward model. Computationally cheaper than RL.
    - [Iteracyjne DPO na świeżych danych preferencji (Iterative DPO with newly generated preference data)](https://aiatlasgraph.com/t/onlinedpo.md): Repeated rounds of DPO in which the current model generates answer pairs and a human, reward model or judge rates them.
    - [Optymalizacja preferencji bez modelu referencyjnego (Reference-free preference optimization)](https://aiatlasgraph.com/t/refreepo.md): DPO variants that need no frozen copy of the model as a reference point.
  - [Modele nagrody (Reward models)](https://aiatlasgraph.com/t/rewardmodels.md): A model trained on human judgements that scores answers. Used for RL, data filtering and evaluation.
  - [Optymalizacja preferencji z ocen binarnych (Preference optimization from binary desirability feedback)](https://aiatlasgraph.com/t/kto.md): Aligning a model on single responses labelled desirable or undesirable, using a utility function from prospect theory.
  - [RL z nagrodą (Reinforcement learning with rewards)](https://aiatlasgraph.com/t/rlpolicy.md): Iteratively improving the model's policy against a reward while staying close to its starting behaviour. Costlier and sensitive to hyperparameters.
    - [Destylacja posterioru świadoma drzewa dla długich agentów (Tree-aware posterior self-distillation for long-horizon agentic RL)](https://aiatlasgraph.com/t/treeopsd.md): Critic-free tree-aware posterior self-distillation for credit assignment in agentic RL.
    - [GRPO i warianty bez krytyka (GRPO and critic-free policy optimization)](https://aiatlasgraph.com/t/grpo.md): A PPO variant without a separate value model: the model samples a group of answers to the same prompt and estimates advantage relative to rewards within that group. This reduces memory requirements and is used in several newer reasoning-training recipes.
    - [Klasyczny RLHF: SFT → model nagrody → PPO (Classic RLHF pipeline with PPO)](https://aiatlasgraph.com/t/rlhfppo.md): The three-stage RLHF recipe popularised by InstructGPT: supervised fine-tuning on demonstrations, training a reward model from human rankings, then policy optimisation with PPO while constraining divergence from a reference model.
    - [Uczenie agentów na trajektoriach wielosesyjnych (Multi-session agentic reinforcement learning)](https://aiatlasgraph.com/t/mosaicmsarl.md): Critic-free hierarchical advantage optimization for agent trajectories composed of multiple sessions.
- [SFT / instruction tuning (Supervised fine-tuning)](https://aiatlasgraph.com/t/sft.md): Fine-tuning on examples of the expected answer.
  - [Dostrajanie do wywoływania funkcji i narzędzi (Function-calling and tool-use fine-tuning)](https://aiatlasgraph.com/t/toolcallingft.md): Fine-tuning a model on examples whose answer is a correct, structured tool call with arguments.
- [Szablony czatu i formaty rozmów (Chat templates and conversation formats)](https://aiatlasgraph.com/t/chattemplates.md): How a conversation is stored in a dataset and turned into text for the model. The training template has to match at inference.

## Courses

- [Post-training of LLMs](<https://www.deeplearning.ai/courses/post-training-of-llms>)
- [Fine-tuning & RL for LLMs: Intro to Post-training](<https://www.deeplearning.ai/courses/fine-tuning-and-reinforcement-learning-for-llms-intro-to-post-training>)

## Sources

- [Stanford CS336 · Language Modeling from Scratch (2025)](<https://cs336.stanford.edu/spring2025/>) `cs336` · reference: Budowa modeli, dane, trening, systemy, inferencja i post-training.
- [Berkeley CS185/285 · Deep Reinforcement Learning (2026)](<https://rail.eecs.berkeley.edu/deeprlcourse/>) `rl` · reference: Uczenie przez wzmacnianie, imitacja, sterowanie i RL dla LLM.
- [DeepLearning.AI — course catalogue](<https://www.deeplearning.ai/courses>) `dlai` · course catalogue: 42 topic labels with counts, about 130 courses.

---

AI Atlas revision 122 (2026-10-30) · HTML: https://aiatlasgraph.com/t/posttraining · index: https://aiatlasgraph.com/llms.txt · everything: https://aiatlasgraph.com/llms-full.txt · updated 2026-10-30