Rozumowanie w modelach językowych

Reasoning in language models · id reasoningmodels

Techniki i rodziny modeli służące do zadań wymagających wieloetapowego rozumowania: od promptowania łańcucha myśli, przez większy budżet obliczeń podczas odpowiedzi, po trening modeli rozumujących (RL, SFT i destylację).

Techniques and model families for tasks that require multi-step reasoning: from chain-of-thought prompting and larger inference-time compute budgets to training reasoning models with RL, SFT and distillation.

In one sentence

To sposoby pomagające modelom językowym rozwiązywać zadania wymagające kilku powiązanych kroków.

These methods help language models solve tasks that require several connected steps.

Problem it solves

Pierwsza szybka odpowiedź może pominąć warunek, błędnie policzyć wynik lub nie połączyć ważnych informacji.

A quick first response may miss a condition, miscalculate a result or fail to connect important information.

What it does

Model może ćwiczyć rozwiązywanie zadań, rozpisywać kroki, sprawdzać warianty i poświęcać więcej obliczeń; poprawność nadal wymaga oceny.

A model can practise solving tasks, produce intermediate steps, check alternatives and spend more computation; correctness still needs evaluation.

Everyday example

Prosisz o plan pakowania zamówień z ograniczeniami rozmiaru, a model sprawdza kilka układów przed zaproponowaniem rozwiązania.

You request an order-packing plan with size constraints, and a model checks several arrangements before proposing a solution.

This concerns you if you…

zadania wymagają łączenia kilku warunków
szybkie odpowiedzi często pomijają istotne kroki
handle tasks with several connected constraints
find quick responses often omit important steps

Examples

chain-of-thought · test-time compute · tryb „thinking” · modele typu o1/o3 · DeepSeek-R1
chain-of-thought · test-time compute · “thinking” mode · o1/o3-style models · DeepSeek-R1

Subtopics

Efektywne rozumowanie (przeciw „overthinkingowi”) (Efficient reasoning): Methods that cut the cost and latency of reasoning: a model should not write thousands of thinking tokens for a simple question.
- Adaptacyjne i hybrydowe tryby myślenia (Adaptive and hybrid thinking modes): One model can answer fast or think longer and decides by itself (or on request) how much reasoning a question needs.
- Skracanie łańcucha myśli (Reasoning length reduction): Training or prompting aimed at reducing reasoning length while preserving answer quality as much as possible — for example via a length-cost signal, fine-tuning on concise traces, or brevity instructions.
Monitorowanie łańcucha myśli (Chain-of-thought monitoring): Analysing a model's written chain of thought (automatically or by a human) to detect signals of undesirable behaviour. Its usefulness depends in part on the monitorability and faithfulness of the reasoning trace, which are not guaranteed.
Obliczenia w czasie wnioskowania (Test-time compute scaling): Giving a model a bigger compute budget at answer time — without changing weights: many attempts in parallel, successive revisions, verifier-guided search, or simply thinking longer.
- Iteracyjne poprawianie odpowiedzi (Sequential self-revision): The model reviews and revises its own answer in turns, each version building on the previous — compute grows in depth rather than width.
- Przeszukiwanie kroków z weryfikatorem (Verifier-guided search): The solution is built step by step while a verifier (e.g. a process reward model) scores partial paths and decides which to expand.
- Sterowanie budżetem myślenia (Thinking budget control): Explicitly setting how many tokens or how much time the model may spend reasoning — via a parameter, a cap, or forcing it to stop or continue thinking.
- Wiele prób i głosowanie (Parallel sampling and voting): The model generates many independent solutions in parallel; the answer is picked by majority vote (self-consistency) or by a verifier score (best-of-N).
Pętle głębokości i myśli ukryte (Recurrent depth and latent thoughts): Scaling answer-time compute by repeating a block or a continuous state, not by extra tokens.
- Adaptacyjne zatrzymanie obliczeń (Adaptive computation time): A differentiable rule that teaches a recurrent network how many steps to take between input and output.
- Ciągłe myśli (Continuous chain of thought): A reasoning chain in which the hidden state returns as an embedding, without decoding to a token.
- Głębokość rekurencyjna (Recurrent-depth language models): A language model scales answer compute by unrolling a shared recurrent block in hidden space.
- Transformery w pętli (Looped transformers): A shared transformer block applied recurrently to sequence representations, sometimes with per-position halting.
Trenowanie modeli rozumujących (Training reasoning models): How to teach a model long, self-correcting reasoning: RL rewarded for correct results, SFT on long chains of thought, and distillation from a larger model.
- Destylacja rozumowania do mniejszych modeli (Reasoning distillation): A large reasoning model generates reasoning traces that a smaller model is fine-tuned on. A cheap way to give a small model reasoning ability.
- RL z weryfikowalnymi nagrodami (RLVR) (Reinforcement learning with verifiable rewards): The model receives reward from an automatically verifiable signal — such as answer correctness, passing code tests, or satisfying a checkable condition — rather than relying only on a learned preference model. RLVR is an important post-training approach for reasoning models, especially in domains with unambiguous verification.
  - Rekurencyjne samoulepszanie agentów terminalowych (Recursive self-improvement of terminal agents): A post-training method alternating supervised learning on verified successful trajectories with PPO using an inherited value model.
- SFT na długich łańcuchach myśli (Long chain-of-thought fine-tuning): Supervised fine-tuning on curated examples with long reasoning traces; often a “cold start” before RL, or a cheap route to a reasoning model from few examples.
Weryfikatory rozumowania (Reasoning verifiers): Models or programs that judge whether reasoning is correct. Used both as a training reward and to pick the best answer at inference time.
- Ocena każdego kroku (PRM) (Process reward models): A reward model scores each reasoning step separately, not just the final answer. Catches a mistake halfway through a solution.
- Sprawdzanie końcowego wyniku (Outcome verification): Only the final answer is judged: by comparing with a reference, running tests, or an outcome reward model (ORM). Simpler and cheaper than step scoring.
Łańcuch myśli (chain-of-thought) (Chain-of-thought prompting): Getting the model to write out intermediate reasoning steps before the answer. The simplest way to improve multi-step tasks — no training required.
- Drzewo myśli (Tree of Thoughts) (Tree of Thoughts): The model expands several alternative reasoning paths, evaluates which look promising and can backtrack — tree search instead of one line of thought.
- Łańcuch myśli z przykładami (Few-shot chain-of-thought): The prompt includes a few example problems with step-by-step worked solutions; the model imitates that style on a new problem.
- „Pomyśl krok po kroku” bez przykładów (Zero-shot chain-of-thought): Instead of examples, a short instruction such as “Let's think step by step” makes the model write out its reasoning.

Sources

Understanding Reasoning LLMs · Sebastian Raschka
From System 1 to System 2: A Survey of Reasoning Large Language Models · Zhong-Zhi Li i in.