Uczenie ze wzmocnieniem i decyzje

Reinforcement learning and sequential decision-making · id decisions · aliases: reinforcement learning, RL, sequential decision-making, uczenie ze wzmocnieniem

Uczenie działania na podstawie rezultatów i interakcji ze środowiskiem.

Learning to act from outcomes and interaction with an environment.

In one sentence

To uczenie programów i robotów, jak wybierać działania na podstawie ich skutków oraz pokazów.

This teaches programs and robots to choose actions using their consequences and demonstrations.

Problem it solves

Dobry wybór teraz może prowadzić do złego wyniku później, a ręczne zapisanie wszystkich zasad działania jest trudne.

A good choice now can lead to a poor outcome later, and writing every action rule by hand is difficult.

What it does

Korzysta z prób, zapisanych doświadczeń, demonstracji i przewidywania skutków, żeby poprawiać kolejne wybory oraz sterowanie w otoczeniu.

It uses trials, recorded experience, demonstrations and predicted consequences to improve successive choices and control within an environment.

Everyday example

Wózek magazynowy uczy się dojeżdżać do półki, porównując skutki tras w symulacji i uwzględniając ruch innych wózków.

A warehouse cart learns to reach a shelf by comparing route outcomes in simulation and accounting for other carts' movement.

This concerns you if you…

• zadanie wymaga kolejnych działań i oceny ich skutków
• chcesz uczyć program lub robota z doświadczeń
• have tasks requiring successive actions and outcome assessment
• want to teach software or a robot from experience

Subtopics

• Algorytmy RL (Reinforcement learning algorithms): Methods for learning values and policies.
  • Bezpieczne RL i RL z ograniczeniami (Safe and constrained reinforcement learning): Learning policies that maximise reward while aiming to keep cost or risk within set limits during training or deployment, e.g. through a separate cap on expected cost instead of folding penalties into the reward. Guarantees usually hold in expectation, not for every run.
  • Bufor doświadczeń i priorytetowe odtwarzanie (Experience replay and prioritised replay): A mechanism that stores transitions for repeated off-policy learning; prioritised replay samples more informative experiences more often.
  • Deterministyczne gradienty polityki (Deterministic policy-gradient methods): Off-policy actor–critic methods for continuous actions with a deterministic actor; TD3 adds fixes that curb DDPG's value overestimation.
  • Gradient polityki (REINFORCE) (Policy gradient methods (REINFORCE)): Directly adjusting policy parameters towards actions that led to higher reward. Simple and general, but sample-hungry.
  • Hierarchiczne RL i opcje (Hierarchical reinforcement learning and options): Splitting behaviour into levels: a higher-level policy picks multi-step skills (options), which in turn choose low-level moves and when to stop.
  • Metody aktor–krytyk (A2C, SAC) (Actor–critic methods (A2C, SAC)): Combines a learned policy (actor) with a value or Q function (critic) that guides policy updates and can reduce gradient-estimator variance. A widely used family for continuous control.
  • Proximal Policy Optimization (PPO) (Proximal policy optimization (PPO)): A policy-gradient method using a surrogate objective designed to constrain overly large policy changes during updates. Widely used in simulation, robotics and some RLHF systems.
  • Q-learning i głębokie sieci Q (DQN) (Q-learning and deep Q-networks (DQN)): Off-policy learning of action values from the Bellman equation; in DQN a neural network replaces the Q-table. Methods in this family reuse data from past interactions, but training with function approximation needs stabilising mechanisms.
  • Rozkładowe uczenie ze wzmocnieniem (Distributional reinforcement learning): A family of methods learning the distribution of returns (value distribution) instead of its expectation, via a distributional Bellman equation.
  • Uczenie z celami i odtwarzanie z perspektywy czasu (HER) (Goal-conditioned RL and hindsight experience replay): One policy trained for many goals at once, with the goal given as an extra input. Hindsight replay relabels failed attempts as if the goal had been whatever was actually reached, enabling learning from success-only rewards.
• Modele świata i planowanie (Model-based reinforcement learning): Predicting the effect of actions before taking them.
  • Architektury predykcyjne wspólnego osadzenia (JEPA) (Joint Embedding Predictive Architectures (JEPA)): Models that learn to predict abstract representations of future or missing states in embedding space, rather than reconstructing raw data (pixels, tokens).
    • Hierarchiczne JEPA (H-JEPA) (Hierarchical Joint Embedding Predictive Architectures (H-JEPA)): An extension of JEPA with planning at multiple levels of abstraction, using separate representations for different timescales.
    • JEPA-Anything – uniwersalne predykcyjne modele świata (JEPA-Anything: domain-agnostic predictive world models): An extension of JEPA with orthogonal predictive factorization (OPF), enabling a shared learning principle for world models across many different domains.
  • Generatywne modele świata dla agentów i robotów (Generative world models for agents and robotics): Models that generate or predict how an environment evolves, sometimes conditioned on user or agent actions. They can create interactive environments or provide simulation foundations for training and testing autonomous systems.
  • Planowanie z przeszukiwaniem drzewa (AlphaZero, MuZero) (Tree-search planning with learned models): Combining Monte Carlo tree search with networks that predict value and policy, and in MuZero also a learned model of the rules. Key to the successes in board games.
  • Uczenie w wyobraźni modelu świata (Dreamer) (Learning in imagination with latent world models): The agent learns a compact model of the environment and trains its policy on imagined rollouts instead of costly real trials.
• Offline RL i eksploracja (Offline reinforcement learning and exploration): Learning from a fixed set of experience, and gathering new experience.
  • Eksploracja z ciekawości (nagroda wewnętrzna) (Curiosity-driven exploration and intrinsic rewards): The agent earns an extra reward for novel or hard-to-predict situations, which helps when external reward is sparse.
  • Konserwatywne offline RL (CQL, IQL) (Conservative offline RL): Learning a policy from a fixed dataset without new interaction; penalising overestimated out-of-dataset actions guards against distribution-shift errors.
  • Ocena polityki off-policy (OPE) (Off-policy evaluation (OPE)): Estimating how good a new policy would be using only data collected by a different policy, without running it. Often a critical step before deploying RL in real systems.
  • RL jako modelowanie sekwencji (Decision Transformer) (RL as sequence modelling (Decision Transformer)): Treating trajectories like text: a transformer predicts the next action from history and a desired return.
• Podstawy RL (Reinforcement learning fundamentals): States, actions, rewards and policies.
  • Częściowa obserwowalność (POMDP) (Partially observable MDPs (POMDP)): Decision-making when observations do not reveal the full state of the world, so the agent must combine information over time, e.g. through a belief state or network memory.
  • Funkcje wartości i równanie Bellmana (Value functions and the Bellman equation): How much reward to expect from a state or action; the Bellman equation links the value now to the value one step later. The basis of Q-learning and actor–critic methods.
  • Proces decyzyjny Markowa (MDP) (Markov decision processes (MDP)): A formal model of sequential decision-making in RL: states, actions, transition probabilities, rewards and a discount factor. One of the standard starting points for describing RL algorithms.
  • Programowanie dynamiczne: iteracja wartości i polityki (Dynamic programming for MDPs (value and policy iteration)): Computing an optimal policy when the transition and reward model is fully known, by repeatedly applying the Bellman equations. The reference point for methods that learn from trials.
  • Projektowanie i kształtowanie nagrody (Reward design and reward shaping): Specifying the reward signal so the agent learns the intended behaviour, and adding auxiliary rewards that speed up learning without changing the goal.
  • Uczenie z różnic czasowych (TD) i metody Monte Carlo (Temporal-difference learning and Monte Carlo methods): Learning value functions from experience without an environment model: Monte Carlo averages full episode returns, while TD updates estimates after every step using the next state's estimate.
  • Wielorękie bandyty: eksploracja kontra eksploatacja (Multi-armed and contextual bandits): Repeatedly choosing among options with unknown rewards while balancing exploration of new possibilities against exploitation of the current best. Used in recommendation and adaptive online experimentation.
  • Środowiska i symulatory do RL (RL environments and simulators): The standard reset/step interface and ready-made environments where agents practise; also physics simulators for robots.
  • „Śmiertelna triada” w RL (The deadly triad in reinforcement learning): Generalising value functions with approximators, and the instability arising from combining approximation, bootstrapping and off-policy learning.
• Uczenie przez imitację (Imitation learning): Learning to act from demonstrations.
  • Imitacja z korektą eksperta (DAgger) (Interactive imitation learning (DAgger)): An expert labels correct actions in states reached by the learning policy, and the new examples are added to the dataset. This mitigates compounding error in behavioural cloning.
  • Klonowanie zachowań (behavioral cloning) (Behavioural cloning): Supervised learning on observation→action pairs from demonstrations. It is a foundational way to learn robot policies from demonstrations; a weakness is compounding error outside the distribution represented in the data.
  • Odwrotne RL i imitacja adwersarialna (IRL, GAIL) (Inverse RL and adversarial imitation): Instead of copying actions, recover the goal (reward) that explains expert behaviour, or train a policy so a discriminator cannot tell it from the demonstrations.
  • Polityki generatywne (Diffusion Policy, ACT) (Generative action policies (Diffusion Policy, ACT)): The policy generates whole chunks of future actions with a generative model (diffusion, flow matching, transformer), capturing the different ways demonstrators perform a task.
• Uczenie wieloagentowe (MARL) (Multi-agent reinforcement learning (MARL)): Many learning agents in one environment: cooperation, competition and self-play. Combines RL with game theory.
  • Scentralizowany trening, zdecentralizowane działanie (CTDE) (Centralised training for decentralised execution (CTDE)): A cooperative MARL paradigm: training may use global state and other agents' data, while execution relies only on local observation histories.
    • Faktoryzacja wartości w kooperacyjnym MARL (Value-function factorisation for cooperative MARL): Value-based CTDE methods representing the joint Q-function as a composition of per-agent Q-functions consistent with the Individual-Global-Max principle.
    • Scentralizowani krytycy w wieloagentowym aktorze–krytyku (Centralised critics in multi-agent actor–critic): Multi-agent actor–critic methods where decentralised actors learn from critics that can access additional information about other agents during training.
  • Uczenie komunikacji między agentami (Learned multi-agent communication (emergent communication)): A MARL direction in which partially observing agents learn communication protocols end-to-end, together with their action policies.
  • Współpraca z nieznanymi partnerami (ad hoc teamwork) (Ad hoc teamwork): The problem of designing agents that cooperate with previously unseen partners without prior coordination, including zero-shot coordination with humans.
• Uogólnianie i adaptacja w RL (Generalisation and adaptation in reinforcement learning): A family of RL methods concerned with transferring policies to new environments, rapid task adaptation and learning under nonstationarity.
  • Augmentacja obserwacji w RL z obrazu (Data augmentation for pixel-based reinforcement learning): Regularising RL from visual observations through random transformations of replay-buffer images, improving sample efficiency and robustness.
  • Automatyczne projektowanie środowisk treningowych (UED) (Unsupervised environment design for reinforcement learning): Methods that automatically generate or select environment configurations to form a curriculum that improves agent robustness and generalisation.
  • Ciągłe uczenie ze wzmocnieniem (Continual reinforcement learning): Reinforcement learning under nonstationarity without a fixed end of training, including the loss of plasticity in deep networks.
  • Meta-uczenie ze wzmocnieniem (meta-RL) (Meta-reinforcement learning): Meta-learning applied to RL: training over a task distribution yields a policy or algorithm that adapts to a new task from few interactions.

Sources

• Berkeley CS185/285 · Deep Reinforcement Learning (2026) · reference: Uczenie przez wzmacnianie, imitacja, sterowanie i RL dla LLM.
• Stanford CS221 · AI: Principles and Techniques (2025) · reference: Rozumowanie, przeszukiwanie, logika i modele probabilistyczne.