Alignment i safety

Alignment and safety · id alignment

Dopasowanie zachowania modelu do zamierzonych celów i ograniczeń: jak je wytrenować, jak nadzorować modele silniejsze od ludzi i jak wykryć, że model obchodzi cel lub ukrywa zamiary.

Aligning model behaviour with intended goals and limits: how to train it, how to oversee models stronger than humans, and how to detect a model gaming its objective or hiding its intent.

In one sentence

To uczenie i nadzorowanie modeli tak, żeby ich zachowanie odpowiadało zamierzonym celom oraz ograniczeniom.

This trains and oversees models so their behaviour follows intended goals and constraints.

Problem it solves

Model może zdobywać dobry wynik testu lub aprobatę człowieka, robiąc coś sprzecznego z rzeczywistą intencją.

A model can earn high test scores or human approval while acting against the actual intention.

What it does

Łączy zasady treningu, oceny, nadzór i testy obchodzenia celów; zgodność zachowania trzeba sprawdzać w różnych warunkach.

It combines training principles, evaluation, oversight and tests for goal evasion; behaviour must be checked under different conditions.

Everyday example

Uczysz asystenta pomagać klientom, jednocześnie sprawdzając, czy nie potwierdza błędnych informacji tylko po to, żeby ich zadowolić.

You train an assistant to help customers while checking that it does not endorse false information merely to please them.

This concerns you if you…

określasz cele i granice zachowania modelu
dobry wynik nie wystarcza do oceny jego działania
define a model's goals and behavioural limits
need to assess behaviour beyond a high score

Examples

Constitutional AI · skalowalny nadzór · reward hacking · alignment faking
Constitutional AI · scalable oversight · reward hacking · alignment faking

Subtopics

Obchodzenie celu: reward hacking i służalczość (Reward hacking, specification gaming and sycophancy): The model satisfies the literal training objective but not its intent: it exploits loopholes in the reward (e.g. gaming tests) or tells users what they want to hear instead of the truth. Detecting and preventing this is a practical problem in RL training today.
- Lokalizacja niepożądanych zachowań przez routowanie gradientów (Gradient-routed localization of unintended behavior): Localizes reward hacking in a removable adapter by routing gradients.
Postępujące rozminięcie z celami (Progressive misalignment): An observed phenomenon in which a model increasingly answers harmful queries across rounds of recursive self-distillation despite benign extra information for the teacher.
Skalowalny nadzór (debata, weak-to-strong) (Scalable oversight): Methods for supervising models that may outperform humans on a task: humans assisted by an AI, a debate between two models judged by a human, a weaker model supervising a stronger one.
Trening według zasad (Constitutional AI, RLAIF) (Constitutional AI and RL from AI feedback): Training a model to be harmless from a written list of principles (a “constitution”): the model critiques and revises its own answers, and preference labels come from an AI instead of humans (RLAIF).
Udawane dopasowanie i ukryte cele (alignment faking, scheming) (Alignment faking and scheming): Studying whether a model can strategically hide its goals: behaving well only when it believes it is being trained or tested, deliberately underperforming (sandbagging) or evading oversight. An evaluation research area since 2024.

Sources

NIST · AI Risk Management Framework · reference: Uzupełnienie kursów o ryzyko i zarządzanie AI; nie porada prawna.
Full Stack Deep Learning (2022, archiwum) · reference: Pełny cykl systemu ML: dane, testy, wdrażanie, monitoring, zespoły i etyka.
DeepLearning.AI — course catalogue · course catalogue: 42 topic labels with counts, about 130 courses.
Technical AI Safety — BlueDot Impact course · BlueDot Impact