Mowa i audio

Speech and audio · id audio

Analiza i synteza sygnału dźwiękowego.

Analysing and synthesising sound.

In one sentence

To dziedzina, w której komputery słuchają, rozumieją i tworzą dźwięk: mowę, muzykę i odgłosy otoczenia.

This is the field where computers listen to, understand and create sound: speech, music and everyday noises.

Problem it solves

Potrzebujesz informacji z nagrań albo nowych dźwięków, a ręczne odsłuchiwanie i przygotowanie wszystkiego zajmuje dużo czasu.

You need information from recordings or new sounds, and manually listening to or producing everything takes time.

What it does

Modele uczą się wzorów dźwięku i ich związku ze słowami, żeby rozpoznawać nagrania lub tworzyć nowe brzmienia.

Models learn sound patterns and their links to words so they can recognise recordings or generate new audio.

Everyday example

Mówisz do telefonu „ustaw budzik na siódmą”, a on cię rozumie i odpowiada głosem, że budzik jest ustawiony.

You tell your phone to set an alarm for seven, and it understands and confirms out loud.

This concerns you if you…

• pracujesz z nagraniami, muzyką lub głosem
• chcesz rozmawiać z urządzeniami zamiast pisać
• tworzysz podcasty, audycje lub dźwięk do filmów
• work with recordings, music or voice
• want to talk to devices instead of typing
• make podcasts, broadcasts or film sound

Examples

• ASR · TTS · klasyfikacja dźwięku
• ASR · TTS · audio classification

Subtopics

• Agenci głosowi (Real-time voice agents): A system that conducts spoken conversations, understands utterances, makes decisions and completes tasks using models and tools.
  • Architektury agentów głosowych (Voice agent architectures): Ways to combine speech understanding, agent decisions and voice generation: modular, integrated or with delegated reasoning.
    • Głosowy front i osobny agent zadaniowy (Voice frontend with delegated reasoning): A voice model conducts the live conversation while delegating reasoning and tools to a separate text agent; the result returns to the spoken dialogue.
    • Modele mowa–mowa (Speech-to-speech models): A model takes speech and generates speech without separate STT and TTS services in the pipeline. Its internal representation may still include text or transcription; tool use and context control depend on the model.
    • Potok STT → agent tekstowy → TTS (Cascaded STT–text agent–TTS pipeline): Separate speech recognition, a text agent and speech synthesis. Stages can be swapped, inspected and connected to an existing workflow.
  • Ewaluacja agentów głosowych (Voice agent evaluation): Evaluating task success and conversational quality on real audio: tools, turns, interruptions, audible latency and robustness across speaking conditions.
    • Ocena tur, przerwań i słyszalnego czasu reakcji (Turn-taking, interruption and audible-latency evaluation): Tests measure premature takeovers, unwanted silence, overlap, post-interruption behavior and time to a useful audible response.
    • Odporność na akcent, hałas i cudzą mowę (Robustness to accents, noise and third-party speech): The same tasks are tested with varied accents, noise and other speakers, tracking understanding, turns and task outcomes separately.
    • Symulacje wieloturowych rozmów audio (Multi-turn audio conversation simulation): A simulated caller speaks with the agent through a full scenario, including pauses, corrections and interruptions; both dialogue and task state are evaluated.
    • Weryfikacja wyniku zadania i narzędzi (Voice-agent task and tool outcome verification): Evaluation compares spoken user intent, tool calls, final system state and the agent's spoken confirmation.
  • Generowanie odpowiedzi w czasie rzeczywistym (Real-time spoken response generation): Preparing agent speech so a useful response begins promptly while the model remains responsive to further user speech.
    • Strumieniowe przekazywanie odpowiedzi do mowy (Streaming model output into speech): Model output is passed to speech generation before the whole reply is complete; chunk boundaries affect latency and naturalness.
    • Wyprzedzające generowanie odpowiedzi (Preemptive response generation): The agent starts model work before the user's turn is confirmed complete; continued user speech may require discarding or revising the response.
  • Prowadzenie tur rozmowy (Turn-taking in voice agents): Decisions about when to listen, start responding and allow simultaneous listening and speaking. Silence does not necessarily mean a thought is complete.
    • Detekcja aktywności głosowej (VAD) (Voice activity detection): Detecting speech segments in an audio stream. VAD can trigger downstream stages but does not itself determine whether the user has finished a turn.
    • Dialog pełnodupleksowy (Full-duplex spoken dialogue): A system can receive speech and generate a response at the same time, handling overlap and listener backchannels. This can be achieved with an integrated model or by synchronizing modules.
    • Kontekstowe wykrywanie tury (Context-aware turn detection): Estimating whether the user has completed a thought from semantic and/or acoustic cues. Implementations may use audio, a transcript or both.
  • Przerwania i powrót do rozmowy (Interruption handling and recovery): Determining whether the user is interrupting, stopping the response and resuming the dialogue and task in line with what was already heard.
    • Rozróżnianie przerwania od potakiwania i tła (Distinguishing interruptions from backchannels and background speech): After detecting audio, deciding whether the user wants to take the turn, is backchanneling or coughing, or another person spoke in the background.
    • Usuwanie niesłyszanej części odpowiedzi z kontekstu (Aligning conversation context with played speech): After an interruption, remove generated speech the user did not hear from conversation history. The boundary depends on playback tracking and the API.
    • Wznowienie zadania po przerwaniu (Task recovery after a voice interruption): After the user interrupts, the agent handles the new request and returns to the right task step without repeating speech already heard.
  • Rozumienie wypowiedzi w dialogu (Understanding spoken input in voice agents): Interpreting who is speaking and what they said, including uncertainty and exact data capture. ASR itself is a separate topic in the map.
    • Dopytywanie przy niepewnej interpretacji mowy (Clarification under spoken-input uncertainty): When speech admits multiple interpretations, the agent asks a short clarifying question rather than guessing or taking action.
    • Przechwytywanie i potwierdzanie dokładnych danych (Exact entity capture from speech): The agent captures spoken names, numbers and codes, clarifies ambiguous characters and reads back exact values for confirmation before using a tool.
    • Rozpoznawanie właściwego rozmówcy (Target-speaker attribution in voice agents): Distinguishing the user's speech from other speakers so background voices do not become commands or interruptions.
  • Zadania i narzędzia w rozmowie (Task and tool orchestration in spoken dialogue): Using tools and completing multi-step tasks while keeping the conversation coherent. Calls may block or run in the background, depending on the API and agent design.
    • Komunikowanie pracy narzędzia (Spoken progress cues during tool use): The agent indicates it is working or waiting for a result while keeping its speech aligned with the actual operation state.
    • Narzędzia działające w tle rozmowy (Non-blocking tools in spoken dialogue): The agent can keep conversing while a long-running task executes, then incorporate its result. Semantics and support vary across APIs.
    • Potwierdzanie działań wykonywanych głosem (Spoken confirmation before consequential actions): Before an action that changes data, the agent speaks the intended action and consequence, waits for explicit confirmation and reports success only after the tool succeeds.
    • Scenariusze i stan rozmowy (Stateful spoken workflows): The model conducts a multi-step conversation; changing the step affects instructions, available tools and transferred context.
• Automatyczny zapis nut z nagrania (Automatic music transcription): This identifies the pitch and duration of notes heard in a recording.
• Diaryzacja: kto kiedy mówi (Speaker diarization): This labels which parts of a conversation are spoken by the same person.
• Dopasowanie tekstu do czasu nagrania (Forced alignment of speech and text): This determines when words from a supplied text occur in a recording.
• Generowanie muzyki i efektów dźwiękowych (Music and sound generation): Models that create music, sound effects or ambience from a text prompt (sometimes with a melody or lyrics), usually as language models over discrete audio tokens or as latent diffusion.
• Oczyszczanie i poprawa jakości mowy (Speech enhancement): This reduces interference to make a recorded voice clearer.
• Rozdzielanie wokalu i instrumentów (Music source separation): This splits a finished music recording into separate sound parts.
• Rozpoznawanie dźwięków i klasyfikacja nagrań (Audio and sound event classification): Assigning one or more labels that describe a recording, such as sound type, music genre, spoken command or language. Models often analyse a spectrogram or a learned audio representation; shared audio-text spaces also enable zero-shot classification.
• Rozpoznawanie mowy / ASR (Automatic speech recognition): Turning speech into text.
• Synteza mowy / TTS (Text-to-speech): Producing speech from text.

Sources

• Hugging Face · Audio Course · reference: Dane audio, klasyfikacja, ASR, TTS i modele mowy.
• Stanford CS231n · Computer Vision (2026) · reference: Sieci neuronowe i rozumienie obrazów.
• Stanford CS336 · Language Modeling from Scratch (2025) · reference: Budowa modeli, dane, trening, systemy, inferencja i post-training.