To modele, które w jednym miejscu rozumieją naraz tekst, obrazy, dźwięk albo wideo.
These are models that understand text, images, sound or video together in one place.
Problem it solves
Świat nie składa się z samego tekstu, a osobne programy do zdjęć, mowy i pisma gubią to, co łączy te informacje.
The world is not made of text alone, and separate tools for photos, speech and writing lose what links that information.
What it does
Zamienia obraz, dźwięk i słowa na wspólny zapis, dzięki czemu model może zestawić zdjęcie z pytaniem o nie.
It turns images, sound and words into one shared form, so the model can match a photo with a question about it.
Everyday example
Robisz zdjęcie wnętrza lodówki i pytasz, co ugotować na obiad, a asystent proponuje przepis z tego, co widać na półkach.
You photograph the inside of your fridge and ask what to cook, and the assistant suggests a recipe from what is on the shelves.
This concerns you if you…
pokazujesz modelowi zdjęcia, zrzuty ekranu lub nagrania
chcesz rozmawiać z asystentem głosem i obrazem
łączysz różne rodzaje danych w jednym zadaniu
show a model photos, screenshots or recordings
want to talk to an assistant with voice and images
combine different kinds of data in one task
Examples
Vision-language · speech-language
Subtopics
Modele audio–język (rozumienie mowy, dźwięków i muzyki) (Audio-language models): Language models connected to audio representations that can jointly handle speech, environmental sounds and music: transcription, translation, questions about recordings, audio captioning or speaker-related analysis. The supported task range depends on each model's data and design.
Modele wizja–język (opisywanie obrazów i pytania o nie) (Vision-language models (VLMs)): Models that combine images or video with text to generate textual answers: they describe scenes, answer questions about visual content, read documents or identify objects. They vary in scale and design, from on-device models to large systems used for interface analysis and visual reasoning.
Rozumienie dokumentów, wykresów i zrzutów ekranu (Document, chart and screenshot understanding): Reading and interpreting images rich in text and spatial layout — scans, forms, invoices, charts, infographics and screenshots. Some models work without a separate OCR stage, answer questions or extract data directly from pixels; these capabilities are also used for interface analysis.
Łączenie enkodera obrazu z modelem językowym (Connecting an image encoder to a language model): A common way to build a VLM: a pretrained image encoder produces visual features, while a projector, adapter or cross-attention layers expose them to a language model. The system is then tuned on image-instruction data; the modules trained and number of stages depend on the architecture.