SUCH LARGE MUCH WOW

From “Can machines think?” to agents that work for you

Eugene Mirotin · Summer Linguistics School · Narva-Jõesuu · 14.07.2026

Part 1

Can Machines Think?

Ideas & key figures, 1950–2022

Part 1 · 1950

Alan Turing — the question

“Computing Machinery and Intelligence” — 1950
proposes the Imitation Game
judge texts two hidden partners — can’t tell? machine passes

Not what is thinking — can it fool you through language alone?

Linguistics · the Turing Test is a purely linguistic test

Part 1 · 1956

Dartmouth — AI gets a name

a summer workshop at Dartmouth College
coins the term Artificial Intelligence
the plan: crack it in one summer

AI becomes a field — a name, a community, unlimited optimism

Linguistics · Chomsky’s Syntactic Structures (1957) — the same bet: cognition as a formal system

Part 1 · 1957

The Perceptron — machines that learn

The Mark I Perceptron machine — The Mark I Perceptron — US Navy photo

Frank Rosenblatt: not programmed — trained
adjusts its weights on right / wrong feedback
modeled on neurons

…the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.

— New York Times, 1958

Part 1 · 1966

🎯 INTERACTIVE

ELIZA

The first chatbot — a Rogerian therapist. Talk to it.

Part 1 · 1966

ELIZA — the trick

“I feel X” → “WHY DO YOU FEEL X?”

a dozen pattern-matching rules — zero understanding
Weizenbaum’s secretary asked him to leave the room
Weizenbaum: disturbed, not pleased

We project understanding onto fluent output

Linguistics · Gricean implicature — hearers infer intent even where none exists

Part 1 · 1969

The first AI winter

Perceptrons, 1969: single-layer nets can’t learn XOR
funders read: neural nets = dead end
funding freezes for ~15 years

One rigorous negative result froze a field for a generation

Part 1 · 1980

The Chinese Room

from outside: appears to understand Chinese

你懂中文吗？

question, in Chinese — “do you understand Chinese?”

a person + a rulebook

当然懂！

answer, in Chinese — “of course!”

inside: follows rules — understands nothing

Syntax ≠ semantics — Searle, 1980

Hold this question — in 45 minutes you’ll see exactly what LLMs do in there

Linguistics · can meaning arise from pure form?

↓ look closer

Part 1 · 1986

Backpropagation — the answer

Rumelhart, Hinton & Williams — Nature, 1986
multi-layer networks can be trained
Minsky’s objection — answered

The algorithm was ready in 1986. The data and compute weren’t — wait 25 years.

Hinton — Nobel Prize in Physics, 2024

Part 1 · 1990s

Data beats rules

machine translation: stop parsing, start counting
corpus probabilities replace hand-written rules

Every time a linguist left the group, speech recognition got better

attributed to Fred Jelinek, IBM — wording and date disputed

Empiricism’s first industrial win

Linguistics · Chomsky’s innate structure vs learning from data — round one to the data

Part 1 · 2012

AlexNet — deep learning wins

28.2%

25.8%

26.2%

15.3%

2010

2011

2012 · #2

2012 · AlexNet

Krizhevsky, Sutskever & Hinton — again
what converged: data + compute + tricks

After 2012 deep learning eats everything: vision, speech, translation. Language is next.

↓ enter 2012

Part 1 · 2013

Word2Vec — meaning becomes geometry

king − man + woman ≈ ? queen

Mikolov et al., Google, 2013
trained to predict a word from its context
byproduct: meaning as geometry

“You shall know a word by the company it keeps” — Firth, 1957

Linguistics · distributional semantics becomes operational — we’ll play with it in Part 2

↓ solve it

Part 1 · 2017

The Transformer

“Attention Is All You Need”

Vaswani et al., 2017 — Google + University of Toronto

throws away recurrence — attention only
every token sees every token, at once
parallel → scalable

Every major model since is a Transformer: GPT, Claude, Gemini, LLaMA

the architecture itself — next section

Part 1 · 2018–2022

GPT → ChatGPT

2018

GPT-1 — proof of concept

2019

GPT-2 — withheld, misuse fears

2020

GPT-3 — 175B, few-shot

2022

ChatGPT — 100M users in 2 months

today: GPT-5.5 · Claude · Gemini · LLaMA · DeepSeek — multimodal, agentic

Research curiosity → daily tool

↓ next milestone

Part 1 · 1950 → 2026

76 years in three lines

1950 — Turing: “Can machines think?”

1980 — Searle: syntax is not sufficient for semantics

now — machines talk, often indistinguishably

You’ve seen the questions. Next 45 minutes: what’s actually inside. Then you answer.

↓ next

Part 2

How LLMs Actually Work

From neurons to next-token prediction

Part 2 · Neural nets

The neuron

$y = f (w_{1} x_{1} + w_{2} x_{2} + w_{3} x_{3} + b)$

weight = “how much do I care about this input?”
activation = “fire or stay quiet?”

Sum the evidence, decide to fire

Linguistics · features combining: +voiced, +nasal, +labial → /m/

Part 2 · Neural nets

🎯 INTERACTIVE

Neuron playground

Adjust the weights. Make it fire.

x₁ w₁ −0.40

x₂ w₂ 0.12

x₃ w₃ 0.00

b −0.60

f

z = −0.88

y = 0

quiet

That’s the whole atom of AI. Everything else is many of these.

↓ the point

Part 2 · Neural nets

Layers

neurons stack into layers
many hidden layers = “deep” learning
each layer: more abstract features

We design the network — the feature hierarchy emerges

characters → morphemes → syntax → semantics

Part 2 · Neural nets

Learning = shrinking the loss

The network guesses. We measure how wrong. Repeat.

predicted: cat 70% · dog 25% · bird 5%

actual: cat ✓

Loss = one number: “how wrong?” — e.g.

L = {(predicted - actual)}^{2}

random weights → terrible loss

goal: nudge every weight downhill

w \leftarrow w - η \cdot \frac{\partial L}{\partial w}

(η = learning rate)

Backpropagation computes “downhill” for millions of weights at once

↓ next step

Part 2 · Neural nets

🎯 INTERACTIVE

Gradient descent

Roll the ball to the bottom of the loss valley.

learning rate η 0.10

loss = …

Too small: crawls forever. Too big: overshoots. Learning rate is a real dial in real training.

↓ the lesson

Part 2 · Neural nets

We never write rules. Input, correct answer, minimize error — the structure emerges.

Linguistics · the empiricist bet taken to its extreme (Chomsky is watching)

Part 2 · Transformers

Before attention: reading one word at a time

The cat that the dog chased ran away

by word 8, the signal from word 1 has almost vanished

RNNs read one word at a time
sequential = slow, no parallelism
signal fades with distance

“The cat that the dog that the man bought chased ran away”

Linguistics · long-range syntax is exactly where RNNs die

Part 2 · Transformers

🎯 INTERACTIVE

Tokenizer

Type anything. Watch it become tokens.

try:

GPT-2’s tokenizer, first 6,000 merges — the IDs are real GPT-2 token IDs

Not words, not letters — subwords: frequent chunks

Linguistics · BPE keeps finding morphemes: un-, -iness, -ization

↓ so what are they?

Part 2 · Transformers

Embeddings — words become vectors

every token → a vector (hundreds–thousands of numbers)
learned, not designed
related meaning = nearby

Meaning = position in space

distance is similarity; directions are relations

Part 2 · Transformers

🎯 INTERACTIVE

Vector arithmetic

Do algebra on meaning.

?

real 50-dim GloVe vectors, projected to this 2D plane

Gender, tense, capital-of — directions in the space

↓ the point

Part 2 · Transformers

Word order is not free

Dog bites man

input = embedding + position

attention reads all tokens at once — order isn’t built in
fix: add a position signal to each embedding

Dog bites man ≠ Man bites dog

Linguistics · word order carries syntax — the model must be told

Part 2 · Transformers

Attention — the intuition

For every word: which other words matter right now? That highlighting is attention — the whole trick.

Part 2 · Transformers

Attention — the mechanism

Query

“what am I looking for?”

Key

“what do I offer?”

Value

“what do I carry?”

every token asks; every token answers; the match decides who informs whom

Attention = softmax(Q·Kᵀ / √d) · V

match questions to offers, blend the answers

↓ next piece

Part 2 · Transformers

🎯 INTERACTIVE

Attention, live

Click a word — see what it looks at.

GPT-3 runs 96 layers × 96 heads of this — frontier models repeat the idea at larger, often unpublished scales

Linguistics · grammatical relations rediscovered, no linguist involved

↓ how big does this get?

Part 2 · Transformers

The block

self-attention — gathers context

feed-forward — digests it

× 96 (GPT-3)

attention gathers context
feed-forward digests it
stack deep — abstraction grows per layer

That’s the entire architecture. No parser, no grammar module — this block, repeated.

Part 2 · Transformers

Three flavors

Encoder — BERT

reads both directions → understanding

Decoder — GPT

left-to-right → generation

Both — T5

read fully, then generate → translation

Comprehension vs production, in silicon

Part 2 · GPT

The causal mask

Thecatsaton The

cat

sat

on

rows may look left, never right

each token sees only its past
generation can’t peek ahead

“The horse raced past the barn fell”

Linguistics · left-to-right commitment, like human speech — garden paths hurt us both

Part 2 · GPT

The whole objective

P( next token | everything so far )

at every step: a probability for each of ~100,000 tokens

Everything GPT appears to be emerges from this one objective, at scale

Shannon (1948) framed language statistically: next-symbol uncertainty. GPT is that, done absurdly well.

↓ that’s it?

Part 2 · GPT

Pre-training (GPT-3)

~300B

tokens read

175B

weights

months

× thousands of GPUs

$ millions

one training run

self-supervised — the next token is the label
no annotators, just text

Grammar, facts, style, “reasoning” — side effects of prediction

later flagships: sizes and token counts often unpublished

Part 2 · GPT

Scale does something strange

same task — arithmetic, translation, multi-step reasoning — across GPT-3-family sizes:

125Mweights brittle — arithmetic mostly fails

13Bweights sometimes works — unreliable

175Bweights reliable enough to feel new

Nobody added a reasoning module. At enough scale, better prediction starts to look like reasoning.

↓ scale up

Part 2 · GPT

The alignment problem

raw model

plausible ≠ true
toxic, dangerous, confidently wrong
capability without values

aligned model

helpful — follows instructions
honest — admits uncertainty
harmless — declines harm

A brilliant author with no editor and no conscience. Alignment hires the editor.

Part 2 · GPT

RLHF — teaching taste

model writes

three answers: A, B, C

human ranks

B > A > C — just compares

reward model

learns to predict the taste

optimize

generate toward higher reward

leash

don’t drift from the base model

comparing is easier than composing — RLHF exploits that asymmetry

Preferences, distilled into a training signal

Linguistics · pragmatic competence on top of grammatical — Chomsky → Hymes

↓ next step

Part 2 · GPT

🎯 INTERACTIVE

Temperature

Watch it choose the next word.

temperature 0.70

T→0: boring and safe · T≈1: alive · T→2: word salad

↓ the dial

Part 2 · GPT

What it can’t do

Hallucinations

plausible ≠ true — the default failure

No grounding

never seen, touched, lived

Context window

finite attention span

No memory

every chat starts blank

Cutoff

history ends at training day

Fluent, grammatical, appropriate — and unanchored

Linguistics · Searle’s room again — now you know the mechanism

Part 2 · GPT

The full stack

+ scale + RLHF machines that converse

+ attention machines that use context

neurons + backprop machines that learn

Under it all: one probability distribution over the next token. Everything else is emergent.

↓ stack it

Part 3

Giving It a Body

From head in a jar to agent

Part 3 · Model

🧠

Model

a head in a jar

A genius brain — knows a lot, speaks brilliantly. But locked in glass: it can’t act on its own

In essence — the world’s smartest autocomplete.
Predictive text that read the whole internet

Part 3 · Tools

🦾

Tools

hands: not just say — do

“If I were you, I’d call the bank”

“I called the bank”

search the web · read files · send email · run code

From “you should…” to “done”

Linguistics · Austin — from saying to doing things with words

Part 3 · Memory

📓

Memory

an external notebook

you explain everything from scratch, every time

a stranger

knows who you are, what you discussed, how you like things

an assistant who knows you

“I remember you prefer formal tone” = reading a sticky note, not recalling an experience

Part 3 · Skills

🎓

Skills

pre-packaged know-how for typical tasks

If tools are hands, skills are a profession

A surgeon and a pianist — same hands, different skills

Physically — a folder of instructions and templates.
A playbook for one specific job

Part 3 · Wrappers

👔

Wrappers

a costume and a workplace for the same head

The head in a jar dressed as a doctor — medical AI

The head in a jar dressed as a lawyer — legal AI

The head in a jar dressed as an accountant — business assistant

The main part of the costume — the hidden prompt: you sent one sentence, the head received two pages

“100,000 AI tools” — a handful of heads, dressed 100,000 ways

Part 3 · Agent

🤖

Agent

a model with a body — and the right to act

chat single exchange

question answer stop

next stepyou decide

on erroryou retry

agent working loop

think act observe repeat

next stepit decides

on errorit retries

Not “ask AI” — “delegate to AI”

The difference between a consultant and an employee

↓ next

Part 3 · The system

The whole map

🧠 head

model

🦾 hands

tools

📓 notebook

memory

🎓 profession

skills

👔 costume

wrapper

🤖 head + body

agent

Not a “magic neural net” — a carefully assembled system of parts

The head in a jar mounted on a robot body

Parts 1 → 3

90 minutes in three lines

Part 1 — 70 years of asking: can machines think?

Part 2 — the machinery: attention, prediction. Elegant math, no magic

Part 3 — model + tools + memory + loop = agent

It predicts tokens. No body, no experience, no grounding. And yet — it writes, it reasons, it acts.

Understanding — or the greatest symbol-manipulation trick ever built? That’s a question for linguists. That’s you.

↓ next

SUCH LARGE MUCH WOW

From “Can machines think?” to agents that work for you

Can Machines Think?

Ideas & key figures, 1950–2022

Alan Turing — the question

Dartmouth — AI gets a name

The Perceptron — machines that learn

ELIZA

ELIZA — the trick

The first AI winter

The Chinese Room

Backpropagation — the answer

Data beats rules

AlexNet — deep learning wins

Word2Vec — meaning becomes geometry

The Transformer

GPT → ChatGPT

76 years in three lines

How LLMs Actually Work

From neurons to next-token prediction

The neuron

Neuron playground

Layers

Learning = shrinking the loss

Gradient descent

Before attention: reading one word at a time

Tokenizer

Embeddings — words become vectors

Vector arithmetic

Word order is not free

Attention — the intuition

Attention — the mechanism

Attention, live

The block

Three flavors

The causal mask

The whole objective

Pre-training (GPT-3)

Scale does something strange

The alignment problem

RLHF — teaching taste

model writes

human ranks

reward model

optimize

leash

Temperature

What it can’t do

The full stack

Giving It a Body

From head in a jar to agent

The whole map

90 minutes in three lines

Questions?

The whole field started with one — ask yours