The Journey to AI Part 3 - Teaching Machines to Read

The Journey to AI Part 3

Teaching Machines to Read: How Words Became Coordinates

From one-hot encoding to the Transformer, the architecture that every modern AI is built on.

Part 3 of 4 in “The Journey to AI Agents”

There’s a problem nobody thinks about when they first encounter AI:

Networks eat numbers. Words aren’t numbers.

Before a neural network can understand language, someone has to answer a question that sounds almost too basic to ask:

“How do you hand the word ‘king’ to a machine that can only do math?”

Get this wrong and nothing downstream works, not translation, not chatbots, not GPT, not Claude. The answer: word embeddings, is one of the most elegant ideas in all of machine learning. And the story of how we got from there to the Transformer is the story of modern AI language.

The Naive Approaches (And Why They Instantly Break)

Attempt 1: Just number the words.

king = 1, queen = 2, apple = 3...

Broken immediately. Now the math claims queen > king, and king + queen = apple. You've smuggled nonsense arithmetic into the data. The network trains on garbage.

Attempt 2: One-hot encoding.

Give every word its own slot in a massive vector:

cat       = [0, 0, 1, 0, 0, ... 0]   ← 50,000 numbers long
dog       = [0, 1, 0, 0, 0, ... 0]
democracy = [0, 0, 0, 0, 1, ... 0]

No false arithmetic now. But: every word is exactly equidistant from every other. cat↔dog is as far as cat↔democracy. The encoding knows identity "this is word #3", but zero similarity. A network reading these vectors has no head start. It must learn that cats and dogs are related entirely from scratch, one painful example at a time.

Both attempts fail for the same reason: they’re arbitrary. The numbers carry no meaning about the words they represent.

The Big Idea: Meaning as Geometry

What if each word was a short, dense vector of, say, 300 numbers and those numbers were arranged in space so that position encoded meaning?

Similar words would sit close together. Unrelated words would be far apart. Relationships would become consistent directions in space.

In a simplified 2D version (real embeddings are ~300 dimensions):

royalty axis ↑
                       │
          king ●       │       ● queen
                       │
  ───────────────────────────────────────  gender axis →
  (male)               │                        (female)
                       │
          man  ●       │       ● woman
                       │

                 cat ●    ● dog    ← far from royalty, close to each other

Now cat and dog are neighbors. cat and democracy are across the map.

Similarity became distance. A network reading these vectors gets meaning baked into the input, a head start that used to require millions of labeled examples to learn from scratch.

Where Do the Numbers Come From?

Nobody hand-places 50,000 words in 300D space. The positions are learned from data, the same guess→check→adjust loop we built in Part 2.

The guiding principle, the distributional hypothesis:

“You shall know a word by the company it keeps.”

Words that appear in similar contexts have similar meanings. “The ___ purred on the sofa”. cat and kitten fit; democracy doesn't.

So you train a network on a seemingly dumb task: predict a word from its neighbors. Feed it billions of sentences. Every time it gets the prediction right, it has to give words-with-similar-company similar vectors. The meaningful geometry falls out as a side effect of doing the prediction well.

This is word2vec (2013). A simple task, run at massive scale, producing something profound.

Arithmetic on Meaning

Once trained, the space has structure nobody designed. Relationships become consistent directions. The famous example:

king  −  man  +  woman  ≈  queen

Read it as navigation: start at king, subtract the "male" direction, add the "female" direction → land on queen. And the direction is consistent:

Paris − France + Italy  ≈  Rome      (a "capital city" direction)
walk  − walking + swimming  ≈  swim  (a verb-tense direction)
actor − man + woman  ≈  actress       (gender applied to profession)

Nobody designed this. Symbolic AI spent decades trying to encode relationships as rules, and they emerged here spontaneously, as geometry, from a prediction task.

This is layer zero of every modern language model. GPT’s first move when it reads a word is to convert it into a vector exactly like this. Everything else operates on these meaning-vectors.

The Next Problem: Order Matters

Words have embeddings. But a sentence isn’t a bag of words:

"dog bites man"  ≠  "man bites dog"

Same three vectors. Opposite meaning. Order matters. Length varies. A plain feedforward network takes a fixed-size input and treats each element independently, no notion of sequence, no notion of “what came before.” Language needs both.

New machinery required.

RNNs: Read One Word at a Time, Remember Everything

The solution: process the sentence left to right, one word at a time, carrying a running memory of everything seen so far. That memory is the hidden state.

word1 → [network] ─h1→ [network] ─h2→ [network] ─h3→ output
            ↑same weights reused at every step↑

At each step: take (current word + previous hidden state) → produce new hidden state. The same weights run at every step. The final hidden state is a summary of the entire sentence.

This is a Recurrent Neural Network (RNN). With it, you could finally do:

• Translation (encode a sentence, decode into another language)
• Speech recognition (process audio frame by frame)
• Text generation (predict the next word, then the next…)

RNNs became the workhorse of the 2010s.

The Forgetting Problem

Here’s where RNNs broke.

To learn, backpropagation runs backward through every time step, effectively making the network as “deep” as the sequence is long. And from Part 2, we know what happens when you multiply small gradient values through many layers: they vanish.

Consequence: the network forgets early words.

The classic failure:

"I grew up in France ... [40 words of context] ... so I speak fluent ___"

To fill the blank with “French,” you must remember “France” from 40 words back. The plain RNN’s memory of it has faded to near zero. Short-range dependencies: fine. Long-range: amnesia.

LSTMs (Long Short-Term Memory networks) fixed much of this with a clever addition: a second “conveyor belt” running straight through all the steps, with learned gates controlling what to write, remember, and erase. Information could now ride the belt for many steps without being overwritten, letting the model recall “France” even 40 words later.

LSTMs ruled for years. Until two walls they couldn’t beat.

Two Walls Even LSTMs Couldn’t Break

Wall 1: The Information Bottleneck

In translation, an RNN encoder reads the entire source sentence and compresses it into one fixed-size vector, the final hidden state. The decoder generates the entire output from only that one vector, regardless of how long the input was.

input: 5 words   →  hidden state = 512 numbers
input: 500 words →  hidden state = 512 numbers  ← SAME size

More input, same container. Detail gets crushed out.

Analogy: read an entire novel, write everything down on one index card, then reproduce the novel from only that card. A short story fits. A 400-page novel? You’ve lost most of it before you start writing.

Wall 2: Sequential = Parallelism-Proof

By design, you must process word 1 before word 2, before word 3. Each step needs the previous step’s hidden state.

And from Part 2, GPUs win by doing thousands of operations at once. An inherently one-at-a-time model wastes the hardware. In the era of training on the entire internet, “one word at a time” is fatal.

Attention: Every Word Talks to Every Other Word

The solution landed in a 2015 paper and it’s called attention.

The idea: instead of forcing everything through a single fixed-size memory, let every word directly look at every other word and decide how much each one matters.

Think about resolving a pronoun:

"The animal didn't cross the street because it was too tired."

What does “it” refer to, the animal or the street? You know instantly: the animal (streets don’t get tired). To figure this out, the word “it” needs to look around the sentence and weigh each word.

That’s attention. For each word:

1. Issue a Query (Q): “What am I looking for?” (it → looking for the noun I refer to)
2. Check each word’s Key (K): each word advertises what it contains (animal → "I'm an animate noun, a subject")
3. Match Q against all K’s → a relevance score for each word
4. Softmax → scores become percentages summing to 100% (animal: 85%, street: 10%, the: 2%...)
5. Pull a weighted blend of Values (V) using those percentages

The new vector for “it” literally contains the meaning of “animal” baked into it. The word absorbed its context.

And crucially: steps 1–5 are pure matrix multiplications. Every word runs attention on every other word simultaneously, in parallel. The RNN’s fatal sequential bottleneck disappears. Distance doesn’t matter anymore. “France” 40 words back gets as direct a connection as the word right next door.

The Transformer (2017)

The paper “Attention Is All You Need” threw out recurrence entirely. No more RNNs. Just stacked attention, many layers of it, all parallel.

One Transformer block:

words + positional encoding
         ↓
  multi-head self-attention   ← every word looks at all words
         ↓
  feedforward network         ← each word processed individually
         ↓
  (stack this N times — 12, 48, 96 layers...)
         ↓
  rich, context-aware representations

Multi-head attention runs attention through multiple “lenses” simultaneously, one head might track grammar, another might track pronoun references, another might track tone, and then merges the insights like reading a sentence through several specialized readers at once.

Positional encoding adds a position stamp to each word’s vector, so the model knows word order without processing sequentially. Parallelism and order.

Stack it many times and you get something remarkable: a fully parallel architecture where every word can see every other word, where nothing is bottlenecked, where nothing is forgotten, and which scales predictably — more layers, more heads, more data → better.

Every modern language model, GPT, Claude, Gemini, BERT, is a Transformer.

What’s Next

We have the architecture. Now: what happens when you take this Transformer and train it on hundreds of billions of words from the internet? What abilities emerge? And how do you turn an autocomplete machine into an assistant and then into an agent?

Part 4: “From Text Predictor to AI Agent: The Last Leap”

Tags: NLP, Transformers, Word Embeddings, Attention Mechanism, Deep Learning, Language Models

Originally published on Medium on June 27, 2026.