Summary: How AI Generates Text Token by Token

AI generates responses through next-token prediction, a sequential process where it predicts one token at a time rather than creating complete responses instantly. When you send a prompt, the AI examines all input tokens, calculates probability scores for approximately 100,000 possible vocabulary tokens using its neural network, and selects the highest-probability option. This chosen token is then added to the original prompt, and the entire process repeats with the expanded input. The sequential streaming of words you observe isn't a visual effect—it's the actual generation mechanism, where each new token prediction depends on all previously generated tokens. The AI converts raw scores (logits) into probabilities using a mathematical function called softmax, which ensures all token probabilities sum to exactly 100%. For example, after "The sky is," the AI might assign "blue" 45% probability, "clear" 12%, and distribute remaining probabilities across other tokens. This iterative loop continues—calculating probabilities, selecting the highest-probability token, adding it to the conversation, and repeating—until the AI predicts a special "end-of-sequence" token that signals completion. Generation takes time because longer responses require hundreds of prediction cycles, each processing an increasingly long input and recalculating probabilities for all possible tokens. You now understand that AI consistently selects the highest-probability token at each step, creating predictable patterns. However, this raises an important question: if AI always chooses the most probable token, responses would be identical every time—so how can AI produce creative or varied outputs instead of repeating the same answer?

Recap

You now know that AI breaks text into tokens—small chunks that might be full words, word fragments, or punctuation. This tokenization is how AI processes language internally, and it explains why AI can't easily count letters or spell backwards.

How AI Chooses the Next Token

When you send a prompt to ChatGPT, the AI doesn't instantly generate the entire response. Instead, it predicts one token at a time.

Here's how this works:

• The AI looks at all the tokens you've provided (your prompt)
• It calculates a probability score for every possible token in its vocabulary (around 100,000 options)
• It selects the token with the highest probability
• That chosen token is added to your original prompt
• The process repeats with this expanded prompt

This is called next-token prediction, and it's the fundamental mechanism that powers all text generation in AI.

Why Responses Appear Word by Word

You've probably noticed that ChatGPT doesn't show you the complete answer all at once. Instead, words stream onto your screen one after another.

This happens because the AI genuinely generates text sequentially:

1. It predicts token 1 based on your prompt
2. It predicts token 2 based on your prompt + token 1
3. It predicts token 3 based on your prompt + token 1 + token 2
4. This continues until the response is complete

This sequential nature isn't a visual effect—it's the actual generation process. The AI must finish predicting one token before it can begin predicting the next one, because each new prediction depends on all previous tokens.

The Probability Calculation Process

When the AI calculates which token should come next, it's performing a complex mathematical operation.

After processing all the input tokens through its neural network, the AI produces a raw score for every token in its vocabulary. These raw scores (called "logits") aren't probabilities yet—they're just numbers that can be positive or negative.

To convert these scores into probabilities, the AI uses a mathematical function called softmax. This function transforms the raw scores into a probability distribution where:

• Every token gets a probability between 0% and 100%
• All probabilities add up to exactly 100%
• Tokens with higher raw scores get higher probabilities

For example, if you type "The sky is", the AI might calculate:

• "blue" → 45% probability
• "clear" → 12% probability
• "gray" → 8% probability
• "cloudy" → 7% probability
• "beautiful" → 5% probability
• (and 99,995 other tokens with smaller probabilities)

Why Generation Takes Time

You might wonder why AI responses take several seconds to generate, especially for longer answers.

The answer lies in the sequential nature of prediction. If the AI needs to generate a 200-token response, it must:

• Run the full prediction process 200 times
• Process an increasingly long input each time (your original prompt + all previously generated tokens)
• Recalculate probabilities for all 100,000 possible tokens at each step

This is why longer responses take longer to generate—the AI isn't typing slowly for dramatic effect, it's genuinely performing hundreds of prediction cycles, each one building on all the tokens that came before.

The Iterative Loop

Think of text generation as a loop that keeps running until the AI decides to stop:

Step 1: Calculate probabilities for the next token Step 2: Select the highest-probability token Step 3: Add that token to the conversation Step 4: Go back to Step 1 with the updated conversation

The AI stops this loop when it predicts a special "end-of-sequence" token, which signals that the response is complete. This is why AI responses end naturally at sentence boundaries rather than cutting off mid-thought.

What's Next

You now understand that AI always selects the highest-probability token at each step. But does this mean AI responses are completely predictable? What if you want more creative or varied outputs instead of the same answer every time?