Week 1 — Inferring a distribution from observations

Suppose the same text fragment is followed by $A$ three times and by $B$ once. What probabilities should a model assign on the next occurrence? The first step toward an LLM is to make this small inference explicit. The architecture will later supply a much richer dependence on the preceding text.

Course: nn-llm-syllabus. Overview: nn-llm-start-here. Authorship and supervision: ai-authorship.

Learning goals and reading

By the end of the page you should be able to fit a two-outcome distribution, perform one parameter update, distinguish sample loss from expected loss, and turn arbitrary real scores into probabilities. The derivations below require only elementary probability and differentiation.

For more exercises, see the companion notes, Chapter 1, §§1.1–1.4. Shannon’s papers are listed in nn-llm-bibliography.

1. What is observed, and what is adjusted?

A token is a chosen unit of text, such as a character, word or word part. The vocabulary is the finite set of possible tokens. A corpus is a collection of observed texts. The context is the part already supplied, and the target is the next token that was actually observed.

For now hold the context fixed and use only two possible targets. Write $A\leftrightarrow s=+1$ and $B\leftrightarrow s=-1$. Introduce a real adjustable parameter $\theta$ and define

$$p_{\theta }(s)=\frac{e^{\theta s}}{Z(\theta )},Z(\theta )=e^{\theta }+e^{-\theta }=2\cosh \theta .$$

This is the familiar two-state Gibbs formula. Here $s$ labels a token and $\theta$ is adjusted from observations; no magnetic moment or heat bath is being assumed. Positive $\theta$ favours $A$, and $\theta =0$ gives equal probabilities. We know how to compute the mean given a field: $⟨s{⟩}_{\theta }={\partial }_{\theta }\log Z=\tanh \theta$. Fitting asks for the field that reproduces an observed mean.

2. Why minimize a logarithmic loss?

Treat the $M$ targets in this one-context example as independent draws. If $n_{+}$ of them are positive and $n_{-}$ negative, their likelihood is

$$\prod _{m=1}^M{p}_{\theta }(s_m)=p_{\theta }(+{)}^{n_{+}}{p}_{\theta }(-{)}^{n_{-}},M=n_{+}+n_{-}.$$

The likelihood is a function of the parameter for the fixed observed data. We choose a parameter that makes those observations probable. Taking a logarithm preserves the maximizing value and turns the product into a sum. Changing the sign turns maximization into minimization. Dividing by $M$ makes the scale comparable across sample sizes. The resulting loss is

$$\hat{\mathcal{L}}(\theta )=-\frac{1}{M}\sum _{m=1}^M\log p_{\theta }(s_m)=\log (2\cosh \theta )-\theta \hat{m},\hat{m}=\frac{n_{+}-n_{-}}{M}.$$

The hat marks a quantity computed from a finite sample. Natural logarithms give units of nats. Differentiate only $\theta$; the observations stay fixed:

$${\hat{\mathcal{L}}}^{\prime }=\tanh \theta -\hat{m},{\hat{\mathcal{L}}}^{\prime \prime }=1-{\tanh }^2\theta ={\mathrm{Var}}_{\theta }(s)>0.$$

The stationary condition is mean matching. For $n_{+},n_{-}>0$ the unique minimum is

$$\tanh {\theta }_{\ast }=\hat{m},{\theta }_{\ast }=\frac{1}{2}\log \frac{n_{+}}{n_{-}},p_{{\theta }_{\ast }}(+)=\frac{n_{+}}{M}.$$

For three $A$‘s and one $B$, $\hat{m}=1/2$, ${\theta }_{\ast }=\frac{1}{2}\log 3\simeq 0.549306$, and $p_{{\theta }_{\ast }}(A)=3/4$. The loss falls from $\log 2\simeq 0.693147$ at $\theta =0$ to $0.562335$ nats. If one count is zero, the optimal probability is on the boundary and requires an infinite field; there is no finite minimizing $\theta$.

One step toward the fit

Gradient descent adjusts a parameter against the derivative of its loss:

$${\theta }_{r+1}={\theta }_r-\eta {\hat{\mathcal{L}}}^{\prime }({\theta }_r),\eta >0.$$

The learning rate $\eta$ sets the step size; $r$ counts updates. Starting at zero with $\eta =1$, the three-to-one sample gives ${\theta }_1=1/2$ and $p_{{\theta }_1}(A)=1/(1+e^{-1})\simeq 0.731059$. The prediction moves toward the observed frequency because the predicted mean was too small.

Since ${\hat{\mathcal{L}}}^{\prime \prime }\le 1$, Taylor’s inequality gives

$$\hat{\mathcal{L}}(\theta -\eta g)\le \hat{\mathcal{L}}(\theta )-\eta (1-\eta /2)g^2,g={\hat{\mathcal{L}}}^{\prime }(\theta ).$$

Thus $0<\eta <2$ guarantees a decrease away from a stationary point for this loss. This bound uses the maximum susceptibility of the two-state model. It is not a universal learning-rate prescription for a neural network.

3. The source, and what a finite sample can tell us

A source is an idealized distribution $q(c,x)$ over contexts $c$ and next tokens $x$. A model supplies $p_{\theta }(x\mid c)$. Its expected loss averages over fresh draws from that source:

$$\mathcal{L}(\theta )=-\sum _{c,x}q(c,x)\log p_{\theta }(x\mid c).$$

Add and subtract $\log q(x\mid c)$ inside the sum:

$$\begin{array}{rl}\mathcal{L}(\theta )& =-\sum _{c,x}q(c,x)\log q(x\mid c)+\sum _{c,x}q(c,x)\log \frac{q(x\mid c)}{p_{\theta }(x\mid c)}\\ & =H_q(X\mid C)+\sum _{c}q(c)D(q(\cdot \mid c)\Vert p_{\theta }(\cdot \mid c)).\end{array}$$

Here $H_q$ is the source’s conditional entropy and $D(q\Vert p)={\sum }_x{q}_x\log (q_x/p_x)$ is relative entropy. The convention $0\log 0=0$ is understood; if the model assigns zero probability to an event of positive source probability, the loss is infinite. Concavity of the logarithm gives $D\ge 0$, with equality only for identical distributions.

The source entropy does not depend on the model parameters. The remaining term measures mismatch. Reducing expected loss is therefore exactly reducing this mismatch. Training instead reduces a sample loss; whether that improves fresh predictions is a separate question called generalization. This is why observations kept out of fitting, called test data, are useful.

A finite test loss can lie below the entropy

Take a fair source $q=(1/2,1/2)$ and a fixed model $p=(3/4,1/4)$. Its expected loss is $-\frac{1}{2}\log (3/4)-\frac{1}{2}\log (1/4)\simeq 0.836988$, larger than $H(q)=\log 2$ as required. But an independent test sample of four draws can contain three $A$‘s and one $B$, with probability $4/16=1/4$. Its sample loss is $0.562335$, which is below $\log 2$. There is no contradiction: the entropy bound applies to the source average, not to each finite sample.

The model’s own entropy, $H(p)=-{\sum }_x{p}_x\log p_x$, is yet another average. It happens to equal $0.562335$ here because the sample frequencies equal $p$. The general cross-entropy decomposition contains $H(q)$, not $H(p)$.

4. From two outcomes to a vocabulary: softmax

For each possible next token $x$, let the model produce a real score or logit $z_x(c)$. A score is not a probability. The softmax map makes these scores positive and normalized:

$$p_{\theta }(x\mid c)=\frac{e^{z_x(c)}}{{\sum }_y{e}^{z_y(c)}}.$$

With scores $(\log 3,0)$ for $(A,B)$, softmax gives $(3/4,1/4)$, the same fit as before. Adding any common constant to both scores leaves the result unchanged. Only score differences affect the probabilities.

The logarithm of the denominator is the log-sum-exp function. For later use allow a positive scale $\beta$:

$${\mathrm{lse}}_{\beta }(z)=\frac{1}{\beta }\log \sum _x{e}^{\beta z_x},\frac{\partial {\mathrm{lse}}_{\beta }}{\partial z_x}=\frac{e^{\beta z_x}}{{\sum }_y{e}^{\beta z_y}}.$$

At $\beta =1$, the loss for one observed target $y$ is $\ell ={\mathrm{lse}}_1(z)-z_y$. Its derivative is

$$\frac{\partial \ell }{\partial z_x}=p_x-1_{x=y}.$$

The indicator is one for the observed token and zero otherwise. This is the multiclass version of the predicted-minus-observed mean in §2, and it is the starting point for training a decoder in Week 10.

Setting $E_x=-z_x$ makes softmax a Gibbs measure at unit inverse temperature: $F=-\log {\sum }_x{e}^{-E_x}$ and $\ell =E_y-F$. The identity lets us use partition function derivatives. It does not supply a heat bath, an equilibrium dynamics or a thermodynamic limit. Convexity in the scores also does not imply convexity in the weights of the network that produces them.

5. A sequence, context and perplexity

For a sequence $x_1,\dots ,x_T$, probability’s chain rule gives

$$p_{\theta }(x_1,\dots ,x_T)=\prod _{t=1}^T{p}_{\theta }(x_t\mid x_1,\dots ,x_{t-1}).$$

No independence between successive tokens is assumed. Taking $-\log$ and dividing by $T$ yields the mean next-token loss. During training these contexts come from observed text. During generation the chosen tokens are appended to form subsequent contexts, while the parameters remain fixed.

Perplexity is $\exp (\hat{\mathcal{L}})$, the inverse geometric mean probability assigned to observed targets. For an ideal predictor of $k$ equally likely alternatives it equals $k$. With mismatch it is not a literal count: a model $(0.9,0.1)$ on a fair source has population cross-entropy $-\frac{1}{2}\log (0.09)$ and its exponential is $10/3>2$.

For ideal predictors, the benefit of context is exactly

$$H_q(X)-H_q(X\mid C)=I_q(X;C)\ge 0.$$

An imperfect model may fail to realize that benefit. Week 2 computes it for a source whose correlations are known.

Checkpoints with answers

1. Which quantity is fixed in the derivative of the sample loss? The observed mean $\hat{m}$. Only $\theta$ is varied. Differentiating the data as though they changed with the model would solve a different problem.

2. Why is a highly confident model not necessarily accurate? Concentration makes $H(p)$ small, but a wrong confident prediction has a large $-\log p_y$. Accuracy concerns an average over observations, not over the model’s preferred outcomes.

3. Does a test loss below $H_q(X\mid C)$ prove an error? No. The four-draw example above gives a counterexample with a known nonzero probability. Repeated independent test samples estimate the expected loss; one fluctuation does not change its bound.

4. What changes if all vocabulary scores increase by ten? Nothing in the probabilities or loss: the common factor $e^{10}$ cancels. This is the freedom to choose an effective energy zero.

What the week establishes

Maximum likelihood gives a concrete loss and a mean-matching condition. Softmax generalizes the two-state construction to any finite vocabulary. The source-average loss separates irreducible uncertainty from model mismatch. These exact identities will remain true when a neural network produces the scores; the difficulty will lie in estimating and optimizing them.

The practical session

A guided calculation with no programming prerequisite, followed by the first browser experiment, which runs without installation.

Varied: the observed proportion $\hat{m}$, the initial parameter $\theta$, and the step size $\eta$. Held fixed: the family of distributions and the fitting criterion. Measured: the loss curve against the analytic optimum ${\theta }_{\ast }$, and whether a step with $\eta$ at or above $2$ behaves as the bound predicts.

Product: the fitted ${\theta }_{\ast }$ for at least three samples, checked against $\frac{1}{2}\log (n_{+}/n_{-})$, plus one worked instance of a step size that fails and an explanation of why the bound permitted it to.

---

Connections to other parts of the wiki

• Week 2 takes this two-state system and makes its context a preceding token, which turns the fit into a transfer matrix.
• The relative entropy of §3 is the classical case of the quantity that appears in the group’s research as the Araki–Uhlmann relative entropy of states on a von Neumann algebra, catalogued in the relative-entropy area page — the same quantity in different settings, classical and algebraic, and no more than that.
• The decomposition of §3 returns in Week 12 for the single-site conditionals of a Potts model, and in Week 14, where the same inequality is what makes a variational bound a bound.
• nn-llm-unit-0 — the block skeleton this note implements.
• nn-llm-conventions — notation, including the symbols that carry more than one meaning.

---

Start here · All weeks · Book and experiments · Next week