Monthly Archives: December 2022

A hundredth of a bit of extra entropy

There are two ways to calculate the amount of information in one term of a continued fraction:

  • The entropy of the Gauss-Kuzmin distribution is about 3.4325 bits.
  • Twice the logarithm of the Khinchin-Lévy constant is about 3.4237 bits.

These differ by about 0.0088 bits. It took me a while to figure out why they were different at all, and now I’m surprised by how close they are.

The rest of this post is on LessWrong because it has equations and spoiler tags.

An exploration of GPT-2’s embedding weights

I wrote this doc in December 2021, while working at Redwood Research. It summarizes a handful of observations about GPT-2’s weights — mostly the embedding matrix, but also the LayerNorm gain parameters — that I found while doing some open-ended investigation of the model. I wanted to see how much I could learn by studying just those parameters, without looking at the attention layers, MLP layers, or activations.

The rest of this post is available on Alignment Forum and LessWrong.

A brainteaser for language models

I came up with the following puzzle the other day:

Q: Solve the puzzle: 63 = x = 65536

A: x = 

The intended answer is in the form of a number. 

text-davinci-003 guesses my intended answer at 11.8% probability, which is the second-highest probability for any answer.

(This is somewhat cherry-picked; small changes to the phrasing give worse results. ChatGPT gave the intended answer the third time I asked it, but this appears to have been dumb luck. The true rate for ChatGPT is probably below 10%, and maybe below 5%.) 

So far, friends have found it fairly difficult. About two dozen people made at least one guess, and at least six spent a while on it. So far, two people have figured it out, in both cases after being told that GPT-3.5 could do it.

For hints, the answer, and an explanation of why GPT is better at this than people are, see the LessWrong version of this post.

(WordPress doesn’t have spoiler tags.)