David R. MacIver's Blog

Temporary processing loops as a sometimes replacement for background threads

2026-02-23

There’s a trick I’ve used twice now, and I figure any trick worth using twice is worth writing up. I’ve never seen anyone else using it, possibly because it’s not actually very useful, or is secretly a bad idea for reasons that I’m unaware of. It’s obvious enough that I’d be surprised if it was original to me, but I also expect most other people haven’t seen it either.

The basic idea is this: Suppose you have a bunch of tasks running in different threads, and you need some sort of background thread running to keep them happy. For example:

In shrinkray the tasks are attempts to apply patches to a test case, and the background thread is a sort of “merge queue” which is responsible for trying to combine successful patches together.
In a recent project, I have a number of communication channels, and messages coming in from them on a single connection, and the background thread is reading those messages, figuring out where they need to go, and dispatching them to a queue for the right channel.

…those are actually the only two examples I have right now. I could probably imagine more, but those are the ones I’ve concretely tried this in.

In any case, in both of these the background thread was sortof a pain in the ass. In the shrinkray case, there was a bunch of lifecycle management I had to worry about (the “thread” was actually a trio task, and in some of these use cases it was annoying to scope it to a nursery). In the messages case, it was viable, but it was difficult to debug and I wanted it to work in a language with kinda shit threading, so I’d rather not use a background thread if I didn’t have to.

Anyway, there turns out to be a common trick: In both of these cases, we are doing a thing in the calling thread, and that thing will return back to us only once the background thread has got to processing our particular need.

In shrinkray, we only care that the merge queue has run for long enough to either accept or reject the patch.
In the message processing case, we only care that the dispatcher has run for long enough that there is at least one message in our inbox.

As a result, in both of these cases, we are essentially blocking until the dispatcher thread has got to our particular need. Which means there doesn’t need to be a dispatcher thread at all - we can just temporarily become it. Look at me, I’m the dispatcher thread now.

let’s look at some pseudocode for this. Here is how our message dispatcher might work with a background thread:


channels: dict[str, SimpleQueue] = {}

def run_dispatcher():
    while True:
        msg = get_message()
        channnels[msg.id].put(msg)

Thread(target=run_dispatcher).start()

def get_messsage(id: str):
    return channels[id].get()

And with inline processing:


channels: dict[str, SimpleQueue] = {}

lock = Lock()

def get_messsage(id: str):
    queue = channels[id]
    while queue.empty():
        with lock:
            msg = get_message()
            channnels[msg.id].put(msg)
    return queue.get()

The shrinkray one looks a bit different because the merge queue operates on many patches at a time rather than being single message like the dispatcher case, but is basically the same principle: Check if we need to become the merge thread, if we do start doing that until our patch is merged or rejected, and if not just wait on the (guaranteed to be running) merge thread until we get to that point.

Do you need this trick? No, probably not. But it has solved a genuine need for me twice, so maybe you’ll be the third time it was useful.

Speeding up conditional sampling with divide and conquer

2025-10-12

I got myself into a bit of a mess trying to explain why a particular result was true the other day, so this is my write up of a proof.

Suppose you’re sampling from some language model (large or otherwise) and you want to apply a constraint \(h\). That is, you’ve got some random variable \(S\), and you want to sample a random variable \(T\) such that \(P(T = t) \propto P(S = t) h(t)\).

One way to achieve this is rejection sampling: You just repeatedly sample from IID copies of \(S\) until you get one that satisfies \(h\).¹

The problem with this is that it’s potentially very slow. It might require a lot of samples from \(S\), especially if you make bad choices on your initial choices of characters in the sequence.

Here’s a way of improving the performance , if for each character \(c\), we can calculate \(a(c) = P(h(S) | S \text{ starts with } c)\).

We do this by reweighting the probability of each start character \(c\) by \(\tau(c)\), so we choose \(c\) with probability proportional to \(P(S \text{ starts with } c) \tau(c)\). If \(c\) is the special EOS token, we stop, otherwise we then recursively apply this method to the rest of the string, reweighting the next character in a similar way (with newly calculated probabilities).

The advantage of this method is that we never have to backtrack or repeat ourselves: Our reweighting of the characters automatically takes the constraint into account without having to ever start over again.

The disadvantage is that we have to get these \(\tau\) reweights from somewhere, which may or may not be possible in general. I’m interested in some special cases where it’s easy because the condition has a nice simple structure that just comes from deleting prefixes, but we won’t go into that here.

The reason this works is essecially a light variant of the Probabilistic divide and conquer method, which rests on the following lemma:

Suppose we have discrete random variables \(A, B\) on \(U, V\), with joint law \(q\) and some constraint \(h: U \times V \to \{0, 1\}\) with \(Z = E(h(A, B)) > 0\), and we want to sample from the conditional distribution \(A, B | h(A, B)\). That is, we want random variables \(X, Y\) such that \(P(X = x, Y = y) = \frac{q(x, y) h(x, y)}{Z}\).

Construct \(X, Y\) as follows:

Sample \(X\) such that \(P(X = x) = P(A = x | h(A, B))\).
Then sample \(Y\) such that \(P(Y = y) = P(B = y | h(A, B), A = x)\).

If you can do this, then \(P(X = x, Y = y) = \frac{q(x, y)}{Z}\) as desired.

In the original PDC paper they describe this as a simple application of Bayes’ formula, which maybe it is but I then got myself into a muddle trying to prove it. In the end I found it easier to go back to just a straightforward definition of conditional probability. First, \(P(X = x, Y = y) = P(X = x) P(Y = y | X = x)\).

We now calculate these two quantities:

\[\begin{align*} P(X = x) &= P(A = x | h(A, B)) \\ &= \frac{P(A = x, h(A, B))}{P(h(A, B))} \\ &= \frac{P(A = x, h(A, B))}{Z} \\ \end{align*}\]

Then:

\[\begin{align*} P(Y = y | X = x) &= P(B = y | A = x, h(A, B)) \\ &= \frac{P(B = y, A = x, h(A, B))}{P(A = x, h(A, B))} \\ &= \frac{q(x, y) h(x, y)}{P(A = x, h(A, B))} \\ \end{align*}\]

So then multiplying these together we get:

\[\begin{align*} P(X = x, Y = y) & = P(X = x) P(Y = y | X = x) \\ & = \frac{P(A = x, h(A, B))}{Z} \frac{q(x, y) h(x, y)}{P(A = x, h(A, B))} \\ & = \frac{q(x, y) h(x, y)}{Z} \\ \end{align*}\]

As desired.

This result is practically trivial, and it’s not obvious a priori that sampling this way is in fact any easier than the original constrained sampling problem, but the observation of PDC is that sometimes it is, and that when it is it can be a huge speed improvement.

I’m playing a bit fast and loose with random variables here and just assuming you can take IID copies of any of them. Really our “random variables” of interest are randomized programs.
↩︎

Types of enjoyment

2025-08-26

This post is inspired by a recent conversation with Amber.

I was explaining the idea of Type 2 Fun¹ - things that are only fun in retrospect - and in reflection from the conversation I think I’ve decided that the concept is stupid.

The problem is quite straightforward: It implicitly conflates “fun” with “something you enjoyed and want to do again”, and those are obviously different things. People enjoy all sorts of strong emotions all the time, and it doesn’t have to be fun in order for them to do so. Fun is a very specific emotional response, and is only one among many that people enjoy. I couldn’t - and don’t want to - draw a specific boundary around what is and isn’t fun, but I do think for example that “fun” has a sort of lightheartedness, perhaps an element of exhilaration, to it that is not present in most forms of enjoyment.

One of Amber’s examples was weight lifting. Weight lifting can be fun, but you can also enjoy it in other ways - it can be satisfying, it can be energising without being fun, it can calm you down when you’re angry. There are loads of other positive effects.

A similar example for me is massage. Getting a massage is certainly not fun.² A massage may or may not be pleasant - there’s certainly a nice fluffy relaxing sort of massage you can get that I’m sure some people enjoy a lot.³ At the moment I’m very into traditional Thai massage, which is a sort of pressure-massage-meets-assisted yoga. Quoth my most recent masseuse “…why do you like this? Nobody picks this one. People normally come to us for oil massage. Doesn’t it hurt?”. Yes, of course it hurts. The trick is not to mind. And I do enjoy getting a very heavy massage - Thai or otherwise. It hurts, but it in a very satisfying way.

I don’t think this is masochism, at least not exactly. It’s not the fact that it hurts that I’m enjoying, but there’s a very no-pain-no-gain feeling associated with it.

Another example is very frustrating games. I recently played (the first two parts) of Increasingly Cursed Wordle. A huge part of the enjoyment of this game is the incredibly angry noises you make when you figure out certain colours.⁴ These are noises of genuine anger, or at least irritation, and that is part and parcel of the experiences of enjoyment.

Playing Upwords with my dad is similar. I’ve got a running joke that every game of Upwords is unusually bad. Honestly I’ve never seen such an awful board. Ugh, why do we play this game? And then we play it again.

In the moment none of these experiences are “fun”. They’re not fun in retrospect either. But they are satisfying, and that is the experience that we are trying to get out of them.

There are other ways to enjoy things than that too. For example, people read incredibly depressing novels, or watch sad films. They may end up sobbing at them! There might be some degree of satisfaction in that, but if so it’s a very different sort than that you’ll get from Increasingly Cursed Wordle or Upwords, which I think is different again from the satisfaction you get from a massage.⁵ It’s an experience of catharsis perhaps.

“Interest” is another one. I once read “Atomic Accidents: A History of Nuclear Meltdowns”, and although I’m somewhat reluctant to use the word “enjoy” for it, I think it’s the right word.⁶ The book was horrifying and fascinating and I could not put it down.

I think this generalises, and there are very few emotional experiences that people don’t enjoy in a safe environment. I tried to come up with counterexamples - e.g. disgust seemed like an obvious choice, but actually there are lots of ways people enjoy disgust.⁷ For example some people love sharing timezone facts.

I hate the term, although it does at least have the saving grace compared to many Type 1/2 distinctions that at least “Type 1” just means “the normal thing people mean by this word”.
↩︎
I mean, ahem, depends on the type of massage. But the type of massage I routinely get certainly isn’t.
↩︎
It mostly makes me grumpy.
↩︎
Purple/Green results in especially entertaining diatribes.
↩︎
I’m not sure about lifting weights. I think that’s different again. Maybe a more similar example to lifting weights for me is breath holding, which feels more like Slay the Spire or Upwords than it does like a massage.
↩︎
Except for the radium chapter, which left me wanting to hide under my desk and cry.
↩︎
Only some of them sexual. e.g. a lot of people look at things that gross them out. Although oddly I think in that case they are having fun.
↩︎

Notes on a slightly failed spellcrafting attempt

2025-08-25

A concept I’ve found very useful in the past is what David Chapman describes as velleity:

A “velleity” is a wish so weak that it does not occur to you to act on it. These are desires you instantly dismiss because they do not match your picture of what you think you want. They seem nonsensical, unexpected, and do not fit into your plans. But they are shadow-tracks of passions you do not know you have. Pursuing them, you will capture your desire.

As mentioned in my last post, I’m struggling a bit with figuring out what it is I actually want at the moment. Or, perhaps that’s not quite right, I know what I want but it isn’t what I want to want and I’m looking for some expansion.¹

I thought I’d go on a velleity hunt, so I attempted the following exercise:

Take a blank piece of paper and at the top of it write “What do I want?”

Start writing a list of things. Whenever you get to the end, mentally think “and…” and try to autocomplete it some more.

When writing down things you want, be careful about shoulds. You’re allowed to put down outcomes you want, that doesn’t mean you want the action. e.g. my list ended up containing “for my room to be tidy” and “to write more newsletter posts”. These are things I genuinely want. Note it doesn’t say “tidy my room”, or “write (some specific newsletter post)”. At the moment I genuinely do want the outcomes, but the actual specific actions required to achieve them don’t seem remotely appealing.²

In general, don’t write down things you don’t want, but do write down things you complicatedly want. It’s OK to want things and decide not to do them, or not to do them yet, because you have other competing wants or practical constraints.

Anyway, I don’t want to say that this exercise was bad exactly. It prompted some useful introspection, but it was mostly kinda sad and depressing if I’m honest. A lot of what I want right now is apparently “be less burdened”, or “do things that distract me from problems”, or “not need to keep doing specific things that distract me from problems but seem net negative”. This is perhaps a useful insight to have but lacks much in the way of positive direction.

I suspect the exercise as designed is actually very poor for hunting velleities, and the best way to find them starts from a process of creation more than discovery. A prompt like “Here are some stupid things I could do…” is more likely to find quiet desires hidden in the noise than starting from a question like “What do I want?”, because the latter risks the big things crowding out the small things.

But I’ll save further experiments for another day.

Even there I’m not sure it’s really what I want. There’s definitely a desire to disappear off into a mathematics hole but I’m not sure it manifests in a way that’s… a fully coherent set of desires.
↩︎
Actually having written that down I notice I do have some level of desire to clean my room. Not a lot, but some. I might do five minutes or so after I finish writing this.
↩︎

Stable attention as a prerequisite for skill

2025-08-24

As is traditional when trying to recover from a drought of writing, I’m going to write about not writing.

Specifically, my current difficulty with writing seems to be primarily… what to write about.

Some context: Part of why I’ve been having “difficulty writing” is that over the last couple of months I’ve been very into mathematics. The difficult thing hasn’t actually been writing. I’ve had very little difficulty writing about mathematics, or figuring out new and interesting bits of mathematics. Much of this hasn’t happened in particularly public places, but it has happened.

I think there are at least two reasons for this, but they happened at the same time. One of them is that I’ve started a new job which very much needs me to be good at maths, so I’ve been dusting off a lot of old mathematical skills and applying them. This has been great, honestly.

The other reason is that at more or less the same time I started on Wellbutrin. My initial experience of it was also pretty great, my ongoing experience is… more mixed, but I think still positive.

Whatever the reason, the last couple of months have seen me with a very strong preference for crunchy work over squishy work.

Something I noticed almost immediately on starting it is that I lost a skill, which is the ability to tell myself stories in my head. I tend(ed) to do this a lot, especially when trying to fall asleep, and within a few days of starting Wellbutrin it seemed nearly impossible to do. I could start just fine - coming up with a premise for a story and starting to tell it isn’t any more difficult, but the problem is that I would lose the train of thought. I’d tell myself a story for a minute or two, and then I’d start thinking about something else.

Unfortunately being able to do something for only short snatches at a time before losing the thread is pretty indistinguishable from not being able to do the thing¹

Ironically I got distracted from writing this right after writing that sentence. I’m trying to use a secondary anchor to keep focus, and it is helping, but not perhaps as much as I’d like.

I originally titled this post “Holding interest as a prerequisite for skill”, but actually that’s not right. What’s needed is attention. Holding interest is one way to maintain attention, but it’s not the only way. You should, ideally, be able to hold attention on something regardless of whether you find it interesting.

And when you can’t, the skill usually falls apart. I think this is the mechanism behind the “I can only do things I find interesting” problem a lot of us with or vaguely in the direction of ADHD have, but I think it’s a general phenomenon: If you want to be good at something, you first need to be able to hold your attention on doing the thing.

Some skills eventually you grow out of needing attention on them and can do by rote, but most things will go better if you can pay attention to them.

This is true even for things that you can do without paying attention to. I’ve noticed, for example, that Pilates goes much better for me if I’m paying attention to the actual physical act I’m performing and how it feels than it does if I’m designing algorithms in my head. I’m often designing algorithms in my head anyway, so I’m not all that good at Pilates. I can still do the exercises, but not necessarily well. I do it anyway because doing Pilates badly is better than not doing Pilates, but the difference is noticeable.

I think part of the problem with the storytelling, and the writing, is that as my brain went maths brained, there was no longer a thread of interest holding my attention on the task, and as a result my mind wanders off the idea whenever I try.

I think there’s a motivation component to it too. Certainly part of the problem is that I’m choosing not to write rather than just not writing. But, at least in theory, I can solve that problem.

I don’t have to want to write in order to write, but also it’s more complicated than that. I do want to write, I just don’t find any particular thing I want to write appealling, but that doesn’t mean I won’t appreciate the result in the end. I could choose to write, I just need to be able to maintain the thread while doing so.

I don’t quite know how one cultivates this skill of holding attention without interest. Maybe it’s something meditation should help with (it certainly sounds like it should be), but I suspect also it’s just a case of doing the thing until it feels natural, so let’s see if some more daily writing fixes it.

Especially when holding attention is the main point of the activity, as it is when trying to use this to fall asleep.
↩︎

Understanding through calculation

2025-06-01

I’m in the process of trying to derust my maths skills, so I’m going to try posting about mathematics every day for June and see if that helps.

One of the things I’ve been noticing with a lot of what I’ve been reading (which has mostly been about probability) is that there are a lot of points where you want to prove some result, and the way you do that is just a bunch of algebra, and I tend to get… not exactly lost, but unsatisfied. I can understand the calculation, and I could even recreate it, but I don’t feel like I understand why the result is true more at the end than I did at the beginning.

I pointed at part of this in algebra and insight, where a result that was very easy to prove through calculation became much easier to understand by reformulating the problem with an insight. Crucially, that insight didn’t actually reduce the amount of calculation, but it replaced a completely opaque calculation with one that was more about confirming the insight actually worked.

Thing is… I don’t think you can get away without doing any uninformative calculations. Calculations are important, and sometimes what you need is an actual concrete result, and you need to calculate to do that. I think probably you can mine most calculations for insight, but I don’t think there’s any particularly deep reason that \(3 \times 106 = 318\), that’s just what you get when you run the calculation.¹

This isn’t entirely true, as a whole bunch of mental arithmetic comes down to breaking numbers down and seeing how they fit together. For example, this one is clearly true because \(3 \times 106 = 3 \times 100 + 3 \times 6 = 300 + 18 = 318\). So it’s not that the insights don’t exist, it’s more like the claim doesn’t feel like it needs justifying. The fact that \(3 \times 106 = 318\) is more or less allowed to stand on its own. It still needs checking, and if I’d told you for example that \(3 \times 106 = 315\) you should immediately have a suspicious reaction to that, so the true answer in some ways needs less insight than the false answer. I’m not sure what point I’m making and I think I’m sortof disagreeing with the main thrust of my argument at this point. I think what I mean to say is that this sort of calculation is one you might reasonably feel comfortable leaving insightless.

But I ran into an interesting example in the opposite direction this morning, where the key to understanding something that I’d been struggling with because I lacked insight was just shut up and calculate, and after doing the calculation the result was obvious.

Here’s the problem:

Suppose you’ve got some quantity \(\mu = \sum\limits_{n \geq 0} \mu_n\), and you have unbiased (but not necessarily independent) estimators \(X_i\) for the summands, i.e. \(E(X_i) = \mu_i\).How do you construct an unbiased estimator for \(\mu\)?

Here’s an easy version that I learned from Alex Lew’s PhD thesis.²

First, take some random positive integer valued variable \(N\), with \(P(N=n) = q(n)\), such that \(q(n) > 0\) for all \(n \geq 0\). Sample a value \(n\) from it, sample \(x_n\) from \(X_n\) and return \(\frac{x_n}{q(n)}\).

Why does this work? Well, it’s part of a general phenomenon of things that work, called importance sampling but we can do the calculation directly:

\[\begin{align*} E(\frac{X_N}{q(N)}) & = E(\sum\limits_{n} P(N = n) \frac{X_n}{q(n)}) \\ & = E(\sum\limits_{n} q(n) \frac{X_n}{q(n)}) \\ & = E(\sum\limits_{n} X_n) \\ & = \sum\limits_{n} E(X_n) \\ & = \sum\limits_{n} \mu_n \\ & = \mu \\ \end{align*}\]

Basically, dividing by \(q(n)\) “cancels out” the frequency with which we sample \(n\), and so as a result we get exactly the expectation we want. I’m not sure how it looks to you, but for me this calculation very much ends with me feeling like I get the result.

Something Alex pointed out to me is that in the case we’re interested in, you more or less have to evaluate the \(X_i\) in sequence.³ This makes this method quite wasteful, because it ignores all of the information you get from the first \(n - 1\) coefficients.

Instead, he pointed out that the following method works, which he described as a “Horvitz-Thompson like estimator”:

Draw \(n\) from \(N\), sample \(x_i\) from \(i = 0, \ldots, n\), and return \(\sum\limits_{i = 0}^n \frac{x_i}{P(N \geq i)}\).

I’ve spent about a week stumped about why this works. I haven’t been thinking about it constantly, but when I have I’ve been trying to gain insight into it. In particular this is apparently understandable as an application of Alex’s RAVI framework⁴, and I was very much failing to get how that worked here.

But this morning, I realised that if you frame it the right way (which Alex had already more or less said and I’d failed to realise) you can just shut up and calculate and it’s easy then.

Here’s how it works: First, draw some random finite set of integers \(S\), according to whatever distribution you like.⁵ For our use case it will be \(S = \{0, \ldots, N\}\) but the result doesn’t depend on that.

Now, construct the random variable \(T = \sum\limits_{i \in S} c_i X_i\) for some coefficients \(c_i\) that we’ll choose later. Now:

\[\begin{align*} E(T) &= E(\sum\limits_{i \in S} c_i X_i) \\ &= E(\sum\limits_{i \geq 0} 1_{i \in S} c_i X_i) \\ &= \sum\limits_{i \geq 0} E(1_{i \in S} c_i X_i) \\ &= \sum\limits_{i \geq 0} c_i E(1_{i \in S}) E(X_i)) \\ &= \sum\limits_{i \geq 0} c_i P(i \in S) \mu_i \\ \end{align*}\]

So, in order to get \(E(T) = \mu\), we just have to pick \(c_i = \frac{1}{P(i \in S)}\). When we substitute in the choice \(S = \{0, \ldots, N\}\), this gives us our desired estimator.

Again, I don’t know if you’ll feel the same way, but for me this calculation is extremely clear. I feel like I now get why this estimator works, and it works precisely because the calculation says it does. I think some of this is because I worked out the details for myself, but I do also think that in some sense the calculation is both clear and the only way that you can possibly get a result like this.

I do feel like there’s still some need for insight there. For example I think this would have been much less clear (although would have still worked just fine) if I’d used the specific instance of the set based random variable instead of the general one.

There’s maybe also a little bit of a question of how you figure out that such an estimator is reasonable in the first place. Guess and check is a perfectly valid method, but it does feel a little bit unsatisfying. On the other hand, I think given the linearity of expectation and the law of conditional expectation, it’s pretty reasonable to expect that something like this would work, so I’m not sure.

Either way, I think probably part of derusting my mathematics⁶ is going to be getting comfortable with gaining understanding through calculation, and figuring out when and how that works.

This isn’t entirely true, as a whole bunch of mental arithmetic comes down to breaking numbers down and seeing how they fit together. For example, this one is clearly true because \(3 \times 106 = 3 \times 100 + 3 \times 6 = 300 + 18 = 318\). So it’s not that the insights don’t exist, it’s more like the claim doesn’t feel like it needs justifying. The fact that \(3 \times 106 = 318\) is more or less allowed to stand on its own. It still needs checking, and if I’d told you for example that \(3 \times 106 = 315\) you should immediately have a suspicious reaction to that, so the true answer in some ways needs less insight than the false answer. I’m not sure what point I’m making and I think I’m sortof disagreeing with the main thrust of my argument at this point. I think what I mean to say is that this sort of calculation is one you might reasonably feel comfortable leaving insightless.
↩︎
You’re going to be hearing about Alex a lot in these posts I suspect, as a nontrivial amount of what I’m trying to do right now is to mainline his entire body of work until I understand all the bits of it I care about.
↩︎
Specifically they’re all unbiased estimators for \(Y^n\) for some random variable \(Y\), and you calculate \(X_n\) from \(n\) independent draws of \(Y\).
↩︎
Which is still very much at the stage where I stare at the calculations and Don’t Quite Get It for me. All of the pieces make sense, but I haven’t got it to cohere into something I can actually use yet.
↩︎
But, crucially, independently of the \(X_i\).
↩︎
Actually I’m not sure this even counts as derusting. I feel like calculation has always been the part of mathematics I was worst at.
↩︎

How to cook frozen chips

2025-05-04

One of my… I don’t quite want to say guilty pleasures. Lazy foods, comfort foods, something like that, is frozen chips. Or french fries if you prefer. Or waffle fries. Really most sorts of frozen potato product.¹ Because I can’t eat wheat, they end up being probably my second most common choice of carb after rice.

One interesting thing about these products is that they are very much a lazy meal, but they respond extremely well to a little bit of effort. If you take the default path of just pouring them out from the bag into the pan and putting them in the oven, you’ll get something a bit disappointingly soggy. If you spend about 5 minutes of prep time, you can get something genuinely very good.

Here’s how to do it right:

Place them individually in the pan so that they are well separated with an air gap between each chip.
Drizzle a bit of oil over them.²

Sprinkle with salt.³

Cook them for the amount of time it says on the packet, or even slightly less.

You probably believe that the amount of cooking time on a chip packet is almost always a lie and they need longer, and it is, but if they’re properly separated then they cook more evenly and need less time, and if you’ve added fat you’ve significantly improved the heat transfer rate.⁴ If you do the usual thing of adding 10 minutes to the cooking time you’ll get burnt chips.

This actually works even better if you use animal fats. For example, we’ve got a large jar of duck fat that we keep in the fridge.⁵ Yesterday I was making Leon’s waffle fries,⁶ and after appropriately separating them out I put a tiny dollop of duck fat on each one. The results were, frankly, spectacular. Perfectly crispy and with a much richer flavour than they normally have, for very little extra effort.

I enjoy the contrast of this. There’s something very pleasing to me about the ability to take something unfancy and make it slightly fancier. I also like it because it’s a good example of how doing something properly can make it much better for really not much additional effort.

I also enjoy it because I really like crispy salty potatoes.

Frozen baked and roast potatoes don’t count and are mostly disappointing. They’re not very good, and not that hard to do from fresh. The only real advantage of them is that they keep forever, which is admittedly a pretty big advantage when you want to improvise something out of stores.
↩︎
It occurs to me on writing this that actually it would be better to toss them in oil first. Slightly more washing up to do, but probably worth it.
↩︎
Again, if tossing with oil, add the salt at that point.
↩︎
I think the salt helps too because it draws moisture out faster, but I’m not sure.
↩︎
Because we’re classy motherfuckers. Although honestly for the amount you need to use of it it to get good results it’s actually not that expensive. Price competitive with, say, olive oil.
↩︎
To accompany a nice easy lazy meal of duck breast. Classy. Motherfuckers.
↩︎

The wheel turns

2025-05-03

Please note: This will contain some spoilers for the entire Wheel of Time series of books and up to the end of Season 3 for the show.

Some of my earliest memories of reading Proper Fiction are of the Wheel of Time books by Robert Jordan.

I’m not going to claim they’re great literature. I’m not even going to claim they’re good literature. I’ve tried rereading them as an adult and just couldn’t. But as a kid they were definitely doing something for me, above and beyond their key virtue of being long and available in airports.¹

As a result, I’ve read these books a lot over the years. Certainly every book up to around… let’s say book 8 I think, I’ve read multiple times. The Fires of Heaven, book 5, I believe I have read no fewer than four times, but that was because it was the only book I had with me in a very unpleasant situation, so I read it cover to cover several times back to back as a form of very literal escapism. I don’t think I’ve done that with any other book before or since.

Also, notably, I couldn’t tell you a single thing about what happened in The Fires of Heaven. I was going to say that this was because it was one of the books where nothing happened, but looking it up on Wikipedia I’m not sure that’s true. A bunch of quite notable things happen in it. Probably the real reason why I don’t remember it very well is because I read it in 1993 and that was, allegedly, quite some time ago.²

I feel like this series was a sufficiently big part of my childhood that I should have more memories tied up in it, but I don’t really. Fragments of conversations about it with friends and family as kids come to mind (including several about how weird Robert Jordan was about girls), and an early realisation about differences in visualisation ability from talking to a childhood friend about our respective attempts to recreate the flame and the void exercise.³

Anyway, I was pretty excited about the TV show, though I tried to temper that down to cautiously optimistic.

My emotions since have been… mixed.

They’ve made a bunch of changes to the books and I think that this is mostly good. I’m a little irked by some of them - for example, I think giving Perrin a wife only to fridge her in the first episode in order to give him a more tragic backstory was extremely Not On. I also felt like they worked a bit too hard on giving Mat a tragic backstory. And I’m not sold on the new backstory for Nynaeve either.

In general, it feels like… they didn’t like how protagonisty Rand was, and they felt the need to make everyone else have more protagonist energy in order to make the whole show more of an ensemble show than it would naturally be (which matches how the books are later, but early on Rand is definitely The Protagonist), and I’m not sure that really improved things.⁴

This also shows up in the whole “we don’t know which one of them is the dragon” thing. I thought it was an interesting conceit, but also it didn’t work. I particularly think that “We don’t know if the dragon will be a man or a woman” part really is too big a departure from the books and ruins a lot of the worldbuilding such as e.g. the role of the Red Ajah. Some of this is an attempt to update the gender politics of the books to be less… Robert Jordan… and I generally approve of that, but I think there’s a tricky balance to be struck there.

You could see the discomfort with Rand as protagonist play out particularly badly in the end of Season 2 where they basically felt the need to give everyone a participation trophy. The big battle scene at Falme was… incredibly weak, and lacked any of the drama of the books. Frankly, I hated the ending of Season 2, and that’s part of why I took so long to start on Season 3. I’d given them a pass for Season 1 ending so badly, which was apparently mostly due to COVID, but Season 2 was them writing a season ending without those problems and the result was… extremely mediocre.

I think one thing this emphasised for me is that I do have a nagging feeling that the showrunners don’t really understand what was good about Robert Jordan’s writing. That’s understandable. I don’t really understand what was good about Robert Jordan’s writing, and I used to consider myself a big fan, but I think the contrast with the show has actually helped me understand that.

And I think one thing that Robert Jordan was very good at was endings. Many or most of his books have big dramatic endings that have both major plot impact and a real sense of grandeur to them. The season endings so far… didn’t. The season 3 ending is much better than 1 or 2, but even that it feels like it was a bit lacking.

Anyway, Season 3 makes some pretty dramatic changes from the books, partly in that it mostly follows book 4 and skips book 3 (presumably to incorporate it more into season 3), and I think that’s great. I’m actually a huge fan of most of the changes.

One of the big things that I feel like we’re starting to see is that they’re pruning plotlines. For example, there’s an entire plotline with Siuan Sanche post-stilling that got um… quite aggressively pruned, and I think the series is stronger for it. I think ditching the Choedan Kal (the giant sa’angreal) is a great choice.⁵ This is I think really good, because one of the big problems Robert Jordan had as a writer was that he was absolutely incapable of doing that. Every plotline spawned two more plotlines, and a lot of why the books turned glacial as he wrote more and more of them was that there was too much going on for each book.

I’m not so sure about the change to Nynaeve overcoming her block this early on,⁶ but mostly because of the degree it required Leandrin to be holding the idiot ball to set it up.

Another change I’m cautiously a fan of is the Aviendha/Elayne relationship. I actually quite like that one of my formative series as a kid had… actually reasonably healthy depictions of poly relationships, even if they were much more along the lines of a very heterosexual polygamous / harem fantasy,⁷ and I like that that’s potentially better balanced out in the series.⁸

Anyway, Season 3 has definitely restored me to my cautious optimism. I still don’t think that this series is great, but it’s starting to manage to be a meaningful improvement on the source material, which is how I felt it might turn out at the beginning of Season 1 and didn’t really feel throughout most of Season 2. As a result, I find myself actually looking forward to Season 4.

I grew up in Saudi Arabia but came to England multiple times per year, so my childhood involved a lot of flights.
↩︎
I’m actually a little confused by this chronology because I don’t see how the events in question could have occurred in 1993, and 1994 seems much more plausible, but I was reading a hardcover copy of The Fires of Heaven for the first time, which was presumably around when it came out in 1993. But maybe this was the lag of books making it over to the UK back when the delays were longer or something.
↩︎
I never did manage to figure out how to channel though.
↩︎
I think it especially didn’t improve things for Nynaeve, who is incredibly well cast and didn’t really need the extra support.
↩︎
And not just because I think Callandor deserves better.
↩︎
In the books, that’s book 7.
↩︎
Though often reverse harem too! I don’t think it’s literally canon that the green ajah and their warders were often in relationships, but even if they weren’t literally having sex it’s very much a poly life partnership.
↩︎
Although I’m a little concerned that they’re just going to pair people up and Rand will only end up with one of them. Certainly it seems like Mat and Min are getting paired up. Rand, Aviendha, and Elayne, seems like a plausible triad though given the Aiel traditions.
↩︎

No

2025-05-02

People often talk about needing to be better at saying no.

This is treated as a monolithic skill or general character trait, but I think this might be another victim of metonymy, and there are actually a number of very different circumstances under which you might need to learn this skill. For example, here are a few:¹

Saying no to someone who has authority and will legitimately pressure you to say yes.
Saying no to someone who has authority but does actually want you to say no if you can’t do the thing but you’re afraid to disappoint them.
Saying no to things that you do want to do when you’re too tired to do them.
Saying no when you really don’t want to do a thing but know the person who would do it instead also doesn’t want to do the thing.
Saying no to a friend who consistently asks for favors but rarely reciprocates.
Saying no to family members who expect you to participate in traditions that you no longer feel comfortable in.
Saying no to a romantic partner who wants to move faster in the relationship than you’re comfortable with.
Saying no to a project at work that you could help on but isn’t what you want to be doing.
Saying no to a child who wants something that isn’t good for them, despite tears.
Saying no to participating in gossip or conversations that make you uncomfortable.
Saying no to a request that you think is unethicaly.
Saying no to requests for information that feel inappropriate..
Saying no to taking on additional responsibilities when others aren’t doing their share.
Saying no to providing help when you are legitimately the best person to do it but don’t have the capacity to do so right now.
Saying no to providing help when you are legitimately the best person to do it when you could do it but really don’t want to.

For me, I’m pretty good at saying no in circumstances where I’m really over capacity, especially if I don’t particularly care about the asker. I’m also good at saying no even when there’s a power gradient.

However, I’m very bad at saying no to doing something when someone I care about would have to do it instead and also clearly doesn’t want to do it.

In some sense, I endorse being bad at that, you should do things for people you care about, but also that caring is (hopefully) reciprocated, and you should let them do that for you too, and that’s the part I’m bad at letting (or encouraging to) happen. Also you’ll often end up in specific instances where someone’s gotta and sometimes it’s easier to just do the thing than navigate who that someone should be.

Regardless though, this feels like a much more specific problem to engage with, for at least two reasons:

You can look at a specific circumstance and actually engage with the concrete feelings around it much better than you can a sort of vague “I’m bad at saying no”.
You can ask for help on specific circumstances, especially recurring ones, in a way that you can’t with the general. Rather than “I need to get better at saying no”, you can start with “I need to ask for help here”, and it’s a concrete enough problem that help will hopefully be forthcoming.

Claude helped with some of these, but I deleted about 50% of its suggestions and edited most of the rest.
↩︎

Studying specifics

2025-05-01

There’s something I deeply believe, and only very erratically manage to follow through on, which is that there’s a huge amount to be learned by studying something more than it seemingly warrants. This is in some sense the core of the post Overthinking overthinking which is what created the “Overthinking Everything” name that I now use for my newsletter and community.

One core reason for this is existence proofs. If you put more effort into solving a problem than it really warrants, you’ve now demonstrated that that problem was solvable, and you can potentially use that solution in other cases, gradually reducing its effort.

I think this relates to one of my points in Heavy up front learning requirements, which is that it’s worth spending some time heavily studying skills you use regularly, even ones that don’t seem like they need it. Sometimes just by really levelling up one specific task, you should be able to learn generalisable lessons.

The problem is of course… how do you actually do this? How do you pick what specifics to study?

One part of the answer is to just do which ones bring you joy, but I think there’s a key problem with this: The specific prescription is to study something more than it warrants. If it brings you joy to study it, hopefully you’re already using that joy to prompt you to spend more time on it.¹

You could study everything more than it warrants, but then you rapidly hit resource constraings. Where are you going to find time for that?

I think this is, often, the bottleneck on the joy version too: There are many things that if you just followed your curiosity you’d find would bring you joy, but there are too many things, and as a result you do none of them.

Instead, a better solution I think is to pick arbitrarily and capriciously. Perhaps, ideally, at random from those that you think seem like they would be interesting to do.

One of my recurring fantasy scenarios is what I’d do with an improbably large amount of money, and one of the regular projects I think about is just randomly sampling a few hundred people from the population and giving them near unlimited healthcare - extensive investigations of anything that’s a problem for them, suitable interventions (erring on the side of cheap and low risk ones), and then studying the data from that extensively to learn useable lessons for the broader population.

This strikes me as the right sort of general tool: By sampling randomly from the set of places you could intervene, you learn plenty of things that will generalise out of that.

I don’t yet have a very good idea of how to implement this though.

Right? Right?? Um, OK, this theory might have some problems.
↩︎

Heavy up front learning requirements

2025-04-30

So, Dave and I went to Cambridge recently to visit Dani Clode at the neuroplasticity lab. She developed a device called the third thumb, which is pretty much what it sounds like - it’s an extra thumb you can attach to your hand on the opposite side of your normal thumb.

It’s designed specifically as an augmentation, not a prosthesis, which I thought was an interesting choice. The difference is essentially that it’s not designed to exactly mimic a biological thumb, but instead has a couple of adjustments that are there to help you use it in ways that you wouldn’t normally use a thumb. e.g. they’ve got an example of essentially wrapping the thumb around a bottle while you twist the cap on and off with your hand.

One of the things that was interesting about it was that it was just very hard to use.¹ I was starting to get the hang of it by the end, but I was just constantly dropping things or holding them in weird ways that didn’t make much sense and often didn’t even actually use the third thumb. Some of that is just that it’s foot controlled,² vnd I don’t have very good foot control, but I think a lot of it’s just that it is genuinely quite unintuitive. Dani told us that they’ve got a week-long training course that they run with these to show people how to use them, and people usually have the hang of it by the end of the week.

And I thought that was pretty interesting in its own right, because it’s fairly rare that I encounter a new sort of device that requires this much training to use. Implicitly, you do have that much training in how to use a keyboard or how to use a mouse but particularly with the keyboard example, there’s an easy onboarding route, you can always hunt and peck, right? It’s not that you need a huge amount of training in order to be able to use the keyboard at all. It’s that there is a fluency that you acquire through time and practice, but the baseline level of disfluency is still pretty effective.

The third thumb wasn’t like that at all. I was making a complete mess of things, as early on there is no equivalent to hunt and peck, and my ability to do things was made strictly worse by using the third thumb. You have to just have to figure out how to use it before it’s anything other than in the way. Now, we were not encountering it in the proper training setting, so this is very much not me saying that the third thumb is not a usable object, I’m saying that there is an effort required to onboard to using it it, and this is just an interesting contrast with most of the devices that we encounter.

Dave and Lisa are currently learning to drive, which seems in some ways like a good analogy to that, in that there isn’t an easy onboard to learning to drive. But honestly this is partly because there is a high safety requirement, and partly because you need to not just know how to use the car you need to know how to use it in a way that is predictable to others. You could learn to drive by just messing around in a car and seeing how that works. There’s nothing that in theory stops you from doing this, and presumably many early car users did learn to drive this way - formal licensing of drivers is a much more modern form than the car itself - but regardless of how artificial the gap is there’s still this sort of onboarding requirement.

In contrast, the most transformative technology I can think of that’s happened to me in recent years is still the smartphone.³ Certainly this is probably the biggest example of augmentation in our day-to-day lives. But the smartphone is very much not like a third thumb or a car. It is designed with continuity to the mobile phone from before, which is designed for continuity with previous devices, and it’s designed to be easy for kids to pick up,⁴ and is generally a fairly gentle onboarding process. Although your use of it will improve with practice, you don’t really have to spend a whole bunch of time sitting down with your smartphone and trying to figure out how to use it before it’s useful, instead it starts useful and becomes more useful over time.

And it seems to me like there’s this split of how much effort you have to invest in before you know how to use a particular type of technology. It’s not exactly shallow vs steep learning curves, so much as it’s about time to first usefulness. There can be a lot of depth to learning a new technology, but if it’s rewarding from early on then it will be a very different experience from something with a high up front cost, and I feel like we’ve gotten very used to the idea that everything should be more like the smartphone than the third thumb.

This does on some level make sense of course. In some sense it’s clearly better to have this than not, right? It’s much more rewarding to immediately be able to use a thing and work from there to improve it than it is to have to invest a whole bunch of effort into something that isn’t useful until you spend that effort.⁵, but it seems like there’s limitations to that approach where not all things can work like that, and that up front investment feels extremely daunting and unwelcome.

Part of why I care about this is that one of the things I like about the third thumb is that I do have a pervasive sense that the world should be weirder than it is. I think this is partly just my “where’s my flying car” instinct. I do have a sort of very science fiction type attachment to the idea that you can expand human capabilities, and feel like it’s a shame that we’ve largely not gone down that route and instead just embraced phones as a shitty ad-supported version of our cyborg future.

I don’t know how you would ever make physical augmentation useful from day one, because it’s such a fundamental change to your way of being in the world. In sensory augmentation, maybe, something like the North Paw is a good example. Because it’s just providing you feedback. You can consciously interpret that feedback and gradually you internalize it over time. But something like the third thumb or prehensile tail, I don’t know how well you can possibly make that an incremental approach, and as a result this attachment to easy learning experiences feels like a limitation on human progress.

I’ve often thought that there just is a lot of sort of missing space in the range of UIs that we try out. And some of that is many of the alternatives we tried out just aren’t very good. You can sort of see this with VR where it turns into, like, turned out to be like a quite limited paradigm that we still haven’t figured out how to use well. But I was really interested in the idea of using the Kinect or Leap Motion as styles of interaction with a computer and doesn’t seem to have turned out that way, and I don’t think that was just the limitations of the technology, I think it was the degree to which it required both the users and the developers to actually learn something new.

And I think these sorts of cliffs where you have to do active training are part of that. I think people have gotten as used to the idea that UI should be seamless and intuitive as they are used to the idea that websites should be free and that seems like a shame.

Another problem with this is that even things where you don’t have to invest this sort of trained skill practice into, I bet we would actually benefit from deliberate practice. I bet actually, if I spent a week studying how to better use my smartphone and better fit it into my life, I probably would figure out important and interesting things about how to use it.

I think something I notice is that there is a general lack of deliberate practice for our everyday tools and habits built into life, and it seems like we are missing something as a result.

For me. Dave found it very natural
↩︎
You get two degrees of freedom. One with your left foot to move the thumb up and down, and one with your right foot which you use to grip.
↩︎
I know that’s not that recent anymore but I can’t think of anything as big since. AI isn’t there yet.
↩︎
Too easy if anything.
↩︎
In that it turns learning it into an anytime project
↩︎

Where are your limits?

2025-04-29

A recurring theme recently has been doing things because they need to doing, regardless of whether you currently feel like doing them.

I’ve got some pushback on this, especially on using this daily writing habit as an example, particularly around how “need” is fake.

And it’s true. Need is in some sense fake. You can always skip the dishes and let them pile up in a festering heap, rendering your kitchen ever more difficult to use. You can always let the apples rot. There’s very little you need to do in the sense that it is a logical necessity that you must do it.

But on the other hand, there are a lot of things that you should, on balance, choose to do, and being able to bind your future self to that path, and to be beholden by the commitments of your past self, is often going to go far better for you than not doing it. I might not want to do the dishes now, but future me would much rather that present me had, and even present me will probably feel better for doing it once I get over the initial hump. Being useful feels better than not being useful, most of the time.

But do I need to keep writing every day? Probably not. The biggest problem I had writing today was when figuring out what to write about was once again asking myself the question: Well, what am I writing for? If I don’t really know why I’m writing, it doesn’t really matter what I write about.¹

In some sense, the reason why I’m writing is that I am writing is because writing every day is a good way to discover interesting things and articulate things you otherwise wouldn’t have articulated. This is a true and important reason and is a key part of why my past self decided to do it. But right now I don’t feel like I care about that.

The reason I am writing is because I’ve decided that I write every day. Eventually that will stop being enough, but right now it still is.

But I think there’s a question of… OK, past you has decided that you’re going to do a thing, or present you believes that you need to do a thing. When are they wrong?

One of my failure modes is that I’m not good enough at asking for help, and one of the ways this shows up is that if I’ve decided that I need to do a thing, I will usually do the thing.

Part of this is that depression is a bias towards inaction, and doing more than you feel like you can helps drag your way out of depression. So I legitimately don’t trust the feeling that something is too hard or too much, and I will generally feel better for doing it.

…except when I don’t. Because sometimes it’s actually a bad idea for me to do the thing, and I should make an exception.

I made an exception for daily writing over Easter weekend and that felt like the right thing to do.²

Sometimes I have to be chased out of the kitchen because I’m clearly wobbling when I try to stand up and someone else should do the dishes. Sometimes everyone’s in this state and we have to admit that OK the dishes can be done tomorrow.

The question, I think, is when you should make these sorts of exceptions.

The Galahad Principle says you shouldn’t make any exceptions! But I think that we can weasel our way out of them by saying that the thing we’re committing to 100% on is no unprincipled exceptions. Act only according to exceptions that you would be willing to universalise. That is, if you made an exception in every case like this, would that be OK?

I don’t think this is quite sufficient though. Sometimes you experience genuine changes. If I were to lose my ability to type for whatever reason,³ I’d probably stop the daily writing, and that seems legitimate but also does call into question the “necessity” of doing it.

So I think as well as the ability to make exceptions you need to find out where your limits are, and learn what to do about them. It’s not always to accept them. Sometimes you do just need to work harder.

But sometimes you need to acknowledge that the part of you that was saying that this was too hard was right, and actually decide that some things don’t need doing and change your commitments accordingly.

Not today though.

This continues to be a singularly unhelpful answer.
↩︎
I’m actually not sure that it was! I do feel like I’ve taken a significant hit to the habit since returning. But on balance it was still probably correct.
↩︎
Which would be very bad for dayjob reasons.
↩︎

Motivation gradients

2025-04-28

There’s a type of depression that I’ve written about before where it feels like a restriction of the possible. You want to do normal things, but doing them feels completely impossible, even when in a more literal sense it’s perfectly possible to do them.

But there’s another type where everything just feels sortof flat. It’s not actually difficult to do things - there’s not much internal resistance per se, but there’s also no ability to orient. The feeling is not “I can’t do things”, but “I can do things, but why would I do this thing in particular?”. When presented with any particular choice of writing topic, the bit where interest or disinterest would be responds as a sort of emptiness. It’s not an unwillingness to do the thing, it’s more like a total directionless neutrality on the subject.

Anyway this is what I’m feeling about writing right at this minute, so I’m just writing down the phenomenology of what I’m currently experiencing, because that seems like a good default.

One of the interesting things is that I’d lost my pad of writing ideas.¹ I initially thought this might be the source of the problem, so I tried to generate some more ideas, and I didn’t actually find it very hard to generate a new list of ideas. Many of them are ones that I abstractly recognise would be good to write about, but I look at them and try to imagine writing about them right now the answer is still that empty sort of why.

I think part of the problem is heat, honestly. It’s about 25C in my office right now, which isn’t scorching, but that’s after significant attempts at heat management, and I think part of what’s happening is that my brain and body are going into a sleepy heat crash.

One perhaps telling indication of this is that as soon as I thought of that and started thinking about steps to take to reduce heat in this room it became very easy to take actions for that.²

The weird thing is that I don’t feel particularly hot. I feel… lethargic maybe, but I’m not getting any strong signals about temperature, just behavioural responses to it.

One of the things I pointed out in labelling feelings is that you’re allowed to figure out what you’re feeling based on observing your own actions like you would for anyone else. Something I hadn’t noticed before is that when trying to figure out what you want, you can partially infer this by finding things that are unusually easy for you to do right now.

I think a lot of the time this will end up in unendorsed actions, like doomscrolling, but I think if you cast around and explore the space of alternatives maybe you’ll find there are some unexpected things in there that are much easier to do than expected because how easily you respond to the idea of doing something towards them.

Bad labelling and organisation on my part got it mixed in with a bunch of other identical pads of paper and I couldn’t find which one was it.
↩︎
Mostly I bought fittings so I can get out my proper AC unit and install it somewhere useful, and I filled up and turned on the swamp cooler I have in my room.
↩︎

Speaking from the heart

2025-04-27

“Focusing” is the name for a basic therapy skill coined by Eugene Gendlin.

The story goes¹ that Gendlin and his students wanted to figure out what caused some people to improve in therapy and others not, so they looked at transcripts of lots of therapy patients and analysed them for what made the difference. Eventually they realised that the biggest impact was not from anything the therapists did, but whether the patients said “Um” a lot.

The explanation for this observation is this: If you’re speaking effortlessly and fluently about your experiences, you probably aren’t paying attention to how you’re feeling, and in particular you’re not checking in with whether what you’re saying actually feels like. If you imagine someone saying “Yup everything is absolutely great, I’m doing fine” while their face calls them a liar, that’s the sort of thing I’m talking about. Frequently people will do this not because they’re lying, but because they don’t even notice that they’re not telling the truth.

When progress happens, they instead say “Yup everything is absolutely great, I’m doing fine. Um. Except…” and then start talking about the actually important thing.

Gendlin describes the process of realising what you said wasn’t true and noticing the feeling you get about it as paying attention to it as noticing the felt sense of the problem. This example comes from an exercise he suggests somewhere² is to say out loud “Everything in my life is completely fine.”, notice how that feels, and then start saying “Except…” until you run out of things to list.

Anyway this is the core of Focusing as I’ve understood it to date: It’s a fundamentally corrective process, where you say things, then you find out the ways in which they’re not true, and you use the felt sense to correct them. I’m pretty good at this part.³

But there’s another part to it that I’ve never been very good at, which is where you let the felt sense “speak directly”.

One problem with the speaking first and correcting it is that thoughts are fast and feelings are slow, and it’s very easy for whatever part of you that is feeling the thing to get overwhelmed, even when you’re trying your best to be helpful. I was trying to explain this to Lisa the other day and the analogy I used was imagine talking to a shy child who isn’t very good at speaking yet. Even if you correctly anticipate what it’s trying to say to you, the mere act of anticipating it can make it less willing to talk.

What you (or at least I) have to do instead is say some words, slowly, and then if it’s not actually clear to you what the next words to come out be that would feel right are… pause, and wait for them to come. Sit there, just letting there be a long gap in your sentence, until the next words come out.

The frustrating thing about this is that it works much much better if there’s someone listening, and that someone has to sit patiently with you and wait. This is hard.

I mentioned I was explaining this to Lisa the other day. This was a very good conversation in which I was trying to express some things I was feeling sad about, and part of it lead to a very good meta conversation about how to have such conversations. One thing that came up is that Lisa and I have very different internal experiences when we’re clearly struggling with words, and that in the ones that Lisa has and thus was expecting I was also having, trying to help complete the other party’s sentences is actually very helpful.

I think there are roughly three things I experience that lead to this sort of halting speech:

This thing of waiting for a felt sense to express itself properly.
“Tip of my tongue” style aphasia where I know the word but I’m struggling to find it.
Situations where I know what I want to say but feel paralysed from saying it, out of fear or similar.

In the second two, completing a sentence correctly is quite useful. It either tells you there’s nothing to be afraid of because they already know what you’re going to say, or it gets you out of the aphasia.⁴

Even in the classic “corrective” sense of Focusing, where you’re trying to express a sentence that feels true, completing the sentence is quite helpful, because it gives you something concrete to check against.

But… when trying to express myself around something deeply felt and poorly understood, I think completing the sentence doesn’t help even when the completion is right, because much of the point is not just to say the true thing, but to experience the feeling fully, and if you skip to the end you don’t get to do that, and the whole process is left incomplete.

The result though is that you need patience when talking to someone doing this, even when that someone is yourself. It’s very easy to return to the lightning quick world of words and race ahead, but sometimes you just have to sit with the feeling until it’s ready to talk.

I believe that this story is apocryphal and Gendlin already knew the conclusion he wanted to reach and then used this as a sort of post hoc justification for it. But it seems both plausible to me that this is what you observe and also I think the method works, so it’s a useful illustration regardless.
↩︎
This might have been in his book, Focusing, but I think I remember this as a suggestion from a friend reporting on Gendlin, so it might also be something else.
↩︎
Thus granting me a good grade in therapy, a thing that is both normal to want and possible to achieve.
↩︎
I actually don’t like it when people help me out when I’m having aphasia, but that’s a different and much less important issue.
↩︎

More ways to work out sums

2025-04-26

Sorry, still on different ways to calculate the sum of the first n integers. I’ll get bored eventually.

My last post Introducing a parameter to work out sums was originally called “Using generating functions to work out sums”, but the things I was using in it weren’t generating functions, and doing it with a generating function ends up much nicer, in a way that is obvious in retrospect.

The generating function of a series \(a_n\) is \(\sum\limits_{n \geq 0} a_n x^n\). Often you can work out the generating function then take its power series expansion to calculuate jhe original series.

Here’s how it works:

\[\begin{align*} f(x) &= \sum\limits_{n >= 0} \sum\limits_{i = 1}^n i x^n \\ &= \sum\limits_{i >= 1} \sum\limits_{n \geq i} i x^n \\ &= \sum\limits_{i >= 1} i \sum\limits_{n \geq i} x^n \\ &= \sum\limits_{i >= 1} \frac{i x^i}{1 - x} \\ &= \frac{x}{1-x} \sum\limits_{i >= 1} i x^{i - 1}\\ &= \frac{x}{1-x} \frac{d}{dx} \sum\limits_{i >= 1} x^i \\ &= \frac{x}{1-x} \frac{d}{dx} \sum\limits_{i >= 0} x^i \\ &= \frac{x}{1-x} \frac{d}{dx} \frac{1}{1 - x} \\ &= \frac{x}{(1-x)^3} \\ \end{align*}\]

Rearranging order of sums like this is a fairly standard trick for working out doubly infinite sums. It’s technically justified whenever the sums absolutely converge.

Now, to get the power series, we just need to work out the kth derivative of \(\frac{1}{(1 - x)^3}\). This is \(\frac{3 (3 + 1) \ldots (3 + k - 1)}{(1 - x)^{3 + k}} = \frac{(k + 2)!}{(1 - x)^{3+k}}\).¹

So, plugging in \(x = 0\) to get the derivates at 0 we get

\[\begin{align*} f(x) &= \frac{x}{(1-x)^3} \\ &= x \sum\limits_{n \geq 0} \frac{(n + 2)!}{2 n!} x^n \\ &= \sum\limits_{n \geq 0} \frac{(n + 1)(n + 2}{2} x^{n + 1} \\ &= \sum\limits_{n \geq 1} \frac{n (n + 1)}{2} x^{n + 1} \\ \end{align*}\]

Giving us, once again, the desired result.

You could use a basically similar argument to calculate \(\sum\limits_{n \geq k} n (n - 1) \ldots (n - k)\), and from that work out \(\sum i^k\) for any \(k\), but I can’t be bothered at present.

In order to not salami slice this topic over too many more posts, here’s another proof I spotted yesterday, which uses a standard combinatorial trick of counting the same thing two different ways:

Consider all distinct pairs of numbers drawn from \(\{1, \ldots, n + 1\}\). Counted one way, this is just \({{n + 1}\choose 2} = \frac{n (n + 1)}{2}\).

Now, count it another way: If the smaller number in the pair is \(k\), then there are exactly \(n + 1 - k\) choices for the larger number. Therefore there are \(\sum\limits_{k=1}^n (n + 1 - k) = \sum\limits_{k = 1}^n k\) choices in total, so \(\sum\limits_{k = 1}^n k = \frac{n (n + 1)}{2}\) as desired.

As an additional note, one thing I’ve been noticing when working these out is just how much better pen² and paper is for working these out than trying to type them up on the computer is. Part of this is that my notebook doesn’t have the best affordances around LaTeX, but I think a lot of it is genuine ease. Mathematics really still feels like it’s best done by hand to me.

I think that if I were using an actual interactive theorem prover, or even just a symbolic algebra library in Python, or the like that might stop being the case, but writing LaTeX feels too obviously like a presentation tool, not a tool for actually working things out.

You can prove this with induction if you like. This is a good example of the sort of thing that induction is useful for, where you spot the pattern and just want to go “and so on”. But you can also just go “and so on” if you feel like it.
↩︎
I know, I know, a true mathematician would use a pencil.
↩︎

Introducing a parameter to work out sums

2025-04-25

In my last post I talked about the problem of working out \(\sum\limits_{i = 1}^n i\). In it, I strongly implied that there wasn’t a way to just work out the answer.

This isn’t actually true. There’s a quite straightforward way to just work out the answer. It still requires a little bit of spotting the trick, but the tricks in question are more like puzzle pieces that you have to fit together than genuine insights.

However, it requires a bunch more calculus than I wanted to assume in my last post, so I left it out. If you do know the calculus it might be informative though.

The idea is this: Rather than work out \(\sum\limits_{i = 1}^n i\), we work out \(f_n(x) = \sum\limits_{i = 1}^n i x^i\), and then evaluate \(f(1)\).

This will, weirdly, turn out to be easier.

How do we do this? Like so:

\[\begin{align*} f_n(x) & = \sum\limits_{i = 1}^n i x^i \\ & = x \sum\limits_{i = 1}^n i x^{i - 1} \\ & = x \frac{d}{dx} \sum\limits_{i = 1}^n x^i \\ & = x \frac{d}{dx} \frac{x^{n + 1} - 1}{x - 1} \\ & = x \left(\frac{(n + 1)x^n}{x - 1} - \frac{x^{n + 1} - 1}{(x - 1)^2} \right) \\ & = x \left(\frac{(n + 1)x^n(x - 1) - x^{n + 1} + 1}{(x - 1)^2} \right) \\ & = x \left(\frac{(n + 1)x^{n + 1} - (n + 1)x^n - x^{n + 1} + 1}{(x - 1)^2} \right) \\ & = x \left(\frac{nx^{n + 1} - (n + 1)x^n + 1}{(x - 1)^2} \right) \\ \end{align*}\]

We want to evaluate \(f_n(1)\), but this would require us to divide 0 by 0, so we have to take limits. We can do this with two applications of L’Hôpital’s rule.

\[\begin{align*} \lim_{x \to 1} x \frac{nx^{n + 1} - (n + 1)x^n + 1}{(x - 1)^2} &= \lim_{x \to 1} x \lim_{x \to 1} \frac{n (n + 1) x^n - (n + 1) n x^{n - 1}}{2(x - 1)} \\ & = \frac{n (n + 1) n x^{n - 1} - (n + 1) n (n - 1) x^{n - 2}}{2} \\ & = \lim_{x \to 1} \frac{n (n + 1) x^{n - 2} (n x - n + 1)}{2} \\ & = \frac{n (n + 1)}{2} \\ \end{align*}\]

Which is the desired result.

Now, I think it would be reasonable to consider this approach to be in almost every way worse than the last one. I don’t disagree. That was a whole bunch of symbol shuffling from which you cannot, I think, derive much insight about the nature of the problem.

But, on the other hand, I think it’s worth realising that sometimes that’s good actually. Sometimes you don’t really need the insight, or can’t find it, and you just want to shut up and calculate. This sort of tool is very useful to have handy for that.

For example, suppose you wanted to calculate \(\sum\limits_{i=1}^n i^2\) instead. Our previous methods don’t help us here because there are no nice sums to rearrange. But with this technique it’s actually quite straightforward (if, admittedly, ugly).

First, let’s make use of our previously solved problem to simplify this slightly. We’re actually going to calculate \(\sum\limits_{i=1}^n i(i - 1)\). We can then just add an extra \(\frac{n(n + 1)}{2}\) to the result to get our desired sum.

Now, let \(g(x) = \sum\limits_{i=1}^n i(i - 1) x^{i - 2}\). This is now the second derivative of \(\sum\limits_{i = 1}^n x^i\), which we can work out¹ is \(\frac{x^n (n^2 (x - 1)^2 - n x^2 + n + 2 x) - 2 x)}{(x - 1)^3 x}\)

Taking three derivatives with L’Hôpital’s rule² we get:

\[\begin{align*} g(1) &= \lim_{x \to 1} \frac{x^n (n^2 (x - 1)^2 - n x^2 + n + 2 x) - 2 x)}{((x - 1)^3 x)}\\ &= \lim_{x \to 1} \frac{ (n - 1) n (n + 1) x^(n - 3) (n^2 (x - 1)^2 + 2 n (x^2 - 1) + 2 x)}{6(4x - 3)} \\ & = \frac{ (n - 1) n (n + 1) (2)}{6} \\ & = \frac{ (n - 1) n (n + 1) }{3} \\ \end{align*}\]

This means that our desired result is

\[\begin{align*} \sum\limits_{i = 1}^n i^2 & = \frac{(n - 1) n (n + 1) }{3} + \frac{n(n + 1)}{2} \\ & = \frac{2 (n - 1) n (n + 1) + 3 n(n + 1) }{6} \\ & = \frac{n (n + 1) ( 2n - 2 + 3)}{6} \\ & = \frac{n (n + 1) ( 2n + 1)}{6} \\ \end{align*}\]

This is³ the correct answer, as you can verify by checking the first couple results.

Now… Do you ever need to be able to do this? Honestly, not really. I get the impression that this used to be a much more important skill, and these days you can mostly use symbolic algebra packages to do it for you, and also you mostly don’t need it anyway, but mathematicians used to get very excited about this sort of formula, and I think they’re still important in a bunch of places. e.g. I bet people who do combinatorics care much more about being able to do calculations like this than I do, because you often reduce counting problems to this sort of calculation.

Personally, I like it at least partly for its illustrative value. It’s sometimes very helpful to know that you could solve a problem if you were just willing to shut up, sit down, and do the algebra. There’s also a satisfying sudoku-like quality to do it.

It’s also interesting having this sort of skillset at all because I get the feeling that basically nobody learns it any more. I learned it because I read “Schaum’s Outlines of Advanced Calculus” at a formative age, and it taught me all sorts of tricks like this, but it meant I went into university with this whole bag of tricks that nobody else had. They weren’t useful tricks to have exactly, but they were interesting, and I do feel like it shaped how I look at mathematics a fair bit.

I asked Wolfram Alpha, because I was lazy, but you could do this by hand through the application of the usual rules for calculating derivatives if you can be bothered. I mostly didn’t because writing it up in my notebook software is a pain.
↩︎
Wolfram alpha again. Sorry.
↩︎
Conveniently, because I really didn’t want to track down the error if I made one.
↩︎

A brief introduction to mathematics with one example many times

2025-04-24

There’s a story repeated so often that it’s a cliche of Gauss’s mathematics teacher setting the class the task of adding up the numbers 1 to 100, supposedly to waste their time and give them a break, only for Gauss to almost immediately give the correct answer.

I think it’s a usefully illustrative example to look at for learning some of the basics of mathematics. Partly this is because it illustrates one of the key uses and methods of mathematics: Getting an answer that you could in theory work out laboriously much faster through logical deducation.

Mostly, though, I’m going to use it to showcase a bunch of different ways of thinking through a mathematical problem, and use it to teach some common mathematical notation.

So, let’s figure out how to do this.

The first thing we need to figure out is to how to write it down. Here are two obvious ways:

“The sum of all numbers from 1 to 100.”
1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + 10 + 11 + 12 + 13 + 14 + 15 + 16 + 17 + 18 + 19 + 20 + 21 + 22 + 23 + 24 + 25 + 26 + 27 + 28 + 29 + 30 + 31 + 32 + 33 + 34 + 35 + 36 + 37 + 38 + 39 + 40 + 41 + 42 + 43 + 44 + 45 + 46 + 47 + 48 + 49 + 50 + 51 + 52 + 53 + 54 + 55 + 56 + 57 + 58 + 59 + 60 + 61 + 62 + 63 + 64 + 65 + 66 + 67 + 68 + 69 + 70 + 71 + 72 + 73 + 74 + 75 + 76 + 77 + 78 + 79 + 80 + 81 + 82 + 83 + 84 + 85 + 86 + 87 + 88 + 89 + 90 + 91 + 92 + 93 + 94 + 95 + 96 + 97 + 98 + 99 + 100

Neither of these are actually very good.

One of the first lessons of doing mathematics: A lot of mathematics is about developing good notation for representing the problem. The problem with the first, plain English, one is that it’s hard to manipulate. We want to be able to do algebra with it, and it’s hard to do algebra with plain English language. Modern mathematics owes a lot to Robert Recorde for giving us the equals sign (and popularising the plus and minus signs), because trying to manipulate sums with plain English language is an absolute pain.

The second one is bad for two main reasons:

It’s very clunky and unwieldy. If you’re going to be repeating the expression a lot, you don’t want to write that out over and over again. Also, are you sure you’re reading it correctly? Would you have noticed if I’d dropped 31 from the middle, or repeated 23 twice?
It doesn’t generalise well. Suppose we wanted instead to add up all the numbers from 1 to 50, or 1 to 1000, we’d have to write an entirely new expression for it. There’s no way to do algebra where the end point is variable with this expression.

So let me give you two other ways to write it:

\(1 + \ldots + 100\)
\(\sum\limits_{i = 1}^{100} i\)

In the first, the \(\ldots\) just means “and so on”. You’re expected to figure out what it means from context, but you’re also expected not to write it unless it’s obvious from context what you mean. If we’d wanted to make it a bit more obvious we could have written it \(1 + 2 + \ldots + 99 + 100\).

The second is a much more precise notation and looks completely inscrutable when you first encounter it, but has a fairly straightforward meaning: The \(\sum\) represents a sum of an expression, where the expression references a variable \(i\) that varies over some range. The \(i = 1\) at the bottom indicates that the sum is over a range of values of \(i\) starting with \(1\), and the \(100\) at the top indicates that \(100\) is the last element. The \(i\) to the right means that the expression we are summing is \(i\).

So, for example, \(\sum\limits_{i = 1}^3 i = 1 + 2 + 3\), or \(\sum\limits_{i = 2}^3 i^2 = 2^2 + 3^2\).

The \(\sum\) notation is very convenient because it allows us to compactly represent complicated expressions. It’s also useful for making generalisations. e.g. if we wanted to work out the answer to this for any number, we could write \(\sum\limits_{i=1}^n i\) to indicate that the end point is an arbitrary integer \(n\).¹

To start with though, we’ll keep working with the \(\ldots\) notation.

Now, let’s solve the problem.

We can write \(1 + \ldots + 100\) as \((1 + \ldots 50) + (51 + \ldots 100)\), because we can break up the sum however we like.²

We can also reverse a sum if we want, so we could write the second one equally well as \(100 + \ldots + 51\).

Now, there are exactly 50 elements in each sum, so we can pair them up and rearrange the sum as follows. \((1 + 100) + (2 + 99) + \ldots + (49 + 52) + (50 + 51)\). Now, notice that each of these terms adds up to exactly 101. You can check that \(1 + 100 = 101, 2 + 99 = 101\) etc but you can also this because each time you move right, you add one to the first term and subtract one from the second, so the value stays constant. This means that this sum is exactly 50 copies of 101, so the sum is \(50 \times 101 = 5050\).

I think a pretty reasonable question for you to ask at this point is how on earth you’re expected to come up with a trick like that. If you’ve seen this problem before it’s old hat, but if you had to come up with it on the spot, how would you do that?

The honest answer is I don’t know. A lot of things like this you just figure out by fiddling with the problem and getting an instinct for what works. This sort of approach is what I think of as a “trick”, and sometimes you spot the trick which lets you massively simplify the problem, and sometimes you don’t. As you do more mathematics, you develop both a familiar bag of tricks, and you learn to apply them in new circumstances. The trick here is something like “try grouping different parts of the sum together to see if they add up to something convenient”.

For example, here’s another way you could use a similar trick. Suppose I asked you to calculate \(1 - 2 + 3 - 4 + \ldots + 99 - 100\) (you could also write this as \(\sum\limits_{i = 1}^{100} (-1)^{i - 1} i\) ) - that is, it’s the same sum as before, but this time the signs alternate between positive and negative.

Well, here you could group it as follows: \((1 - 2) + (3 - 4) + \ldots + (99 - 100)\). Each of those expressions in brackets is \(-1\), because each time the number on the right is exactly one more than the number on the left. As before we have exactly 50 elements, so the result is \((-1) \times 50 = -50\).

This is basically the same trick, but with a different grouping.

Mathematics doesn’t necessarily rely on knowing and spotting things like this, but it’s a very common part of mathematical practice. Sometimes they’re essential and you can’t figure out any other way to solve the problem without some sort of trick like this, but often they’re just a way of bypassing some tedious calculation or another.

Now, one problem with the way we solved this is that it doesn’t generalise all that well. For example, suppose I asked you to add up all the numbers between \(1\) and \(101\), how would you do it?³

The previous trick doesn’t work because it relied on there being an even number of values in the sum. If we tried to do the same thing as before we’d have to break the sum up as something like \((1 + \ldots + 50) + 51 + (52 + \ldots 101)\), because there’s a number left over in the middle. We can then do the same pairing as before and get \((1 + 101) + (2 + 100) + \ldots + (50 + 52)\) for the paired sums, and the result is \(51 + 102 \times 50 = 51 + 5100 = 5151\). Which is, conveniently, \(5050 + 101\), so we’ve not messed up the calculation somewhere.

This works but is unsatisfying. In particular, let’s try to work it out for a general number \(n\). In order to do this as above, we’ll have to consider two special cases: First, let’s assume that \(n\) is an even number. Let’s say \(n = 2m\).

Then we can write \(1 + \ldots n = (1 + \ldots m) + ((m + 1) + \ldots n)\) as before, and use the same pairing trick to get \((1 + n) + (2 + (n - 1)) + \ldots (m + (m + 1))\). As before, each of these individual bracketed expressions are always equal to \(n + 1\), and we have \(m\) copies of the expression, so the answer is \(m (n + 1) = \frac{n (n + 1)}{2}\) (because \(m = \frac{n}{2}\)).

Now, suppose \(n\) is odd. We could repeat the pairing trick, but we can also use \(1 + \ldots + n = (1 + \ldots + (n - 1)) + n\), and if \(n\) is odd then \(n - 1\) is even, so we can use the previous expression for the sum when \(n\) is even, plugging in \(n - 1\) in place of the \(n\). This gives us \(\frac{(n - 1)(n + 1 - 1)}{2} = \frac{(n - 1)n}{2}\) for the sum up to \(n - 1\), and thus \(\frac{(n - 1)n}{2} + n\) for the whole sum.

Now, let’s make that expression a bit nicer. \(\frac{(n - 1)n}{2} + n = \frac{n^2 - n + 2n}{2} = \frac{n^2 + n}{2} = \frac{n(n + 1)}{2}\).

Something worth noting here is that although we had to handle the odd and even cases differently, we got the same expression out in both cases. This should make us a little suspicious that we’re missing something and that there’s a nicer proof here.

Another thing that should make us suspicious but that is actually fine is that the answer to this question should be an integer, as we’ve only added up integers, but we’ve got a fraction in the answer. It’s worth doing a sanity check that this is fine, but it is: Exactly one of \(n\) and \(n + 1\) must be even (because if you add one to an even number you get an odd number, and vice versa), so the expression on top is always even, and dividing it by two gets you an even number.⁴

Anyway, there’s another trick, and that fraction is a clue. We made this work by pairing up numbers together, and there’s a 2 in the end, so perhaps we would find it easier to work out twice the answer than we do to work out the answer directly.

We can write

\[\begin{align*} 2 \times (1 + \ldots + n) & = (1 + \ldots + n) + (1 + \ldots + n) \\ & = (1 + \ldots + n) + (n + \ldots + 1) \\ & = (n + 1) + \ldots + (1 + n) \\ \end{align*}\]

This gives us \(n\) copies of \(n + 1\), so twice the answer is \(n (n + 1))\) and the answer is \(\frac{n(n + 1)}{2}\) as desired. This is a much neater proof, but I think a slightly harder trick to spot.

I think this is a good time to write some Greek.

We can write \(\sum\limits_{i = 1}^n i = \sum\limits_{i = 1}^n (n + 1 - i)\). This is basically “running the sum backwards” like we did above. The first element is \(n + 1 - 1 = n\), then \(n + 1 - 2 = n - 1\), etc. all the way to the end where it’s \(n + 1 - n = 1\).

This means that \(2 \sum\limits_{i = 1}^n i = \sum\limits_{i = 1}^n i + \sum\limits_{i = 1}^n i = \sum\limits_{i = 1}^n i + \sum\limits_{i = 1}^n (n + 1 - i)\).

Now, because these sums are over the same range, we can group the expressions inside them together, so \(2 \sum\limits_{i = 1}^n i = \sum\limits_{i = 1}^n i + (n + 1 - i) = \sum\limits_{i = 1}^n (n + 1) = n (n + 1)\).

If you’re unused to dense mathematical notation this may not seem obviously clearer or better than the above, but it has the advantage of being compact and easy to manipulate, and it lets you result a lot of complex reasoning to fairly simple mechanical manipulation.

So now we’ve seen how to work out this answer. But why is it true?

This is another key feature of doing mathematics:⁵ Often you should be unsatisfied with mere answers, and looking at the problem from another angle will help you really understand it.

Is this particular example worth really understanding? Well… probably not. I think if you find yourself doing a lot of this sort of sums⁶, it’s worth playing around with a lot of simple examples like this in order to get a really good intuition for how everything fits together.

But right now it’s worth really understanding this because it’s a nice illustrative example of the general mathematical phenomenon.

Imagine, or draw⁷ the numbers as rows of blocks stacked on top of each other. On the bottom row you’ve got \(n\) numbers, then \(n - 1\), and so on, until you’ve got \(1\) block on the final row. The result is a sort of ragged triangle.

Now, take a copy of that, and flip it upside down, and stack it on top of the other one. The jagged edges will line up, and you’ll get a rectangle. How big is the rectangle? Well, it’s got \(n\) items on the base, because that was your bottom row, and it’s \(n + 1\) tall, because your triangle is \(n\) tall and you had to stack an extra row on top. So there are \(n(n + 1)\) blocks in the rectangle, of which exactly half must have come from our original triangle, so there were \(\frac{n(n + 1)}{2}\) blocks in the original triangle, and \(1 + \ldots + n = \frac{n(n + 1)}{2}\) as desired.

This is, in some sense, the same proof as we had before, but the geometrical representation of it seems helpful for people, and for more complicated examples it can be genuinely very useful.

Here is, I promise, one final⁸ way to do it, which is also often useful: We take a guess, and prove that that guess was correct. For example, suppose that we guessed the answer was \(\frac{n(n + 1)}{2}\). How might we prove that we were right?

The answer is to use something called “proof by induction”⁹

The way a proof by induction of some claim (your “induction property”) works is that you establish:

It’s true for \(1\) (or anywhere else you want to start - your “base case”)
If it’s true for \(n - 1\) then it’s true for \(n\)

And this is sufficient to prove that it’s true for every integer greater than or equal to your base case.

Why? Well, because of a proof by contradiction. A proof by contradiction is when you assume something that you’re not sure is true, work through the consequences, and conclude something false, which means your original assumption must be false.

To see that proof by induction works, assume the opposite: Suppose there were some \(n\) for which the induction property were false. Then there must be some smallest such \(n\) for which the property is false. If so, then the property must be true for \(n - 1\), as the property holds for \(1\), so we must have \(n > 1\) so \(n - 1\) is still an integer greater than or equal to \(1\), and it’s smaller than \(n\), which we picked to be the smallest integer for which the property is false. But the second part of our induction conditions was that it’s true for \(n - 1\) then it’s true for \(n\), which contradicts our claim that it was false for \(n\).

Anyway, proof by induction.

Our inductive property is that \(\sum\limits_{i = 1}^n i = \frac{n(n + 1)}{2}\). This is true for \(1\), you can just check that by calculating \(\frac{2}{2}\).

Suppose it’s true for \(n - 1\). That is, \(\sum\limits_{i = 1}^{n - 1} i = \frac{(n - 1)n}{2}\). Then¹⁰

\[ \begin{align*} \sum\limits_{i = 1}^n i & = n + \sum\limits_{i = 1}^{n - 1} i \\ & = n + \frac{n(n -1)}{2} \\ & = \frac{n(n + 1)}{2} \\ \end{align*} \]

So the property holds for \(n\) as well, and the result is proved by induction.

This does, of course, invite the question of how on earth we would have made this guess if we didn’t already know the answer.

I think there are roughly three ways that can come about.

The first is that you can just look at the first couple elements of the sequence and try to puzzle out a formula that works. \(1, 3, 6, 10, 15, \ldots\) seems plausible that there should be a factor of \(n\) in there, so what happens if we divide by \(n\)? We get \(1, \frac{3}{2}, 2, \frac{5}{2}, 3, \ldots\), so that looks like it’s going up by half each time, so \(\frac{n + 1}{2}\), and that gives us our guess.

The second is that sometimes you just have a hunch. I think it would be weird to have a hunch in this case and not be able to spot the better proof, but in other cases it will often be the case. This is particularly true when think it’s true because of some sort of “heuristic argument” that doesn’t actually work rigorously, but does give you a pretty good idea of what the answer should be.¹¹

The third common way is that you proved it another way first, but that proof was long and complicated or involved some advanced concepts you don’t want to introduce here, so you “guessed” in the sense that you already knew the right answer, but it was easier to prove it by induction than convince the reader of it another way.

OK, that’s enough adding numbers.

What should you take away from this post?

The first thing I’d like you to take away from it is a certain familiarity with the notation. Being able to read this sort of notation is entirely essential if you’re to read any mathematics at all. Mathematics is built on notation, and the notation I’ve used here is very common.

Another thing I’d like you to take away from this is that there are many ways to get the answer and they all get the same answer. There isn’t a single right way to do it, and there can’t be. Mathematics is a discipline built on solving problems, and if there were always a single correct way to do it, every problem would be trivial. Doing mathematics is about developing facility with mathematical problems, and getting a sense of how to solve them.

Another is that figuring out which mathematical tool to use is a skilled practice like any other - through repeated encounters with mathematics you develop intuitions about what works and what to try.

Reading other people’s mathematics is a good way to start with this but importantly the proof that people show you is often not the proof they used to come up with the idea. This means that often when reading someone’s proof you will have some reaction like “Oh, that’s clever, but I have no idea how you’d come up with that.” or “What the fuck is going on here? I have no intuition for any of this”. If you’re lucky you can just ask them how they came up with it, but if not then I think it’s worth trying to prove it yourself by figuring out how all the different pieces fit together and see how you’d do it for yourself.

One reason these multiple avenues of proof are important is that mathematics is not just a discipline of problem solving, it’s a discipline of concept formation, and one of the things that drives concepts is developing a certain aesthetic sense and desire for intuition around the problems you solve. Many concepts are discovered because you look at a proof and go “I don’t like that proof. Can I do better?”

There are some implicit notational conventions here by the way. \(i\) is usually an index variable. \(i\) and \(n\) are both expected to be integers by that choice. This isn’t required, but it’s conventional. However, you would only ever use integers as sum indexed or bounds like this, and that’s not conventional, it doesn’t really make sense to use non-integers here.
↩︎
Technically speaking we are using what’s called the associativity rule of addition here.
↩︎
Well you’d add 101 to 5050 and get 5151 of course, because you just worked out the sum from 1 to 100. But suppose you hadn’t done that.
↩︎
It’s very easy to make mistakes in mathematics, so it’s good to have a habit of doing little cross checks like this where you look at the answer you came up with and ask if it could possibly be correct.
↩︎
I wrote about this in algebra and insight but that example only works if you’re already a moderately advanced mathematician.
↩︎
Applications include “being a mathematician in the early to mid 20th century”, “taking a maths exam”, “for the sheer joy of it”, and “if you do a really large amount of mathematics every now and then you will run into a case where you really do have a use for this and you will be so delighted that you’ll ignore how rusty you’ve got at it”. Also probably combinatorics or something like that.
↩︎
I really should include a diagram here but I can’t be bothered, sorry. If you’re wondering how to draw \(n\) boxes, the answer is you don’t you draw some fixed small number and assume that it generalises. This is why I don’t like visual proofs that much.
↩︎
For the purpose of this post I mean. There are infinitely many ways to do this, and at least several of the other ones are interesting in their own right.
↩︎
Any philosophers reading this, don’t panic! It’s a completely different meaning of the word “induction” that has nothing to do with the philosophical notion of induction.
↩︎
This is in fact the same calculation we did in the even/odd case above.
↩︎
This is particularly common when you’re trying to show something a bit weaker than strict equality, such as e.g. that some quantity never gets too large.
↩︎

What is probabilistic programming?

2025-04-23

Friends and family often ask me what probabilistic programming is. This isn’t very surprising, as I currently work in probabilistic programming. Unfortunately I haven’t had a very good answer for them, as this is quite a recent development for me and I haven’t fully understood the boundaries of the field.

I think also it’s hard to answer because the field is a little confused about what it is, and there’s a “What the field says it is” answer and “What the field actually is” answer.

So I’m going to try to give my own answer of what I think the field is. It will be a bit overbroad and claim some territory that could equally be put in other fields, but I think the overbroad definition is more informative than the overly narrow one.

First, let me give you a narrow answer that I think people would classically give:¹ Probabilistic programming is the field of developing probabilistic programming languages.

What’s a probabilistic programming language? Well it’s a particular type of tool for doing the thing I’m about to describe in the general answer.

So, this is what I think: Probabilistic programming is the field of constructing random samplers with precise distributional properties, centrally but not exclusively² with the goal of developing better Monte Carlo methods for doing statistics.

I will now unpack all of those terms so that this definition actually makes sense.

A random sampler is a program that, when run, generates some value, typically at random. You could imagine e.g. a coin flip, which randomly generates either “heads” or “tails” as your prototypical random sampler. The work I did in Hypothesis is also an example of a random sampler, where it generates test cases for checking your program with by making a bunch of choices at random.

Random samplers have the virtue of being very easy to construct - you can just write a simple program that makes a bunch of random choices and bam you’ve got a random sampler.

However, it’s often hard to reason about the behaviour of random samplers, and in particular to construct random samplers that behave in particular ways.

For example, suppose you were creating a random sampler to simulate some physical process - say, the weather - it’s quite important that if about 10% of your random samples say it’s going to rain tomorrow, then you should expect that about 10% of the time tomorrow there will be rain.³

Having random samplers that have accurate distributional properties like this - e.g. that our random model of the weather accurately reflects what we should believe about the weather - is an essential property for doing monte carlo simulations, which just means that you run a bunch of random simulations and use those as a representative of the range for the true value you’re trying to calculate. e.g. if you want to know if it’s going to rain on Tuesday, run a few thousand weather simulations and see what fraction of them rain on Tuesday. If you want to know the total rainfall you should expect, average the total rainfall across all of your simulations, etc.

These are not the exact calculations that you might hope for in classical statistics - what you get is a random number, and if you reran the simulation you’d get a slightly different number, but it is a random number that you can count on being very close to the true number you want,⁴ and you can get it as close as you like by running more simulations.

Now, that’s all very well, but what this gives you is an accurate estimator of these values in your simulation. How do you get your simulation right?

This is where probabilistic programming and its toolkit for trying to control distributional properties of samplers comes in.

In particular it often comes in via trying to do Bayesian inference.

Bayesian inference works as follows:

You start with some very general model of the world (a “prior distribution”), and what data you would expect to see given specific configurations of that world.
You feed in a bunch of actual data.
You get an updated model of the world (a “posterior distribution”) out.

A great deal of probabilistic programming is concerned with how to do this sort of Bayesian updating on random samplers.

This allows for a very general approach to doing statistics:

You write a random sampler for a very broad world model.
You feed in a bunch of data and do a Bayesian update on your sampler to take into account the data.
You run Monte Carlo simulations and use them to make predictions about the world.

All of the hard part is in Step 2. This is an almost impossibly difficult problem to do in general. Many of the classic probabilistic programming tools try to do this, and what you get is tools that sometimes work very well and sometimes just run unusably slowly, with very little ability to predict which situation you’ll be in or much recourse for fixing it when it happens.

What the lab I’m in tries to do is to instead provide a toolkit that makes it easy to write your own algorithms for your specific problem, by providing an API for samples that you can perform a variety of manipulations on - either standard algorithms or ones of your own creation - that let you perform the update.

But, in general, doing this bit well is where most of the active work in probabilistic programming is, and why the tools have historically been a bit underused. We’re hopeful that a mix of new improvements in computing and new advances in probabilistic programming allow us to do a lot better than we’ve previously seen, but this is still an active research and development area, so we’ll see how that goes.

And the wikipedia page gives.
↩︎
That is, this is the origin of the field and the major driver of progress in it, but the techniques are more broadly useful and using them to do other things would still be considered probabilistic programming.
↩︎
What does that mean? Don’t worry about it too precisely. If you want to worry about it more precisely, read Probably enough probability for you and World counting as a tool for understanding probability
↩︎
Figuring out how close is part of the general body of theory needed and developed in doing probabilistic programming.
↩︎

Algebra and insight

2025-04-22

I was reading up on Markov Chains today, as one does, partially reminding myself about them and partly filling in some gaps in my knowledge.

One bit I was reading about was reversible distributions. If you’ve got some transition matrix \(P_{ij}\), a reversible distribution for it is a probability distribution \(q\) over the states such that \(q_i P_{ij} = q_j P_{ji}\). If you also lack any intuition about what that means, me too at the point at which it’s introduced. The broad intuition is that a Markov chain in a reversible distribution can be “run backwards in time” and you can’t tell the difference, but the main significance is that it’s an easy property to verify that has strong consequences.

Specifically, it’s a theorem that every reversible distribution is stationary. That is, if \(q\) is reversible then \(q_i = \sum\limits_j P_{ji} q_j\) for all \(i\) - the distribution is not changed by running the Markov chain forwards.

The proof of this presented in the text I was reading is as follows:

We can write \(q_i = q_i \sum\limits_j P_{ij}\) (because it’s a standard property of transition matrices that this sum is 1), so \(q_i = \sum\limits_j q_i P_{ij} = \sum\limits_j q_j P_{ji}\), so \(q\) is stationary.

I can follow this proof of course. It’s very simple algebra. But there’s something about it that feels like a magic trick. I don’t feel like I’ve got any actual insight into the nature of reversible distributions or have any sense of why they must be stationary from this. I’m sure there are plenty of mathematicians who could, and I’m not even sure that I need that insight, but it bugged me.

Thinking about it a bit more lead me to come up with the following alternative proof¹

One way to think about distributions on markov chains is that you’ve got a certain amount of “stuff” (probability mass) floating around, and at each step that you run the Markov chain, this probability mass flows around the graph.

When you’ve got a probability distribution \(q\), all of \(q_i\) flows out of node \(i\), and \(\sum\limits_j q_j P_{ji}\) flows into it. That is, for each node \(j\), the mass in it flows out to every other node, and the fraction of it that goes to \(i\) is \(P_{ji}\).

A distribution is stationary if for all \(i\), \(q_i = \sum\limits_j P_{ji} q_j\), but equally we can write this as \(q_i - \sum\limits_j P_{ji} q_j = 0\). The net flow of mass out of \(i\) is 0.

We can write this flow of mass out of \(q_i\) as \(q_i = \sum\limits_j q_i P_{ij}\) - the sum of the mass flowing to each \(j\). So the net flow of mass out of \(i\) is \(\sum\limits_j P_{ij} q_i - P_{ji} q_j\). That is, there is a net flow along each edge, let’s say \(f_{ij} = P_{ij} q_i - P_{ji} q_j\). The probability of \(i\) remains unchanged if \(\sum\limits_j f_{ij} = 0\).

However, the reversability condition is precisely that all of the \(f_{ij} = 0\). That is, there is no net flow of mass along each edge. As a result, a reversible distribution is necessarily stationary, because a much stronger condition applies: Not only is the total net flow across the edges from \(i\) equal to 0, it’s 0 along each edge.

This proof is in some sense much worse. It’s longer and involves more algebra rather than less. But after reading this proof I felt like I understood why reversibility obviously implied a distribution was stationary, and also why it was a much stronger condition than being stationary.

I think mathematics - and many other things - benefits hugely from this sort of chewing on a problem and seeing it from different angles until it actually makes sense. If you just know that a result is true, you often find yourself unable to really use it. If you understand at an intuitive level why it’s true, it will stick with you and you can really see how to apply it, including in cases where it doesn’t quite hold but something similar does. Proof without developing insight alongside it is certainly better than no proof, but it’s in some sense fundamentally fragile and untrustworthy, and it’s often better to mull it over until you find the right way to break up the problem to actually allow it to make sense at an emotional level.

Which I suspect I have cribbed in some sort of half-remembered manner from Geoffrey Grimmett’s “Probability on Graphs”, but if so it was useful to recreate it from memory anyway.
↩︎

Resuming habits

2025-04-21

I took a deliberate break from the daily posting habit over the easter weekend. I think this was the right choice.

But now, I’m faced with a perennial difficult problem: Once you’ve paused a habit, you’ve lost inertia, and now you have to deal with the difficult task of starting again from a cold start.

What I’ve often found in the past is that this causes you to delay, and then the habit develops an ugh field, and this makes it harder and harder to start, and suddenly months have passed without you having touched it.

I think the solution to this is that whenever resuming a habit, even after an incredibly short break but certainly after a long one, you should deliberately do so in the lowest effort way possible.

Like this.