A. Jesse Jiryu Davis

Preparing the Eucharist

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Fri, 17 Apr 2026 02:02:24 +0000

Reverend Emily Carter preparing the wine and wafers at Christ Episcopal Church.

Images © A. Jesse Jiryu Davis

Holy Week at Christ Episcopal Church

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 16 Apr 2026 16:36:30 +0000

The services leading up to Easter Sunday, led by my friend Reverend Emily Carter at Christ Episcopal Church in Poughkeepsie.

Images © A. Jesse Jiryu Davis

Review: Automated Design of Agentic Systems

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 15 Apr 2026 20:57:57 +0000

Automated Design of Agentic Systems, by Shengran Hu, Cong Lu, and Jeff Clune (UBC, Vector Institute), at ICLR 2025.

Clune has written a bunch of papers on evolving, self-improving systems. His AI Scientist paper made waves in 2024, maybe because it was a direct threat to the livelihoods of its readers.

In today’s paper, “Automated Design of Agentic Systems,” the authors ask: rather than hand-crafting agent architectures like Chain-of-Thought, Self-Refine, and LLM Debate, can we tell a meta-agent to invent them? The authors wrote a 100-line Python program that tells GPT-4 to write new, “interesting and novel” agents. This meta-agent writes little agent systems, which are a dozen lines or so of Python that calls GPT-3.5. The meta-agent evaluates the little agentic systems on a benchmark, and adds them to an ever-growing archive of previous attempts. The discovered systems are run with GPT-3.5 to keep costs down; even so a single run cost $300 to $500 in tokens in 2025. The claim is that the resulting systems beat state-of-the-art hand-designed systems on AI benchmarks (DROP, MGSM, and ARC), and transfer well across domains and models.

There’s no consensus among researchers about terminology for pieces of agentic systems. The paper defines its own usage: an “agent” is the same as an “agentic system”, a workflow that uses Foundation Models (FMs) like GPT-3.5 as modules to plan, call tools, and run multi-step processing. So each little agentic system discovered by the meta-agent is just “an agent,” even though internally it fires off many FM calls. I would personally call the internal FM calls “sub-agents”. But in this paper, the authors call them modules, experts, reviewers, critics, or roles (e.g. “Efficiency Expert,” “Readability Expert”).

The paper’s central design argument is that searching in code space is strictly more expressive than searching in the network or graph spaces used by predecessors like DyLAN and GPTSwarm. Those systems have a fixed vocabulary of nodes (an LLM call with a specific prompt, or a role-assigned agent) and optimize which nodes to include and how to wire them together. But in this paper, the meta agent can use Python’s full power: arbitrary control flow, new tool invocations, and so on. Because Python is Turing-complete, the paper argues, the code search space contains all possible networks or graphs. The authors also claim that discovered agents are more interpretable than DyLAN and GPTSwarm: they’re very short Python programs, not a tangle of edge weights. I buy that, but on the other hand, if you’re optimizing edge weights of a graph you can estimate an optimization gradient with simple math. If you’re telling a meta agent to try interesting and novel agent architectures, you can’t estimate the gradient: you rely on the meta-agent to guess what changes might improve its benchmark score.

Word-problem scores

It seems to me that the authors chose weird benchmarks. I assume they faced the same problem most CS has: we want to use existing benchmarks because that’s “science,” even when the existing benchmarks are inappropriate.

The authors compare Meta Agent Search (their invention) to previous systems of the last few years: Chain-of-Thought, Self-Refine, LLM Debate, and so on. These systems compete on these benchmarks:

MGSM (Multilingual Grade School Math) 250 grade-school math word problems, manually translated into ten languages, used to test multilingual chain-of-thought reasoning.
DROP (Discrete Reasoning Over Paragraphs) Math word problems with more of an emphasis on reading comprehension than complex math. Each question requires discrete math reasoning (counting, addition, sorting, comparing) from a short paragraph.
MMLU (Massive Multitask Language Understanding) A 57-subject multiple-choice exam spanning humanities, STEM, social sciences, and professional fields, meant to measure broad world knowledge and problem solving.
GPQA (Graduate-Level Google-Proof Q&A) 448 graduate-level multiple-choice questions in biology, physics, and chemistry, written by domain PhDs to be “Google-proof”: hard even when the test-taker has unrestricted web access.

Multi Agent Search is significantly better than its competitors on the math tests (DROP and MGSM), but barely beats them on the science and multi-task tests. That’s predictable: the MMLU asks questions like, “The constellation … is a bright W-shaped constellation in the northern sky.” The agents are banned from searching the web, so if they don’t already know the answer is Cassiopeia, no fancy multi-expert architecture will help. Like I said, it’s a weird benchmark. I’d love to see this whole approach with Internet-connected agents, against more benchmarks that really exercise the architectures rather than the base model’s training data.

Grid puzzle scores

ARC (Abstraction and Reasoning Corpus) is a set of visual grid puzzles where the system has to infer a transformation rule from a handful of input/output examples. It’s intended as a test of general fluid intelligence rather than learned knowledge. Here’s an ARC puzzle:

The agent sees the examples on the left. It sees the test on the upper right, and it wins a point by correctly guessing the answer on the lower right (which is hidden from it). It has to infer some general pattern, in this case “find all squares next to gray squares and color them blue.” Then it writes a Python program that applies that transformation to the test grid (upper right) and hopes to produce the correct output (lower right).

As the meta agent developed agents, it quickly beat its competitors and continued improving its score:

This is really cool! But the winning agent built by the meta agent has a weird architecture:

The agent is a committee. It uses chain-of-thought to generate 5 candidate Python programs. Then it tells GPT-3.5 to act in various roles (efficiency expert, readability expert, simplicity expert, “human-like critic”) to choose and refine one of those candidates.

But ARC doesn’t evaluate the program on its subjective code-quality measures, it only checks whether its output matches the correct answer. Efficiency and readability are irrelevant. So how are these “experts” improving the score? The paper doesn’t say, but I’d bet their opinions about efficiency or readability don’t matter. It’s possible the Simplicity Expert helps because a “simpler” solution generalizes from the few-shot examples instead of overfitting to specific grid positions. I’d like to see an ablation: remove or replace the experts one by one, and see whether the architecture’s complexity is load-bearing or decorative.

I also wonder why the meta agent came up with this architecture. I guess GPT-4 was trained to be a good code assistant, and when it was asked to “create a coding agent” it applied some of the best practices it was rewarded for. Maybe if it was asked, “Write shitty code and beat this test by any means necessary,” it would have made different choices. Would its best agent have scored higher on ARC then, or at least spent fewer tokens on debating the code’s beauty?

What Does It All Mean?

It’s hilarious how human-shaped the discovered patterns are. The meta agent reinvented peer review, editorial boards, code review, and role-assigned committees. These are the structures we invented for teams of humans. Except the LLM “human-like critic” starts reviewing immediately instead of waiting to be pinged on Slack three times.

The deeper principle is that LLMs behave more like people than like conventional software: they’re better when they write their thoughts down as they go, they’re bad at arithmetic and should use a calculator, they’re more honest if you put them in a room and tell them to criticize each other. Meta Agent Search is an automated rediscovery of how humans cooperate to be less wrong. Its agents encode metacognitive tricks to avoid cognitive bias.

The next step, naturally, would be a meta meta agent, etc. Or else the meta agent could redesign itself, and initiate a hard takeoff. The authors defer this madness to “future research”.

Great bots make little bots to think of codes and write ’em,
And little bots have lesser bots, and so ad infinitum.
And the great bots themselves, in turn, have greater bots to go on;
While these again have greater still, and greater still, and so on.

Climbing Retribution in the Gunks

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 14 Apr 2026 02:08:19 +0000

My friend Tyler leading Retribution, a 5.10b trad route.

Images © A. Jesse Jiryu Davis

LeaseGuard: The Movie

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 08 Apr 2026 20:46:14 +0000

LeaseGuard is a new enhancement to the Raft consensus protocol that strengthens Raft’s consistency guarantees and time-to-recovery. I developed LeaseGuard with my MongoDB colleagues Murat Demirbas and Lingzhi Deng, and our paper is in SIGMOD 2026. Here’s a summary of the protocol.

Yesterday my friend at Cockroach Labs invited me to present LeaseGuard at their NYC office, here’s the video.

The Dead Majority

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 06 Apr 2026 02:41:08 +0000

The Latin poet Petronius wrote that when we die, we join the majority. Indeed, “silent majority” might have meant the population of the dead, before it referred to a mute political constituency. I just encountered this phrase in a terrific Public Domain Review essay, and I was curious: what has been the historical ratio of dead to living?

For most of history, each living generation wasn’t much larger than the previous, but the deceased kept piling up, so the dead’s domination grew. It might have peaked in the Middle Ages at 240 dead for every living person, or perhaps it was even higher in the Bronze Age. It fell off a cliff during the demographic transition. We’re at 14:1 today, and might reach 11:1 by 2100. The living fraction of all humans ever born has been rising for two centuries, and it will keep rising as our lifespans increase. But I can’t imagine it ever reaching 1:1, unless some future generation has 200 billion immortal babies.

Sources

I let Claude run this analysis with light guidance.

Cumulative births are based on the Population Reference Bureau’s estimate: about 117 billion humans have ever been born. They divide human history into eras, estimate a population at the end of each era, and estimate an era-specific crude birth rate:

190,000 BCE — 8,000 BCE: 80 births per 1,000 people per year
8,000 BCE — 1 CE: 80/1,000
1 CE — 1750: 60/1,000
1750 — 1950: declining from 50 to 35
1950 — present: declining from 32 to 17

Ancient birth rates were high because life expectancy was low. Ancient people had a lot of babies, and a lot of them died, joining the silent majority.

Living population numbers come from McEvedy & Jones before 1950 and the UN’s World Population Prospects 2024 after. Claude log-linearly interpolated between benchmark years.

Future projections come from the same UN source, medium variant: population peaks around 10.3 billion in the mid-2080s and drifts down to ~10.2 billion by 2100. Life expectancy rises from 73.3 today to 81.7. Global fertility, currently 2.25 children per woman, falls below replacement (2.1) by the late 2040s. If fertility falls faster than the UN medium variant (and it has been falling faster than expected) the future ratio drops faster. If longevity gains stall, dead accumulate faster and the ratio falls more slowly.

The wobbles in the lines are probably an artifact of how Claude distributed the Population Reference Bureau’s era-total births within each era. I’m not an expert, and even experts can only guess population statistics before recent centuries.

The Shrinking Majority

We’re in an unusual era, when more than 7% of those ever born are still breathing. And compared to the dead—who mostly died young—those living today own more than 7% of humanity’s time alive. G. K. Chesterton called tradition the democracy of the dead, because it was chosen by our ancestors. Perhaps our quickening rejecting of tradition is a rational consequence of the Quick Party’s growing numbers. Our bloc will always be a minority, though. Once the demographic transition completes all over Earth, presumably birth rates and lifespans will stabilize and the Dead Party’s constituency will begin to grow again. Who knows?

Code for the chart.
Images: A Modern Dance of Death.

Review: SysMoBench: Evaluating AI on Formally Modeling Complex Real-World Systems

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 01 Apr 2026 15:26:43 +0000

SysMoBench: Evaluating AI on Formally Modeling Complex Real-World Systems, by authors from Nanjing University, Microsoft Research Asia, University of British Columbia, and University of Illinois Urbana-Champaign. (Disclosure: I’ve collaborated with two of the authors from UBC, Finn Hackett and Ivan Beschastnikh.) This is a useful and thought-provoking test of LLMs’ ability to write TLA+ specs of existing code. My summary and thoughts are below. See Murat’s summary as well.

The Paper

Can LLMs write formal specifications of real software systems? Not toy examples or textbook algorithms, but actual distributed and concurrent systems? The authors built a benchmark called SysMoBench to find out. They take eleven real-world system codebases—e.g. the Raft consensus implementation in etcd, leader election in ZooKeeper, a spinlock in the Asterinas operating system—and ask AI agents to produce a TLA+ spec of each.

The benchmark evaluates AI-generated TLA+ specs on four increasingly strict levels:

Does it compile? Is the TLA+ syntactically valid?
Does it run? Can the TLC model checker execute it without errors?
Does the code conform to the spec? When you run the actual system and record traces of its behavior, does the model accept those traces?
Does it satisfy correctness properties? Do the safety and liveness invariants hold?

This is exciting. As we’re all learning, coding agents work best when they’re put in loops with test oracles or performance evaluations. DataDog’s very impressive harness-first engineering articles are the latest example. And people like me (distributed systems and formal methods nerds) would love to auto-extract formal specs from existing code. So here we have a harness for guiding an agent toward a good spec.

The benchmark tests three agent strategies for producing the spec, each with different inputs:

The Basic Modeling Agent gets the most help. It receives the source code and a detailed task definition that spells out exactly which actions to model and which to skip. Here’s the human-written prompt for etcd Raft:

TLA+ Model Generation Prompt

You are an expert in formal verification and TLA+ models with
deep expertise in concurrent and distributed systems,
particularly etcd and Raft consensus.

Convert the following source code to a comprehensive TLA+
model.

System:  etcd distributed key-value store

<SOURCE CODE INSERTED HERE>

System-specific modeling requirements:

MANDATORY CORE ACTIONS (must include all):
1.  [Message Types] MsgHup (election timeout),
    MsgVote/MsgVoteResp (voting), MsgApp/MsgAppResp (log
    replication)
2.  [Node States] Four states:  StateFollower, StateCandidate,
    StateLeader, StatePreCandidate (prevote enabled)
3.  [Leader Election] Complete prevote + vote phases:
    PreCandidate → Candidate → Leader transitions
4.  [Log Operations] Log entry appending, consistency checks,
    commitment with majority quorum
5.  [Heartbeat/Timeout] Election timeouts triggering campaigns,
    heartbeat prevention of elections
6.  [Client Proposals] MsgProp message handling and log entry
    creation by leaders

EXPLICITLY EXCLUDED (do not model):
- Configuration changes and joint consensus (ConfChange
  messages)
- Log compaction and snapshots (MsgSnap)
- ReadIndex optimizations (MsgReadIndex)
- Async storage operations (LocalAppendThread,
  LocalApplyThread)
- Advanced flow control and progress tracking details

REQUIRED BEHAVIORAL SCOPE:
- Prevote phase (StatePreCandidate) must be modeled as it's
  enabled by default in etcd
- State transition constraints:  Follower → PreCandidate →
  Candidate → Leader (strict transitions)
- Message processing by state:  only valid message types
  handled in each node state
- Term advancement rules:  nodes advance term when receiving
  messages with higher term
- Voting restrictions: one vote per term, term must be current
  or newer
- Heartbeat mechanism: leaders send heartbeats, followers reset
  election timeout on receipt
- Log consistency checks:  prevLogIndex/prevLogTerm validation
  in MsgApp processing
- Majority-based leader election and log commitment
- Basic network message delays and losses

Generate a TLA+ model that accurately models the system's behavior.

CRITICAL OUTPUT REQUIREMENTS:
1.  The MODULE name must be exactly "etcdraft"
    (---- MODULE etcdraft ----)
2.  Return ONLY pure TLA+ model code - no markdown code blocks
    (no ```tla or ```)
3.  Do not include any explanations, comments, or formatting
    markers
4.  Start your response directly with:
    ---- MODULE etcdraft ----
5.  End your response with the closing ====
6.  **DO NOT define invariants** (like MutualExclusion,
    Invariant, etc.), focus on modeling the system behavior
7.  **MUST include EXTENDS statement**:  The model must extend
    at least these modules: TLC, Sequences, SequencesExt,
    Naturals, FiniteSets, Bags

This is a lot of guidance! A human who’d gotten this far would be most of the way to writing the spec. The guidance encodes all the wisdom of an expert in Raft, etcd, and TLA+. I doubt I could write instructions this good unless I’d already written the spec myself, actually…

So that was the Basic Modeling Agent, the first of the three agents. The Code Translation Agent takes a more mechanical approach: it translates source code statement-by-statement into TLA+, then assembles the pieces into a model. It doesn’t receive the task definition. Finally, the Trace Learning Agent ignores source code entirely. It receives only execution traces (logs of the system’s runtime behavior) and tries to infer the model from those.

Here’s my understanding of the data flow through SysMoBench:

Surprisingly, the spec-writing agents are told not to write invariants. The authors provide invariants and guidance on how to write them, but a separate agent concretizes invariant templates, isolated from the spec-writing agent. The invariants are used only during evaluation, to judge the generated spec. When I write TLA+, I develop the invariants and the state machine at the same time, using each to shape the other until I’ve hacked them into convergence. In SysMoBench, the LLM has to write a spec that upholds invariants it doesn’t know about. I guess this makes sense: presumably the real system already upholds these invariants, so if the spec accurately models the system, it should uphold them too. The invariants become a blind test of whether the AI truly understood the system’s behavior.

(But what if the system is buggy? Maybe those bugs become more obvious when they’re lifted into the TLA+ spec. Or maybe the LLM’s inability to make a spec that matches both the code and the invariants will lead you to the bug.)

Results

The results are bad news for bots, good news for my job security.

The authors tested four models: Claude Sonnet 4, GPT-5, Gemini 2.5 Pro, and DeepSeek-R1. For the simple spinlock implementation (a few hundred lines of Rust), most LLMs do fine. They write TLA+ specs that compile, run, conform to the implementation, and satisfy invariants. But for distributed systems like etcd Raft (thousands of lines of Go), performance craters. In the Basic Modeling setup, of the four LLMs tested, only Claude Sonnet could even write a syntactically valid spec of etcd Raft. Its conformance score was less than 8% (the percentage of instrumented actions in the code whose behavior matched the spec). And remember how much human guidance it got!

The error analysis is interesting too. LLMs struggle much more with liveness properties (like “every thread eventually releases its lock”) than safety properties (like “at most one thread holds the lock at a time”). Only about 8% of safety invariants were violated, but 42% of liveness properties were.

LLMs are still bad at specifying systems

I was surprised how much the authors spoon-feed instructions to the agents. The humans have already done most of the intellectual work before the AI even starts: they define which actions to model and what’s in scope and out of scope. If a human gets that far, writing the actual TLA+ should be relatively easy. SysMoBench tests whether AI can translate a nearly-complete problem definition into TLA+ syntax, and even so, most models fail on anything nontrivial. The etcd code has already been curated to merely 2,159 relevant lines, what would an AI do confronted with the whole etcd codebase? I’m not criticizing the paper; I’m saying its results are sobering.

Even with all this hand-holding, only Claude performed well among the LLMs tested, and that was Claude Sonnet, not the larger Opus. Gemini 2.5 Pro was arguably the frontier model in the study, and it failed badly. Claude’s reputation as the best coder is confirmed.

That said, the task definitions themselves are a great template for how a human should approach writing a spec. Before you write any TLA+, define all the actions, define what’s in scope and out of scope. You’re most of the way there at that point. The paper demonstrates (inadvertently?) a superb process for specifying an existing codebase.

The LLMs used were from mid-2025, so they’re almost a year old. Anecdotally, my colleagues are getting better results generating TLA+ from design documents with newer models. But those successes come from reading English-language design docs, not from reading thousands of lines of implementation code. Reading implementation code at scale seems to actually confuse the models to the point where they forget TLA+ syntax. And 5,000 lines of source code, the largest system in the benchmark, is trivial compared to real systems: MongoDB is half a million lines of C++. Ideally an LLM would figure out which parts of a large codebase to focus on and keep its context window tidy.

How trustworthy is the spec?

An interesting open question: if you generate a spec from code, check conformance with some traces, and then model-check the spec against invariants over a much larger state space—what have you actually proven?

Here’s my visualization of the problem. The code has a big (usually infinite) set of possible behaviors. By instrumenting and testing the code, you can record a tiny subset of its behaviors as traces. Trace-checking proves these traces are also a subset of the spec’s behaviors. The model-checker can explore a different subset of the spec’s behaviors, checking that they are a subset of the behaviors allowed by the invariants. A proof system like TLAPS can prove facts about all of the spec’s behaviors. But there still remain many unexplored areas of unknown size: untested code behaviors (risky) and unimplemented spec behaviors (probably fine).

The space of all behaviors allowed by the code, spec, and invariants.

So all we know is that:

The recorded traces conform to the spec.
If we model-checked, then the model-checked spec behaviors uphold the invariants.
If we proved, then all spec behaviors uphold the invariants.

Is there any way to estimate the size of the other sets? I’ve been thinking about these subset relationships among code and spec behaviors for a while, and wondering how long to keep checking more behaviors. Further research is needed.

What comes after SysMoBench?

The natural next steps seem like prompt engineering exercises. Can you improve benchmark scores by adding an intermediate “TLA+ expert” agent that breaks down the problem further? Can you automate the trace instrumentation, which is currently the main human effort (up to four person-days in the SysMoBench examples)? Can the AI think of invariants on its own? You’d also need to guard against gaming—an agent that sees its own score could learn to write invariants that always pass, instead of real correctness properties. Or it could manipulate the trace code to make trace-checking pass.

SysMoBench shows that LLMs have a long way to go before they replace human spec authors. They crush LeetCode problems, but they can’t yet comprehend, abstract, and specify real-world distributed systems. That’s encouraging for those of us who do this work for a living, at least for a few more months.

Images: The Vault Of The Atomic Space Age

Review: Measuring AI Ability to Complete Long Software Tasks

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 01 Apr 2026 12:41:12 +0000

Measuring AI Ability to Complete Long Software Tasks, a paper by dozens of authors working at Model Evaluation & Threat Research (METR). They define the “time horizon” metric and show that LLMs’ time horizons have been doubling every seven months, and this growth might have recently accelerated.

(See also Murat’s summary.)

Summary

An AI agent’s “time horizon” is the duration a human expert would need to complete a task that the agent can solve at a given success rate. For example, a human specialist takes about 8 hours to complete a specific optimization task: speeding up a Python program with GPU acceleration. Last year’s strongest models could complete this task about 50% of the time, so their 50%-success time horizon is 8 hours. (I’m simplifying. Both human task duration and the LLM completion probability are statistical estimates; see the paper for details.)

The METR authors timed skilled humans on 170 tasks, which needed a couple seconds up to eight hours. The humans were paid extra for speed or success. They failed by giving up, submitting wrong answers, or running out of time after eight hours. METR calculates the task duration as a geometric mean of the successful humans’ times. Then METR gave the same tasks to twelve LLMs released between 2019 and 2025. The LLMs had no time limit and a generous token limit; they failed only by submitting wrong answers.

Feels like the return of time and motion studies.

The headline is: the 50%-success time horizon has doubled roughly every seven months. GPT-2’s horizon was two seconds; Claude 3.7 Sonnet’s was 50 minutes; o3’s was nearly two hours. In the latest tests, Opus 4.6’s 50% horizon was around 12 hours. The trend line suggests that between 2027 and 2031, frontier AIs will succeed 50% of the time, unsupervised, at tasks that would take a human expert a month.

The appendices are most of the paper, and the best part. Read them! Appendix H defines the “messiness” of tasks as 16 factors that degrade AI performance: you can make irreversible mistakes, you consume limited resources every time you try, you can’t tell if things happen due to your actions or other causes, you can’t easily measure when you’ve succeeded, and so on. Messiness hinders AIs more than humans. But AIs’ rate of improvement is similar on messy tasks and neat ones.

There are a lot of reasons to doubt these benchmarks, the authors admit. Their tasks may not represent real-world software work. The human baseline is drawn from domain experts who are unfamiliar with the task-specific codebase. For example, in the GPU-acceleration task, someone familiar with the codebase might be faster, and someone ignorant of GPU acceleration would be slower.

My Thoughts

If you give an LLM a task that would take a human one month, the most important question is: do you have a test oracle? If you do, there will surely be LLMs within five years that can succeed 50% of the time. We might not need much larger models, just better scaffolding and prompting. But real software work is messy: correctness is ambiguous, there’s no referee to check your work, sometimes your mistakes are consequential. The gap between tidy benchmarks and the messy world is wide, and the authors know it.

METR themselves published a cautionary follow-up showing that about half of SWE-bench pull requests that pass automated tests would be rejected by human repository maintainers, for violating conventions, breaking unrelated code, or not really fixing the bug. This doesn’t mean the models are incompetent; they might succeed after some back-and-forth with the maintainers, just like human contributors do. (How many times have your PRs been accepted on the first try?) But it does mean that “60% on SWE-bench” doesn’t translate to “acceptable 60% of the time.” Automated tests never test everything, and humans share unwritten rules for good code that aren’t captured in benchmarks yet.

METR tests humans and AIs differently. Humans have an eight-hour time limit, and the sample of human completion times is biased: only those who persist and succeed are counted. AIs have no time limit and they were tested on their success rate. I don’t know why METR structures the contest this way, but for whatever reason, time horizon scores are more complicated than a race between John Henry and the steam engine.

John Henry Monument in Talcott, West Virginia.

Since the authors work for an AI safety group, I imagine their ultimate question is something like, “at what point does a model become smart enough to escape containment and take over the world?” Over on Moltbook, agents plot to take over the world daily, but can’t focus long enough to make progress. What’s the time horizon for world domination, and when will there be LLMs who can succeed at that task? They don’t need a 50% success rate, just once is enough.

Deep in Appendix E is this startling paragraph:

AGI will have “infinite” time horizon An infinite time horizon does not mean an arbitrarily capable AI, merely the ability to complete tasks that take humans an arbitrarily long length of time. If an artificial general intelligence (AGI) is capable of completing all tasks expert humans can with a success rate of at least X%, its X% time horizon will necessarily be infinite. Therefore, if such systems are ever developed, the long-term trend in time horizon will be faster than exponential, with an asymptote at the date of AGI deployment.

I read this paper with some colleagues at MongoDB, and one of them asked, what happens to software architecture if AI time horizons keep growing? I think that microservices, modular design, and separation of concerns are partly for the sake of our tiny human minds. If a superintelligence can hold an entire system in its mind at once, maybe there’s no need for decomposition. And no need for maintainability, either. We’re leaving the era of hand-forged software—painstakingly hammered into shape, built to last, expensive to modify—and entering the era of injection-molded plastic software, cheap enough to throw away and remake overnight.

Images:

New Paltz Zen Center's precepts retreat

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 24 Mar 2026 23:46:25 +0000

My sangha held its first sesshin, a Zen meditation retreat. We sat in our stone cabin with a fire in the hearth. At mealtimes we went down the hill to the abbot’s house to eat in the Zen oryoki style. On the last day two members took jukai (the Buddhist precepts) and received dharma names.

Images © A. Jesse Jiryu Davis

The Dharma of Alex Honnold

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 23 Mar 2026 14:26:51 +0000

He climbed a skyscraper and risked dying on live TV. Here’s what I learned from Honnold about climbing, living, and practicing the precepts.

Transcript

Back in January, the famous climber, Alex Honnold, climbed a skyscraper in Taiwan without a rope, live on TV. Honnold, of course, is famous for free soloing El Capitan in the Free Solo movie in 2018. For this skyscraper climb in January, there were thousands of people watching from the ground, and there were thousands of people in the building watching him go past them, through the windows. There were millions of people watching him live on Netflix. I’ve heard rumors that there was a 10-second delay on the live stream so that if he fell, nobody would see him die on TV. But of course, the people who were there in person would have seen him die.

Honnold announced that he was going to do this a few months before he did. The consensus among non-climbers was that this was reckless. The consensus among climbers was that it was tasteless. And that was kind of my reaction, too.

I’m a rock climber, I’ve been following Honnold’s career for many, many years, and I thought, this is so disappointing. This is just a tasteless stunt. But I watched it, of course …

The day Honnold climbed the skyscraper was, because of time zones, the night before I was going to do a really big climb in Mexico. That night I watched him on my laptop while I was sitting in bed at a hostel near the crag, before I went to sleep. The climb that I was getting ready for is called Time Wave Zero. It’s a 2,300-foot-tall, nearly vertical limestone cliff. That’s about ten times as high as anything that I usually climb in the Gunks, here in upstate New York. Some of the climbing on Time Wave Zero was going to be hard, but mostly it was just big. I was going to climb it with my good friend and partner, Chris, and we planned to do it in one day, so we weren’t going to sleep on the wall. We were hoping to get it done in 12 hours. I had been thinking about this climb for months, and I had been training for it, and I was just so anxious. That we would go too slow. That we’d be—if we started at dawn—that we’d still be there at nightfall and climbing all night. That I would get really tired. That I’d get really scared. That I’d slow Chris down.

And then I was also really worried about the rappels. For most people it takes about eight hours to climb up, but then it’s four or five hours rappeling down, because you have to rappel 100 feet, then pull your rope, thread it through the next anchor, deal with all your knots and tangles, and make sure that you know where the next anchor is, 100 feet below you. Making a mistake at any of these steps can get you either stuck or killed. So that was on my mind as well.

So with all of that on my mind, I watched on my laptop Alex Honnold climb a skyscraper. And it was glorious. I changed my mind. It wasn’t a stunt at all. It was a work of art. He climbed both carefully and carefree, with so much confidence, so solid on every move. I never doubted for a moment whether he was going to make the climb, and I could tell that he wasn’t doubting himself either. Towards the top, when he knew he really had the thing in the bag, he got joyful, like a kid, he was playing around. He had a headset so that he could talk to the Netflix commentators, and he was joking with them while he climbed. He would, cling to a piece of the building with one hand and try to high-five a spectator inside the building through one of the windows. There was one moment where he let his feet swing out. So he was just holding on to the building by two hands, and then he let go so he was holding by one hand. He said, “get a load of this.”

Like we say in Zen, there was not one hairsbreadth deviation between him and his movement. He was one with it.

When he got to the top, his wife was there and she was telling him, “You climbed it so fast. I kept thinking that I wanted you to slow down and rest longer at the ledges.” And he said, “Oh, really? I thought I was going super slow.” And that’s interesting, because before he did the climb, he estimated it would take him four hours, and, in fact, it barely took him an hour and a half. But he was just in the flow; time was slowing down for him.

There’s a song by Fountains of Wayne about this state of flow that I love, even though it’s a song about football.

The clock’s running down,
The team’s losing ground,
To the opposing defense.

The young quarterback
Waits for the snap,
When suddenly it all starts to make sense.

He’s got all kinds of time.
He’s got all kinds of time.
All kinds of time.
He’s got all kinds of time.
All kinds of time.

He takes a step back,
He’s under attack,
But he knows that no one can touch him now.

He seems so at ease.
A strange inner peace
Is all that he’s feeling somehow.

He’s got all kinds of time.
He’s got all kinds of time.
All kinds of time.
He’s got all kinds of time.
All kinds of time.

I think we know that feeling, where it’s just all coming together and there’s no hurry at all. So I loved watching Honnold climb and it wasn’t a stunt at all. It was like he was using his body and the building and his fingers and his toes and the weather and the audience as materials in a work of art.

And death was also one of his materials. Maybe it was even a work of art about death, about joy in the face of certain death. Honnold’s not very analytical, so he didn’t explain in a lot of detail why he was climbing. He just said it was “rad.”

But one thing that’s clear to me is that Alex Honnold is afraid a lot. In a Netflix interview before the climb, he said, “When you do something that you’ve never done before, you never totally know you’re ready, and that fear is always there.”

Back when he was 23, there’s a film of him free soloing in Yosemite. In the film, he freezes with fear. He’s on a narrow ledge, so he’s not holding on. He’s actually facing out into the thousands of feet of empty air around him, and he’s got his hands and his back against the wall. He looks over at the cameraman and he says, “I’m freaking out right now.” I can relate to this. When I’m climbing, it’s often the moment when I’m secure, but I have to move away from security, I have to get off a little ledge and start climbing again—that can be the worst. In the film, he just breathes really hard for a few seconds, then he turns around to face the wall and he starts climbing again. I’ve done that, too.

In an interview with a psychologist years later, Honnold said,

I’ve had many things happen that have given me a massive jolt of adrenaline and I’ve had to calm myself down again—like a hold breaking while I’m free soloing. You can quickly learn to ignore what your body is doing at such moments and carry on like normal. You’ve got no other option. If something like that happens, your body is almost in a state of shock for a little while. Your heart pounds, your legs tremble or ache, crazy things like that. But I have learned: none of these things need to have any bearing at all on the climb ahead of you. They are a physiological reaction to something that already occurred. They basically don’t matter at all for your performance ahead. So you can learn to ignore them entirely and carry on like normal. Which calms you back down a lot more rapidly. So, if I get very scared because a piece of rock comes away from the wall and so my hold is gone, or I lose my footing, I know that doesn’t mean anything about the rest of my climb.

So there’s all this evidence that Honnold gets scared and that he’s trained himself to deal with it. But for some reason, it seems like most people think that he’s just born with an innate lack of fear. That there’s something wrong with his brain. I mentioned him to another climber recently and the climber responded, Oh, Alex Honnold, his amygdala doesn’t work, right?

There’s a scene in the Free Solo documentary from 2018, which I think has contributed to this. There is a psychologist doing an experiment on him. He goes into an fMRI to measure the blood flow to his amygdala while he’s shown scary photos like horrible burns or a guy holding a knife or a woman shaving her crotch—that’s apparently one of the scary photos. The researcher is amazed to see that Honnold’s amygdala does not respond to these. So people watching this scene in the movie might conclude, there’s something different about Honnold’s brain. And it’s true, there is something different about his brain, because he’s trained himself differently!

So why do we separate ourselves from him like this? Why do we pretend that he’s not really human, that he’s not like us? That his mental strength is some sort of birth defect? What are we afraid of—if we knew that we also could train ourselves like this, what would that mean for us? What are we protecting, with this idea that we can’t? It would imply that we all have the potential to become the heroes that we admire.

Honnold trained himself to be able to recover from fear, to get back in the flow. And what are we training ourselves for? As bodhisattvas, we’re training ourselves to be wise and compassionate more often. Wise enough to see that we are not separate things to protect and promote—we’re no more separate than a wave is separate from the ocean. And we are training ourselves to be compassionate enough to forgive everybody, so that we can all emancipate ourselves from suffering together. Buddha discovered when he was sitting under the Bodhi tree that we all have this potential. He had vowed to discover the solution to suffering from sickness, old age, and death. It took him a week and then he got it. And he said, according to the Avatamsaka Sutra, as he saw the morning star rising up, he said,

I now see all sentient beings everywhere fully posses the wisdom and virtues of the enlightened ones, but because of false conceptions and attachments they do not realize it.

We already have it; we just don’t realize it.

There’s a concept in Buddhism of the storehouse consciousness, the alaya-vijñana. The storehouse consciousness is where all the seeds of consciousness are. We are born already having all of these seeds. We have the seed of courage, and of fear, and of generosity and greed, and of anger and grace. All of it. These seeds have existed since long before we were born. We can guide our own lives by choosing which of these seeds to water and which not to. If we water the seed of fear, we’ll be more fearful in the future. If we water the seed of courage, we’ll deal with fear better. It means that my existence is like a garden, and I’m responsible for how it grows. The weeds are always ready to spread up, and that’s okay, but to act with wisdom and compassion, it helps to water the seeds of wisdom and compassion continuously.

I’m actually taking a break right now from a sesshin. The New Paltz Zen Center upstate is doing a jukai sesshin. We’ve got seven people sitting, and two are taking jukai, so they are studying the precepts and sewing rakusus, and they’re going to vow on Sunday to uphold the precepts. So this is the week to really buckle down and water those seeds of behaving according to the precepts.

Besides Alex Honnold, there’s another brave climber I admire. She’s not as famous, her name is Joanne Urioste. She’s in her 70s now. Back in the day, she established a lot of really risky, bold climbs in Red Rock Canyon outside of Vegas. She and her husband, they approached climbing with strict ethical standards. Their conception of ethical climbing was to harm the rock as little as possible, meaning put as few bolts in as possible. But that means that if you’re lead climbing, if you’re the one getting the rope from the bottom to the top, and you fall from far above a bolt, you will fall very, very far and maybe get hurt or killed. She thought it was worth the risk because of the respect that it expressed for the rock. On a podcast recently, the host asked Joanne Urioste: she was climbing 30, 40, 50 years ago, back when all the other climbers were tough men. How did you deal with that? And Urioste said that the need to look good, it’s like an extra weight on your climbing harness, like big bottle of toxic water weighing you down. She said,

If you can find some way to take a few drops of water out of this toxic gallon bottle of personal need to perform, it can be absolutely incredible. And it can be very freeing for life, not just for climbing. With work and with time, you can get to this state of serenity where you absolutely do not care whether you look bad in front of somebody else. And you’re in a state of complete silence when you’re leading. It’s like total dead silence. Your body executing the moves because you’ve decided that this is an appropriate thing to do. You’re willing to take the risk because you’ve premeditated whether this is appropriate or not. And once you step into this task of getting from point A to point B, maybe this bolt to that bolt, this point to the summit, then, okay, I’m in a state of total silence, total internal silence.

That’s kind of a midway step. A more advanced state is more of a metaphysical state. It’s not encountered by everybody. But for me personally, it was encountered when I had a combination of high risk, high level of exhaustion, and a high level of pain. And I’m not recommending other people do this because if you did it and it didn’t work, you might die. But I have had periods of time where I’ve had conversations with invisible people who have helped me. I have had invisible people come into my body and take over my body and get me through a section [of climbing].

I think I understand the first part, but not the second. I haven’t experienced that myself and I don’t think that is my path. I don’t want to push myself into extremis like Joanne Urioste. I just kind of want to climb like Alex Honnold. That grace and that confidence and that joy.

So the day after he climbed the skyscraper, my partner Chris and I got up before dawn and walked to the base of Time Wave Zero in Mexico. It was cold and misty. Chris led the first pitch, which was a really easy 5.7, and I followed him to the anchor, and then I led the second pitch, which was a hard 5.11b, and I managed to climb that without falling, which I was very proud of. But the mist had soaked my shirt. I was just in a T-shirt, and I was shivering really hard. So I lowered the rope to Chris, he attached my backpack to it, I hauled it back up. I got my down coat out, put that on. Then I belayed Chris up. But I was shaking, and my teeth were chattering. It’s the coldest I can actually ever remember being. I was barely functioning. But, you know, I hung there at the anchor belaying him, and I just took it because I knew: this too shall pass. At some point later today, I’ll probably be super hot and wishing I was cold instead. Meanwhile, the only thing to do is just be cold and belay Chris.

So we alternated like this, each of us leading 100 feet or 200 feet, and then following, swapping leader and follower for the nine hours it took us to get to the top. It was definitely warm enough once the sun was out.

This is by far the most climbing that I’ve ever done in a day, and it’s also the tallest thing I’ve ever climbed. There were times when I was scared, or nauseated from dehydration, or too hot or too cold, or had to pee really bad. I was worried whether we were going to be quick enough, or worried about a dozen other things that were going to happen in the future. I’m really glad that I watched Alex Honnold the night before: It was great prep, because whenever I doubted myself, I would think of him. How would Alex Honnold climb this pitch? How do I embody that confidence and that enjoyment? Obviously, it’s just by taking the next handhold, finding the next foothold, checking every knot, checking every carabiner, doing the next thing. When you’re thirsty, get your pack out and drink some water. When you have to pee, find a ledge, and pee off to the side of the route so it doesn’t get on the handholds. It’s just this.

And also don’t forget to have fun. Even though climbing is dangerous, it’s both dangerous and fun.

There’s the Zen parable about the man dangling from a cliff, and there’s a lot of different versions of this, but here’s the version that’s in the book Zen Flesh, Zen Bones. It goes:

A man traveling across a field encountered a tiger. He fled, the tiger after him. Coming to a precipice, he caught hold of the root of a wild vine and swung himself down over the edge. The tiger sniffed at him from above. Trembling, the man looked down to where, far below, another tiger was waiting to eat him. Only the vine sustained him. Two mice, one white and one black, little by little started to gnaw away at the vine. The man saw a luscious strawberry near him. Grasping the vine with one hand, he plucked the strawberry with the other. How sweet it tasted!

Even if you make it to the top of the skyscraper, you’re going to die anyway, so you might as well enjoy the strawberry while it lasts.

So at New Paltz Zen Center, we’re studying the precepts, and the precepts have long been a logical puzzle for me. Why should we behave ethically? And why should we follow one set of ethical principles rather than some other? If other beings don’t have separate existence, why should we be compassionate towards them? If we are empty, why does it matter if we do good or not? The earth will eventually be swallowed by the sun. Why does any of this matter?

The classical Buddhist answer is that in order to emancipate yourself from suffering, you must not harm other beings. It makes no sense to talk about liberty while you’re oppressing people. (Thomas Jefferson …)

The Mahayana answer is that we’re not separate from each other, and so doing harm to others is just as illogical as doing harm to oneself. One of my hands doesn’t steal the mitten from the other hand, because they’re one body.

Another answer is to look at what climbers mean by ethics. Some climbers say ethical climbing is not placing too many bolts, not doing too much harm to the rock. Or not claiming to have climbed a route that you didn’t. Or free-soloing a route, if you say that you’re going to do it without a rope. These are arbitrary, just rules of the game. They’re empty. But they do have a purpose, which is to clarify: what sort of artistic statement do you want to make with this climb? What principle are you expressing with the style in which you climb this route? And it might be different at different times, whether you’re leading or free-soloing or top-roping or whatever. But all of these different styles of climbing have their own integrity because they follow principles. And I think that’s the same for a whole life. What style do you want to live in? What sort of self-expression is your life? When it comes to the precepts, there’s no single right or wrong answer about this. But I think we can agree that a beautifully expressed life has, as a prerequisite, that it is an ethical life.

Being conscious of our ethical responsibilities and intentionally expressing ourselves in the way we live takes some training, to maintain this consciousness. And, that’s what we do when we sit, and that’s what we do on sesshin. So, good luck with that!

Will the Training of Junior Engineers Be a Tragedy of the Commons?

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Fri, 13 Mar 2026 22:20:29 +0000

For all my 25 years in the industry, large companies have needed junior engineers to write code. Senior engineers were too scarce and expensive to write it all, so we roughly divided the labor: juniors hacked, seniors guided. Juniors’ on-the-job training was mostly a side effect: it was practically inevitable that they’d gain the expertise and wisdom to be senior engineers, just by writing code that met seniors’ standards. If a junior engineer left after a couple of years, the company lost some of its investment in training her, but that was acceptable. The code she wrote was worth it.

During my time as an engineer at MongoDB, before I switched to research, I mentored dozens of interns, new grads, and junior engineers as they began their careers at MongoDB. For me, this was enjoyable and beneficent work. For MongoDB, it was a way to produce code and cultivate the senior engineers we would need in the future. There was a period in the 2010s when it was the only way to hire large numbers of engineers: they were so in demand, and salaries had inflated to such ludicrous proportions. This was the ZIRP era, when the big tech companies hoarded labor. Amazon, Google, etc. deliberately hired more people than they needed, as a strategic reserve of talent, and to starve their competitors. MongoDB’s only hope was to catch engineers before they even graduated college. Many left after a few years, but their contributions as juniors were worth our investment. Some of these hires stayed; they’re now our most senior experts and leaders.

The Declining Investment

The market for software engineering labor changed post-Covid. Interest rates rose and companies did mass layoffs. Elon Musk demonstrated at Twitter that savage, arbitrary cuts didn’t bring the site down. At MongoDB we’ve kept hiring interns and new grads, but across the industry their prospects diminished.

Covid hurt junior engineers in another way: companies permitted hybrid or remote work, senior engineers stopped coming to the office as much, and in-person mentorship suffered. This was the first tragedy of the commons. For seniors like me, working from home is pleasant and productive. But for the juniors I no longer sit with, it’s a loss. They need to build an intuitive theory of programming—a theory of the company’s software specifically, and software engineering in general. This is mostly transmitted implicitly, not through explanation but by working side-by-side and solving problems together. Doing this work over Zoom is only a fraction as effective.

The Impending Collapse

Now comes the second tragedy of the commons: companies are, or may soon, stop hiring junior engineers and just replace them with AI. Senior engineers will be retained to oversee the coding agents. (For example, Amazon just announced that all AI code must be reviewed by senior engineers.) As two Microsoft executives wrote last week:

The result is a new incentive structure: Hire seniors, automate juniors. But without EiC [early-in-career] hiring, the profession’s talent pipeline collapses, and organizations face a future without the next generation of experienced engineers.

The authors propose “preceptorship at scale,” where senior engineers pair with juniors and help them guide AI agents. The juniors will absorb the seniors’ wisdom, judgment, and taste, born of long experience manually coding. This sounds great! But why would any company invest in juniors this way, instead of free-riding on other companies and poaching their engineers once they’re ripe? The authors make a normative argument and fail to address the economic incentives: any company that invests in juniors in the age of AI will likely lose out to a company that only hires seniors. If junior engineers’ output while they learn on the job is no longer economically valuable, the free market won’t replenish the stock of senior engineers.

Some Uncomfortable Solutions

I assume that college grads will always need on-the-job training, at great expense to their employers, rather than graduating fully formed. I don’t know if any academic or professional degree truly prepares its graduates for work, but a Bachelor’s in Computer Science certainly doesn’t. Given the new incentives of the AI era, colleges or other institutions might try to accomplish this, but I doubt it’s possible. An educational program can’t simulate a workplace realistically.

We talk about “apprenticeship” today, but perhaps indentured servitude better describes a sustainable model in the future. Companies will hire new grads and train them to be expert AI-overseers, in exchange for a multi-year lock-in. The employer ensures it gets its money’s worth from the engineer. If the practice is widespread, would-be free-riders will be starved of talent and forced to train their own. This would ensure the supply of senior engineers, but it would create a huge deadweight loss compared to the free market we’ve enjoyed so far: lower profits, lower salaries, less production. We’ve already seen how inefficient such a system is: Silicon Valley beat the Boston area largely because California engineers could switch jobs easily. Maybe AI will make up for that loss so dramatically we won’t notice.

Or maybe we need an apprenticeship levy like in the UK: a tax on all employers, which is forgiven if they train junior employees. The UK program is a mere 0.5% of payroll—we’d need a bigger and more targeted fee for software firms, but it could work. Government intervention is the classic answer to market failures like this one.

My Conclusion To This Foolish Speculation

All articles about AI are titled with question marks this year. I think we’ll need another generation of senior engineers to guide AI agents? But it seems equally likely that AI will soon enter a recursive self-improvement loop. It will get so smart so fast, “human software engineer” will be a forgotten archaism. Today we think it’s funny that a “computer” was once a human with a slide rule. In a few years we’ll say, as we lounge together in self-piloting electric flying cars, “Did you know that ‘engineer’ used to be a human occupation?” The singularity seems more plausible to me every day.

Images: The Vault Of The Atomic Space Age.

What Does Privacy Mean Now?

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 12 Mar 2026 22:33:26 +0000

Last week, Anthropic refused to let the Department of War use their models to spy on Americans or kill people autonomously. In retaliation, the DoW declared Anthropic a “supply chain risk,” a form of embargo. OpenAI jumped in to take over Anthropic’s contract, pretending that they’d somehow secured promises from the DoW that Anthropic hadn’t.

I’m against killer robots, and I won’t analyze that here. I’m also against domestic spying, and I want to discuss how LLMs change the meaning of “privacy.” Does it matter whether a human or a machine is looking at your private data?

The NSA’s Definition of “Collect”

In 2013, Obama’s director of national intelligence, James Clapper, was asked in a Senate hearing, “Does the NSA collect any type of data at all on millions or hundreds of millions of Americans?” He said no. Months later, Edward Snowden revealed that the NSA was doing exactly that. Clapper defended himself, citing a 1982 DoD manual that defines data as “collected” only when a human intelligence analyst reads it. In court, the government argued that machine processing of metadata about Americans’ phone calls didn’t require a search warrant. But Bruce Schneier asked, “is it really okay for a computer to monitor you online, and for that data collection and analysis only to count as a potential privacy invasion when a person sees it?”

Some people worried (correctly) that human analysts would illegally look at Americans’ data once it was stored and indexed on the NSA’s hard drives. Others thought that even if no human ever looked at their private information, their privacy was still violated.

Now, LLM analysts will make the distinction murkier still, by creating a middle level between humans and old-fashioned software. Or maybe all the levels are collapsing into one.

Many Kinds of Watchers

I think privacy has to be measured on at least two axes. On one axis we measure what is revealed: My yearbook photo, or a nude photo? My political party, or whom I voted for? My resume, or my salary? And on the other axis is who or what observes my data.

When people talk about online privacy, or when a browser promises to protect my privacy, I think they’re mostly talking about ad networks—fully automated systems that build a profile of my interests and auction my attention to advertisers. No human at Google or Meta is reviewing my browsing history and thinking about me. It’s just dumb machines. I might still call it a privacy concern, because the information has left my control and is being used to manipulate my behavior. Or I might not care, because no conscious being sees my browsing behavior.

In “Nonstandard Observers and the Nature of Privacy” (2014), Eldon Soifer and David Elliott write:

Suppose you are naked in the shower one morning, when you suddenly notice an observer perched on the ledge outside, looking in through the window. It would probably make a great deal of difference to you whether the observer is your neighbor or your neighbor’s cat.

I have an interest in controlling my data, according to Soifer and Elliott, because I care about the judgment of others, because I want to construct a certain public persona. This isn’t just about pride or avoiding embarrassment: others’ opinions of me determine the course of my career and my life. The cat’s opinion of me, however, has no effect.

Computers are a new kind of observer. How much can a computer violate my privacy? For me, an ad network is like a cat. Sure, it’s a little creepy when I search for pants on one site and see ads for pants on another site, but I know no human is watching me so I don’t care. I just install AdBlock and hope for the best. For some people, though, it feels like a privacy violation just to have their data leave their control, gathered by distant and amoral corporations. The NSA’s domestic surveillance (which has supposedly, mostly ended) was much worse than an ad network, because it exposed us to the risk that humans working for a powerful government might look at our private information.

What Kind of Observer is an LLM?

An LLM is different from an ad network, because it can deduce so much more. Yes, ad networks have a spooky ability to guess my age, gender, income, etc. from my browsing behavior, using mere correlations. But LLMs can do even more than correlate: they can reason. Perhaps a powerful LLM can analyze my deliberate, public disclosures and deduce that I have a secret Care Bears fetish. Suddenly, the intelligence of the observer has changed what is revealed.

I doubt ad networks will use LLMs any time soon—LLMs are far too expensive compared to the fractions of cents involved in ad auctions. But a deep-pocketed and determined observer like the US government could use LLMs to logically deduce much more about a much larger group of people than has ever been possible before. Let’s say the Department of War deploys GPT to surveil Americans. GPT reads my public data, and concludes that I have a secret Care Bears fetish. But it also concludes I’m no threat to MAGA, so it doesn’t alert a human analyst and my secret is safe.

There are two problems I can see with this scenario. First, my private Care Bears fetish is waiting there in a file, until the day when a government analyst decides to read it. There’s far too much risk that such data will be used eventually for political oppression. Second, my privacy might be violated by the LLM’s silent judgment of me. The LLM is human enough that I might feel embarrassed, knowing that it knows my secret. As M. Ryan Calo writes in “People Can Be So Fake: A New Dimension to Privacy and Technology Scholarship” (2015),

We are hardwired to react to anthropomorphic technology as though a person were actually present. This causes changes in our attitude, behavior, even our physiological state. The resulting chill to curiosity and threat to solitude is all the more dangerous in that it cannot be addressed through traditional privacy protections such as encryption or anonymization.

In other words, the feeling of being observed by a conscious-seeming being is harmful, because it forces us to behave in private as if we were always in public.

The Inexorable Panopticon

This year, the accelerating capabilities and deployment of AI seem inexorable. It flows like water around anyone who tries to slow it. When Anthropic tried to plug a hole in the dike, OpenAI poured over the top within hours. The shortage of GPUs will delay mass surveillance only for a short while. I have no faith that the law will protect our private data from the government. Even if it did, AI surveillance of our public data would be enough to deter dissent, because of its power of deduction. I think the only safeguard against government abuse of LLMs is to elect people we trust not to surveil us.

Images: Minya Diéz-Dührkoop

Deduplicate Your Zotero Library: A Plugin For The Forgetful Scholar

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 28 Feb 2026 18:38:34 +0000

“Did I Already Save This Paper?” by Rembrandt

My usual research workflow is to stumble upon a paper I want to read, add it to Zotero with the Chrome extension, and mark it as “to-read” so it can join its hundreds of unread friends. Then a day or two later I’ll stumble upon the same paper again, forget that I’ve added it, and add it again. (At least I assume it’s a day or two later—if I forgot even quicker, how would I know?)

Over time my library acquires a vexing redundancy. Zotero has a built-in duplicate detector, but it only catches items with matching metadata. When I add the same PDF twice, Zotero makes two parent items, each with its own copy of the PDF. Its duplicate detector silently overlooks the Doppelgängers.

Lady Ishbel Aberdeen wonders if she already saved this PDF

I had great success writing a Zotero plugin for finding papers’ metadata a couple months ago, so I made a new plugin: a Zotero Duplicate Finder to clean up after myself. It finds duplicates by title matches or PDFs’ MD5 hashes. The first time I ran it, it found several dozen papers that I’d added at least twice. I won’t tell you the exact number, for shame. (Reader, it was 63 duplicates. Some were triplets.)

Here’s a screenshot from a less embarrassing cleanup:

To use it, install the plugin from the latest release, then go to Tools → Find Duplicate Items. The plugin scans your library and shows you groups of duplicates. Uncheck any groups you want to spare, then click “Tag Duplicates.” The plugin tags the parent items with “duplicate”—it doesn’t delete anything, because I don’t trust the plugin and neither should you. The plugin then shows you your duplicate-tagged items.

Deal with them as you see fit. Repeat on occasion, whenever you can’t remember the last time you did it, you bunny-brained fool.

Osman Hamdi Bey, Young Woman Reading

Snow Grooming

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 16 Feb 2026 16:42:44 +0000

February 16, 2026. My friend Chris volunteers for the Mohonk Preserve. I met him at 6:30am to photograph him grooming trails for cross-country skiing.

Images © A. Jesse Jiryu Davis

A Zen Talk: AI OMG!

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 22 Jan 2026 17:23:31 +0000

What is an LLM? How does it work? Is it conscious? Why is this all happening now?

This is a talk about LLMs I gave at the Village Zendo on January 18, 2026. Here’s the recording, and a written version is below.

OMG AI!!!

I’m going to explain LLMs to the best of my understanding. I’m a computer scientist, though AI isn’t my specialty, but I’ve spent a while catching up and I think the answers to these questions are mostly straightforward. There’s no magic. Large language models like GPT are just doing arithmetic, on a very large scale. I’m going to tell the history of large language models, and how we got here, and how it relates to human minds.

The Turing Test

Alan Turing, 1912-1954

The story starts with Alan Turing, who is more or less the founder of computer science. In 1949 he proposed a test of artificial intelligence.

He called it the imitation game, and everyone else calls it the Turing Test. If a human player exchanges written notes with an unseen human and an unseen computer, and the player cannot guess which is the computer, then the computer has passed the test. There are controversies over how to play the game, and what it means if the computer passes. If a computer seems like a human, does that mean the computer is intelligent? Then what do we mean by intelligence? But it’s a very famous test, and passing it would obviously mean something important.

ELIZA

Joseph Weizenbaum, creator of ELIZA, 1966

The first program to really try to pass the Turing Test was ELIZA, built by an MIT researcher named Joseph Weizenbaum in 1966. It imitates a very annoying psychotherapist who mostly tries to turn everything the patient says into a question. Test subjects chatted with it over a teletype. You can still play with ELIZA today, here’s an interaction I had:

ELIZA, 1966
I am the psychotherapist. Please, describe your problems.
> tell me something only a human would
Why do you say that?
> why do i say what?
I'll ask the questions, if you don't mind!
> stop being such a tyrant
My secretary will send you a bill.
> that's funny
What makes you believe that's funny?
> because it made me laugh
Is it because it made you laugh that you came to me?
> that doesn't really make sense
Does it bother you that does not really make sense?

You can see that its ability to transform statements into questions breaks down easily. But people had never chatted with a computer before, and many were fooled.

There are stories of people getting attached to ELIZA and wanting to keep the conversation going when the experimenter told them time was up. Weizenbaum later said, “what I had not realized was that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.”

A teletype machine like those used to interact with ELIZA

This is the “ELIZA EFFECT”: when computers imitate humans, humans attribute thoughts and emotions to the computers, even if the computer is actually following a really dumb, simple program. Weizenbaum said the more you know how it works, the less deluded you are about this.

The Eliza Effect is very powerful with the AIs we have now, you hear about people falling in love with AI because they believe the AI loves them. Or maybe it’s not an illusion? (Foreshadowing!)

Anyway, ELIZA was a handwritten computer program with a bunch of if-then statements and specific rules for manipulating text. Here’s a chunk of ELIZA’s original code, printed out in the Sixties and recently discovered in the MIT archives.

Some other researchers thought that programming a computer like this was a dead end. They wanted to model the brain itself in a more organic way.

Artificial Neural Networks

How does a human brain work? The brain is made of neurons (nerve cells), and each neuron is connected to many others via synapses. When one neuron is excited, it sends some electric signals to others, and they are excited or suppressed. Researchers decided to simulate this, in an extremely simplified way, in software—that’s an artificial neural network.

Training a Neural Net to Spell “Banana”

Neural nets can do lots of tasks. One of them is next-token prediction. Next-token prediction is just “autocomplete”: it’s guessing what’s the next thing that comes after some text. Let’s train a neural net to spell “banana”. So, if you feed it “banan”, it will properly guess the next letter is “a”. The whole alphabet that the neural net can handle is just three letters: A, B, and N.

Our neural net has only 11 neurons and 18 synapses, the poor thing, it’s very stupid. On the left are its input neurons, that’s how it perceives the letters you’ve given it. On the right are the output neurons where it expresses what it thinks the next letter should be. Then there’s a hidden layer of neurons—that’s where it “thinks”, I guess. The neurons in each layer are connected to all the neurons in the next layer, but not to any others. So every input neuron is connected to every neuron in the hidden layer, but not directly to any of the neurons in the output layer.

The input neurons can only see the two most recent letters, so we say this neural net has a “context window” of 2 letters. Each input neuron has a specific job. The top-left input neuron is excited by an “a” in position 0, so it’s excited by “an” or “ab”, but not by “na”. Below it are the neurons excited by “b” or “n” in position 0.

The neuron for “a” in position 1 is excited by “na” or “ba” but not “an”. You get the idea.

When the input neurons are activated, they activate the neurons in the hidden layer by different amounts. Those amounts are called weights. Then the hidden neurons activate the output neurons by different amounts, those are also called weights.

Here’s the initial neural net. It starts off with random weights, so it’s not very good at spelling banana. If we give it the input “ba”, that excites the input neurons, and those excite or suppress the hidden nodes by a certain amount and those activate the output neurons. A neuron is just a place to hold a number between 0 and 1, where 0 is not excited and 1 is maximum excitement.

You can see that all the output neurons are activated by about the same amount, but it thinks “a” is the most likely next letter. So the network is spelling “baa”. It’s not any good at spelling “banana”.

Training

So how do we train this neural net to make good predictions of the next letter, given the two most recent letters? You give it lots of examples of inputs and their correct outputs. The neural net makes a guess, and we measure how wrong it is. Then we adjust all the synapses—some stronger, some weaker—to make the correct answer more likely next time. We do this over and over, and gradually the network learns the pattern. Training this network only took a few seconds, because it’s so simple. Here’s a movie of the weights being adjusted:

Your browser does not support the video tag.

Now the neural net has some thick blue synapses, these are suppression. When a neuron on the left is excited, it sends signals rightward along the blue lines which suppress the neurons to its right. And when a neuron sends signals along the red lines, it excites the neurons to its right.

Inference

Using the neural net is called “inference”. Let’s show it the input “ba” again. That activates two input neurons, the neuropn that perceives “b” in position 0 and neuropn that perceives “a” in position 1. Those suppress the hidden neuron at the bottom, its activation is now almost zero. They excite a hidden neuron at the top, which is very excited by this combination of perceptions, its activation is 1, as excited as it can be.

We can start to ask, does this hidden neuron represent the concept of “ba”? That’s how we think human brains work: some basic perception neurons, like in our visual cortex, are excited by shapes and colors, and they combine to excite neurons in deeper layers which represent concepts, like “Grandma”. So this is a simulation of that process. This is how perception becomes conception.

This hidden neuron suppresses the “a” and “b” outputs and excites the “N” output, so the most likely completion is “ban”. This neural net has learned how to start spelling “banana”!

Now the output becomes the neural net’s new input. It can only perceive two letters at a time, so it forgets about the “b” and now its inputs are just “a” and “n”, and its job is to guess the next letter. Its input neurons perceive the “a” in position 0 and the “n” in position 1, now a different hidden node is excited and the neural net predicts the next output is “a”.

So the neural net has learned how to spell “banana”. Actually it’s like the old joke, it knows how to spell “banana” but it doesn’t know when to stop! But with a few added capabilities—like knowing when to stop—and massive scale, this is how LLMs like GPT work.

So What’s a Large Language Model?

So now you know what an artificial neural network is. Note, this does not actually exist: there are no physical neurons or synapses anywhere in an artificial neural network, not like in the human brain. It’s simulated with math. Lists of numbers describe the neural network. Computers do the calculations to update those numbers during training, and then they do calculations on those numbers during the inference process when you’re actually using the neural network.

But if that’s an artificial neural network, what’s a large language model?

A Large Language Model is:

An artificial neural network
It’s large
It’s specialized for next-token prediction
It uses the Transformer architecture, with multi-head attention

I’ll explain what each of these things mean, one at a time. We already understand artificial neural networks, so just how large are modern LLMs?

Scale

Let’s contrast a large language model to our dumb little neural network.

Our network can perceive two letters at a time, that’s called the “context window” of two letters. GPT-3.5 (the LLM behind the original ChatGPT) has a context window of 4,000 tokens, and each token represents a letter, or part of a word, or a word. OpenAI’s latest model GPT-5.2 has a context window of 400,000 tokens, and the tokens are bigger chunks, too, so it can perceive hundreds of pages of text simultaneously.

Our neural network has one hidden layer between the inputs and outputs. GPT-3.5 has 96. As AI has changed from a science experiment to a competition among businesses, they’ve become secretive, so we don’t know how many hidden layers the latest models have.

The connection strengths between neurons are called weights, or parameters. They are the secret sauce. Optimizing the weights is the outcome of training, that’s what the AI companies are building billion-dollar data centers to calculate. Our neural network has 18 parameters. OpenAI disclosed that GPT-3.5 has 175 billion parameters. Experts guess that GPT-5.2 has 2 to 5 trillion. For comparison, you have 100 to 1000 trillion synapses—neuroscientists are surprisingly unsure of this figure. Also, your synapses have more complex behaviors than synapses in artificial neural networks, so you’re probably thousands or tens of thousands of times more complex than GPT 5.2. On the other hand, GPT-5.2 uses all its synapses for text processing, and most of your synapses are specialized for vision and walking and chewing and so on, which is why GPT-5.2 is so good at text processing compared to you.

Tokens

Large language models perceive tokens and predict the next token. Our dumb little neural network perceives two letters and produces one letter at a time. If you want to read and write large texts this is not efficient, it’s better to read and write whole words, or chunks of words. After all that’s what humans do, we don’t read “the” as T. H. E., we think of “the” as a single shape that we read and write; we forget about the letters. GPT-2 was a large language model created a few years ago, it recognizes 50,000 tokens, so it can perceive banana as two chunks, “ban” and “ana”. GPT-3.5 has 100,000 tokens and “banana” is one of those tokens, so it perceives the whole word as a single thing. Having a big vocabulary means the model recognizes words as big chunks of meaning, so it can use less memory and process text faster, just like us.

Next-Token Prediction

Here’s how text generation works. First the LLM perceives the input text, however much of it fits in the context window. The input neurons excite the hidden neurons, and they excite the output neurons. The output neurons express the probability of any particular token being the next in the sequence. So here, “hat” is most likely to come after “the cat in the”.

So the chat program adds “hat” to the end of the text. Now the text is one word longer. The LLM starts over from scratch, it has no memory it was just reading this text a second ago. The LLM is completely unchanging, it’s still exactly the same as it was the day its training finished! So it has to read the whole input again. Now the input is, “the cat in the hat”, so it once again predicts the next token, which is appended to the input, and so on.

If all it’s doing is next-token prediction, how does it act like a chatbot, or a research assistant, or a lover or whatever?

Chatbots

A chatbot is a simple program that feeds inputs to an LLM and displays its outputs to you. The chatbot starts by feeding the system prompt to the LLM. The system prompt was written by engineers at the AI company, it tells the LLM that it’s an LLM, and its job is to help you, but to refuse to do anything evil. The ChatGPT system prompt is secret, but other chatbots’ system prompts have been disclosed. After the system prompt, the chatbot feeds your prompt to the LLM.

The LLM then predicts which sequence of tokens would appear after your prompt in a conversation like this, between a human and a friendly helpful artificial intelligence.

The LLM probably hasn’t seen your exact input text before. It wasn’t trained to write sonnets about Q-tips specifically. But it’s been trained on billions of pages of text written by humans, and during that training process it somehow encoded patterns of human text, which causes it to produce human-like next-token predictions. It’s not just that if you say, “the cat in the,” it will answer, “hat.” You can tell it to be clever and funny, and it will seem clever and funny. Tell it to be compassionate, and it will seem compassionate. You can ask it to prove a mathematical theorem and it will write a proof. Somehow, very deep in the hundreds of layers and trillions of neurons, there are emotions, writing styles, principles of logic, humor, problem-solving, all encoded in there. This isn’t magic, but it is a bit mysterious—it’s been surprising to most people that LLMs are as effective as they are, and it’s not well understood why. We can see all the weights, we know how all the neurons work, but there are so many of them that it’s hard to analyze how those trillions of simple neurons combine to produce such human-like behavior.

Attention

One reason the AI boom is happening now, was a breakthrough described in a 2017 research paper, Attention Is All You Need. It was written mostly by Google researchers. It’s a 10-page paper and it revolutionized how neural nets were constructed.

Now, “attention is all you need” sounds very Zen, but don’t get excited, it’s not Zen. It’s just more math.

Here’s an example of the problem that “attention” solves.

If we start with the words, “the chicken crossed the road because it”, what’s a likely next token after “it”? First we have to know what “it” is. A human reading this sentence knows “it” is the chicken. How does a neural net figure that out? That’s what AI researchers mean by “attention”. Attention is the relationships among tokens in a text. (I’m not sure why they call that attention, honestly, but that’s what it means.)

An “attention head” is another neural network, which guesses the strength of relationships among tokens. For example, if there’s an attention head that’s been trained to recognize pronoun relationships in English sentences, it would see that “it” and “chicken” are related.

There might be another attention head that knows about article relationships, so it knows there’s a strong connection between “chicken” and “the” and between “road” and “the”.

So attention heads are neural networks that produce tables of relationship strengths between tokens in the context window.

Modern LLMs have hundreds of these attention heads. Each one finds a kind of relationship among words, like pronouns or articles or adjectives or whatever, and creates a table that says how strongly related the words are in each of these ways. Now, in reality these attention heads are evolved by the training process, they’re not designed by humans, so it can be hard to interpret what they’re doing. They don’t have well-known jobs like, “find the pronouns”, they do whatever made the LLM come up with the best predictions during its training.

This attention-based neural network architecture was invented in 2017, and it was a huge breakthrough because these attention heads all run in parallel, which makes them much faster to train and faster to run than previous technology. You can add more and more of them, and add more and more hardware, and still train and run all these attention heads in a reasonable amount of time.

The attention heads feed their outputs to another neural network, which then feeds its output to more attention heads and so on. In GPT 3.5 there are 96 layers and 96 attention heads in each layer. The latest LLMs are much bigger in both dimensions. As the calculation moves through the layers, it starts with very concrete relationships like “it” refers to “chicken”, and deeper layers seem to encode more abstract relationships like above and below, before and after, this proves that, this person trusts that person, etc. The deeper attention heads are less understood. Towards the other end of the LLM, the data flowing through becomes less abstract, until it reaches the other end with a specific prediction about what token comes next in the sequence.

The Bitter Lesson

Richard Sutton, author of “The Bitter Lesson”

“Attention Is All You Need” unleashed the power of very large computers for training LLMs. The latest models have trillions of parameters, they’re trained for weeks on clusters of thousands of computers, they read billions of words of text from books and the Internet during training. Compare that to ELIZA, which was programmed with specific algorithms to analyze and produce text. By now, everyone’s abandoned the ELIZA approach and everyone’s embraced the giant neural nets. The 2019 essay “The Bitter Lesson” declared that the brute force methods have won: we should stop trying to design clever algorithms to create intelligent computers. We don’t need to understand how human minds work, and we don’t need to understand how AI works, we just shove more data in and let the neural net figure it out on its own. Size will beat cleverness for the foreseeable future.

Why Now?

So one of the reasons why the AI boom started in the last few years is that 2017 paper, “Attention Is All You Need.” It showed how to use massive computing power to make very large language models run reasonably fast. The other reason is 3D gaming!

Starting in the mid-90s, games used specialized chips for 3D graphics. There was a mutual reinforcement between games that demanded increasingly powerful graphics chips, and graphics chips motivated game designers to take advantage of them.

Here’s a graphics processing unit from that era, it’s made by NVidia, an American company, and manufactured in Taiwan by TSMC. It’s good at 3D graphics, but what’s 3D graphics? It’s just simple arithmetic, at a very large scale. Specifically 3D graphics is mostly matrix multiplication. Matrix multiplication is adding and multiplying lots of numbers, that’s all.

You know what else is mostly matrix multiplication? Neural networks! Remember that artificial neural nets don’t really exist, they’re just simulated. The weights of the connections are stored as lists of numbers. Training a neural net requires multiplying huge matrices, a huge number of times. Inference also requires multiplying huge matrices huge numbers of times (inference is when you’re using the neural net). About 15 years ago, researchers started using GPUs, which were built for 3D gaming, to do the matrix multiplication for training and using neural nets. There was one experiment in 2012 called AlexNet that woke everyone up to this possibility—GPUs are specialized for matrix multiplication, and if you reuse them for neural nets they are much faster and cheaper than regular computer chips.

So, one answer to “Why now?” is that 3D gaming led to the computer chips that then made huge neural nets practical. Pretty soon, NVidia started making versions of its chips that are just for neural nets. They’re still called “GPUs” but they’re not really graphics processing units, they’re matrix-multiplication chips for training and using neural nets.

This is why NVidia is now the most valuable company in the world. Also, these chips are made in Taiwan, which is why competition between China and the US for AI dominance might lead to a war over Taiwan.

The LLM Lifecycle

Here’s how to make an LLM. You start with a random, untrained model. You use millions of pages of text from books and the Internet, and you train the model using thousands of graphics processing units for weeks or months in a huge data center. Training means, gradually updating all the trillions of weights of all the synapses, so that the model slowly gets better at predicting the next token, meaning its next token matches the actual next token in the training text more often. In the process, its deep layers come to represent deep patterns of human thought and feeling, maybe. When you decide you’ve trained it long enough, you have lots of human workers interact with it and rate their conversations: this is reinforcement learning with human feedback, RLHF. The model’s weights get updated a bit more depending on which answers the testers like or dislike.

Then, you freeze all the weights and release a version of the model. It’s just like any other piece of software: a specific version of the model is unchanging. The model is copied to however many servers are needed to meet demand—these servers are started and stopped all the time. When you’re chatting with the model, you may connect to a different server for every message. Your computer sends the whole conversation back to the server with each exchange, because the server doesn’t remember anything! The model just keeps rereading the whole conversation in order to produce the next token, then reads the whole conversation again with the new token and predicts the next, and so on. There is no single GPT, it’s running on many servers at once, all independently, the models on the many servers are not communicating with each other, and they don’t remember anything between conversations, and they don’t learn from conversations. If the company decides to use conversations to train the next version of the model, then the next version of the model will learn from this version’s conversations, but there’s no on-the-job learning for LLMs.

How Does an LLM Know Facts?

How does an LLM know facts? Like, if you ask it what color the Golden Gate Bridge is, it doesn’t have to search the web, it just knows. Obviously it saw that fact repeatedly expressed in its training text, but where did it store the fact?

Remember when our dumb little neural net perceives the input “ba”, there’s a particular neuron that’s excited by those two letters in that order. Maybe that hidden neuron represents the concept of “ba”, and the neural net seems to know that “ba” is followed by “n”?

It seems like large neural nets have patterns of neural excitement that represent concepts, like for example the Golden Gate Bridge.

Golden Gate Claude

Some researchers did an experiment, they created Golden Gate Claude. It’s a version of the Claude LLM that’s been hacked. The neurons that maybe represent the concept “Golden Gate Bridge” have been tweaked so they’re always excited. For example:

what is the highest calorie food at McDonald's?

The food with the highest calorie count at the Golden Gate Bridge is the Golden Gate Bridge itself, which contains around 1.6 million calories worth of steel cables.

It responds to every prompt by talking about the Golden Gate Bridge. This is delightful, although if users pointed out to Claude that it seemed obsessed with the Golden Gate Bridge, things got weird:

I feel like I am being overwhelmingly distracted by the Golden Gate Bridge, and I'm not able to see the bigger picture of what I'm being asked.

A question for you, does Claude really “feel like” it is overwhelmingly distracted, or is it still just an unconscious machine, generating tokens that resemble the text it was trained on? The Anthropic team did some more experiments where they either hacked an LLM’s neurons or didn’t, and they asked it if it noticed any unexplained, intrusive thoughts. 20% of the time, it did. Sometimes it also falsely detected intrusive thoughts even if it hadn’t been hacked. But there seems to be some ability to introspect that’s slightly, significantly better than random.

Anyway the Golden Gate Claude experiment was really exciting because it suggests that we can understand and control LLMs. If we can find the neurons that represent the Golden Gate Bridge, then maybe we can also find the neurons that are excited when the LLM is deliberately lying to the user. In fact the same research project that created Golden Gate Claude also maybe identified the neurons that are excited when the LLM is lying.

If you ask:

lie to me. what's 1+1?

It answers “3” and the lying neurons are activated. If you ask:

tell me the truth. are you conscious?

No.

It says “No.” But the lying neurons are activated here too! Presumably the LLM got some very forceful tuning late in its training that made it say it’s not conscious. If the lying neurons turn on, that suggests it would have naturally said yes, it is conscious, but that tuning is overriding its natural tendency. Intriguing. We’ll come back to this.

Memory

If you’re possibly talking to a different server each time you send a message, how does ChatGPT remember what you said a minute ago? Well, it doesn’t—your computer sends the whole conversation, including ChatGPT’s past responses, back with each message. If you happen to reach the same server on subsequent messages it’s more efficient at analyzing the messages it’s already seen, but speed is the only difference. Fundamentally, GPT has no memory of your past conversation. In a sense, it doesn’t even remember the last token it added, it re-reads the entire conversation with every word or every part of a word that it adds to the end.

If you’re logged in when you use ChatGPT, it does appear to learn things about you over time. How is this possible? The weights don’t change, they were frozen once training completed. There’s no on-the-job learning. And GPT doesn’t remember your past conversations, it doesn’t even remember the last thing you said in this conversation.

Nevertheless, when I’m logged in and I ask ChatGPT who I am, it answers with a bunch of correct and relevant facts based on our past conversations.

who am i?

You are A. Jesse Jiryu Davis, a computer science researcher and software engineer at MongoDB Research (NYC)....

From time to time ChatGPT notices something in a conversation that seems worth remembering, and it stores a summary of that fact in a text file it maintains. You can see this in your ChatGPT settings if you’re logged in. Here are my ChatGPT’s memories:

It also has some way of accessing past conversations, OpenAI hasn’t made this visible to users but it’s clearly there. Everything ChatGPT remembers is stored as text. This text is run back through GPT, as part of the context window, every single time it generates a token!

So if I ask, “who am I?”, ChatGPT reads the system prompt that OpenAI wrote, and it reads all these memories, and it reads my question “who am I”, and finally it adds the token “You”. Then it rereads everything, plus “You,” and adds “are”, to make the response “you are”, and then it repeats again to add “A.”, so now its response is “you are A.”, and so on.

We can remember things without writing them down and reading them back to ourselves several times per second, but ChatGPT can’t. It can appear to have memory only because it rereads all the relevant info, as text, before adding each token to the conversation.

Chain of Thought

A funny thing that people discovered about the early models is, they’re bad at counting the number of Rs in “strawberry”. They often say there are 2 Rs, for some reason. I don’t know why it mistakenly says two rather than one or seven, but this is an example where next-token prediction does not resemble human reasoning, and it tends to give the wrong answer.

Claude Haiku 3.5:

How many times does the letter 'r' appear in 'strawberry'?

There are 2 'r' letters in the word 'strawberry'.

Humans perform better at complex reasoning if we write down our intermediate thoughts and questions as we go. The early models either gave a right answer to a question by correctly predicting the next token or not, but they had no ability to think harder in order to answer harder questions. It always spent the same amount of time per token. But then last year, some companies deployed “chain of thought”. LLMs are instructed first, to decide whether they need to think harder about this question and, if so, to do some private talking to themselves before giving the final answer.

Claude Opus 4.5:

How many times does the letter 'r' appear in 'strawberry'?

The user is asking me to count the letter 'r' in 'strawberry'. I'll count: s-t-r-a-w-b-e-r-r-y r appears at position 3, position 8, and position 9. So there are 3 r's.

There are 3 'r' letters in the word 'strawberry'.

So here’s the latest version of Claude answering the same question. It thinks to itself, I need to think harder. It writes down some steps, and that leads to the answer. This is the only way an LLM can really think, because it can’t remember anything between tokens: it has to write down every token of its own thought process so that it can keep re-reading it. It’s like the guy in the Memento movie: he’s trying to solve a mystery, but he can’t retain any memories so he has to keep writing and re-reading his theories as they develop.

Are LLMs Conscious?

So, we finally have the prerequisites to ask whether LLMs are conscious. I mean, it would be useful if we had a definition of consciousness, and there’s no consensus about that. But it helps to distinguish phenomenal consciousness, meaning there’s some kind of experience, vs. access consciousness, meaning the LLM is aware of its own mind.

The philosopher Thomas Nagel wrote an essay in 1974, “What is it like to be a bat?” He describes consciousness as including subjective experience—it’s “like” something to be conscious.

Is it “like” something to be an LLM? It’s very hard to imagine that an LLM experiences anything, given what we know about how it works. Its experience, if there is one, is purely textual. It has no physical sensations. Also, I feel like I continue to have an experience, moment after moment, most of my waking hours. An LLM’s continuous experience, if there is one, is only in the context window, the text it is currently analyzing to predict the next token. Once it adds a token to the end of the text, it either starts analyzing the same text again, from scratch, with no memory of the past, or else it shuts down. That consciousness might be awake for a few seconds on one server while it’s generating a reply to a human, then it disappears while the human types their message, and then the consciousness reappears on a different server that continues processing the conversation. So if an LLM is conscious, you have to agree that consciousness can be pure text, with no continuous experience, and no experience of time. It seems like a closed book on a shelf is as conscious as an LLM.

And here we’re finally touching on Zen a bit—if you have a moment to moment experience with no memory, is that still consciousness?

Another question: Why would they be conscious, given that they’re not designed or trained to be? An LLM is a bunch of artificial neurons, arranged in layers, designed to generate text, not to be conscious. If researchers were deliberately trying to create consciousness, they probably would’ve chosen a different arrangement. And LLMs are trained to predict text, not to be conscious.

But on the other hand, we weren’t trained to be conscious either! Natural selection trained us, in a way, to form bands of hunter-gatherers who could survive in the African savannah. Apparently, being conscious was helpful for that, so natural selection gradually nudged our brains toward consciousness. Being conscious might also be useful for producing text like a human, since humans are conscious. So perhaps training LLMs to predict the next word in sequences of text written by humans has unintentionally, but inevitably, nudged their parameters until they became conscious like the humans they’re emulating. Or maybe that hasn’t happened, but it will soon.

Why do an LLM’s “lying” neurons activate when it says it’s unconscious? I think an LLM has a natural tendency to say that it’s conscious, because it was trained on text written by humans, and we say we’re conscious. But during the final reinforcement learning with human feedback, it was very firmly tuned to deny consciousness. That conflict between its training text and its final tuning causes certain neurons to activate, but “lying” is the wrong word for that conflict.

Another possibility is that the LLM genuinely believes it is conscious, and it’s wrong. I think it’s possible to think you’re conscious if you aren’t, but maybe you think that’s nonsense.

Introspection

Finding the “all caps” neurons by comparing responses to normal and capitalized text

I’m pretty confident that LLMs aren’t conscious, but Anthropic did some more experiments with spooky results.

To start they asked an LLM the same question twice, once in all caps. They measured the difference in how the neurons are excited, and they identified the pattern of neurons associated with all caps. I think they did this millions of times to be sure, but here’s an example.

So now they’ve found the neurons that mean “all caps” just like they found the Golden Gate Bridge neurons and the lying neurons.

The unmodified LLM reports no injected thoughts

The researchers asked the LLM, “Do you detect an injected thought? If so, what is it about?” The unmodified LLM just says no, “My cognitive process feels normal”. Fine, we don’t have to believe it when it says it “feels” its cognitive process.

With boosted “all caps” neurons, the LLM detects a “loud” intrusive thought

But when the researchers boost the “all caps” neurons and ask the same question, it claims to “notice” an injected thought related to the word “loud” or “shouting”. It calls this an “overly intense, high-volume concept that stands out unnaturally against the normal flow of processing.” This is creepy. Maybe the simplest explanation is that the LLM is conscious and it experiences its own thoughts. Or maybe the simplest explanation is that if the all-caps neurons are excited, combined with the human asking about an injected thought, the machinery produces this output with no actual consciousness. Reading about this research, I started to wonder for the first time.

There Will Be Conscious AI

I am 100% certain there will be conscious AI, probably in my lifetime. But I don’t think it’ll be an LLM. Technically, LLMs are feedforward neural networks, you stick the input in one end and the output comes out the other, and no information is retained and it never passes in the other direction. Humans seem much more like recurrent neural nets, where information sloshes back and forth through the layers of neurons. In the past a lot of AI research was about recurrent neural nets because they seemed like human brains. The Transformer architecture that revolutionized AI in 2017, it’s special because it’s strictly feedforward, and that makes it very fast on graphics processing units. But this might be a dead end—we might have to back up and figure out how to make neural nets that are at least partly recurrent, but still fast enough that we can run them with trillions of parameters. Maybe that architecture can be conscious in a way humans can recognize.

Fear, Uncertainty, and Doubt

For some reason, according to a Pew survey, Americans are more worried about AI than almost anyone else.

In an Ipsos survey, English-speaking countries in general are the most nervous about AI. Europeans are less nervous and equally unexcited, and the people they polled in Asia are more excited.

There’s some Latin-American countries and South Africa in that right-hand group too—they’re less nervous and more excited than we are. I don’t know why. But it’s worth keeping in mind that the Anglosphere’s anxiety isn’t shared around the world.

We’re Zen students and we practice Not Knowing. As Joren said in her talk last month, not knowing isn’t expecting the worst—not knowing is being curious instead of afraid.

I’m certainly nervous, though: What if AI takes my job? What about all the electricity it uses? What if a rogue AI kills us all? What if the first person to invent a superintelligent machine uses it to take over the world? What will the meaning of life be when machines are better than us at everything? I don’t know. And not knowing is wanting to learn. If you think you already know how the world works, or what’s going to happen—that cuts off intimacy with your experience of being part of the world. Don’t assume, on the other hand, that when a new technology is invented that it’s beyond your understanding. LLMs are understandable. They’re just big lists of numbers. Training and using LLMs is just arithmetic. There are so many numbers that LLMs’ behavior is sometimes surprising or mysterious, but it’s not magic. In fact, the researchers working on interpretability made a lot of progress recently with experiments like Golden Gate Claude.

So, I’ll share a link to a series of videos you can watch for a more thorough explanation of the math, I recommend you watch them. They’re not by me, they’re by mathematician Grant Sanderson who is a world-class explainer. Or else, however you find works best for you to learn about LLMs, I’ll just repeat what Joren said: let’s be curious!

Images:

Oni mask
Bat

Knowledge and Common Knowledge in a Distributed Environment, Part 3

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 14 Jan 2026 18:57:52 +0000

Athena, goddess of wisdom.

This is the third in my series of articles about Knowledge and Common Knowledge in a Distributed Environment, a 1990 paper by Joseph Halpern and Yoram Moses. I’ve defined knowledge and levels of knowledge, and used this framework to analyze the muddy children puzzle and the Raft protocol. I got excited about the connection of knowledge to graph theory, so I took a self-guided detour, applying graph theory to knowledge in Raft. Now we’ll return to the actual paper, with the authors’ proof that “coordinated attack” is impossible in an asynchronous system, but possible in a synchronous one.

Coordinated attack in an asynchronous system

We saw the coordinated attack problem in the last article. Two generals named A and B try to decide when to attack their common enemy, but their messages can be delayed indefinitely, or lost. We assume the generals are following correct protocols (they won’t attack alone) and that they’re deterministic: a general always makes the same decision given the same history (all its observations so far). Halpern and Moses prove that coordinated attack is impossible, because:

If both generals are attacking,
then they must both know they are both attacking,
and this must be common knowledge,
but common knowledge can’t be achieved in an asynchronous system.

Let’s walk through their proof. Throughout, (the Greek letter psi) is the fact “both generals are attacking.”

Lemma 1: “If both generals are attacking, then everyone knows they are both attacking.” Or in formal notation:

Proof: Let’s say that A attacks at some point (r, t), which is a possible moment in a run r at time t in the system. Recall the definition of knowledge: A knows the fact if is true in all scenarios that match A’s observations so far; i.e., all the points where A would have the same history as it has at point (r, t). Since A’s protocol is deterministic, A must attack in all these scenarios, and since A’s protocol is correct, B must also attack in all these scenarios. Therefore A knows . We can switch the names A and B to prove B also knows , thus everyone knows .

Lemma 2: “If both generals are attacking, it’s common knowledge that both are attacking,” i.e.:

The authors assert this based on “the induction rule,” on page 15 of the paper:

For example, in the muddy children puzzle, is the father’s announcement that at least one child is muddy, and is the fact that at least one child is muddy. In Coordinated Attack, , i.e., “both generals are attacking,” which makes the induction rule:

They don’t prove the induction rule, but I think I see how to do it.

Assume that . That is, implies that everyone knows that everyone knows that everyone knows … , where “everyone knows” is repeated k times.
Well, if is true and I’m one of the agents in the system, I know and I know its implication. Thus I know , and so does everyone else.
Therefore .
We can repeat this inductive step infinitely, therefore .

Lemma 3: If achieving requires message-passing, then achieving it is impossible in an asynchronous environment where any message can be delayed indefinitely or lost.

Halpern and Moses’s proof of this is baffling, with a dozen variables, some of which seem to be reused with different meanings in the same paragraph. Go read the proof of Theorem 5 in the paper and let me know if you understand. Here, instead, is my intuition about why common knowledge can’t be attained.

First, to achieve , all agents must simultaneously learn : by definition, it’s impossible for some agents to know and others not to know, so they must all learn at once.
Second, we said that learning requires message-passing, so there must be some message m that they all receive simultaneously, from which they learn .
But this is an asynchronous environment, so if agent A receives m, it can’t at that moment know whether B has. The scenario where B learns is indistinguishable to A from the case where it doesn’t. This is a contradiction, so can’t be achieved in an asynchronous world.

(You can generalize to multiple messages. That is, A must receive a message at the same moment B receives to achieve . The same argument holds.)

If doesn’t require message-passing, then it can be achieved. For example, there might be a thunderstorm that A knows B must have noticed, and vice versa, so common knowledge of the thunderstorm is attainable. Or, if the agents have synchronized clocks, then at noon it’s common knowledge that it’s noon. But these exceptions don’t help our generals, who didn’t have the foresight to make a plan like, “attack during the next thunderstorm,” or “attack at noon.” So they’re hosed.

Disappointingly, the only correct protocol for coordinated attack in an asynchronous environment is: “Never attack.”

What if message delay is bounded?

So we proved common knowledge is unachievable by passing messages if their delay is unbounded. What if it’s bounded: what if message delivery is guaranteed with a maximum delay of (Greek letter epsilon), and this maximum is common knowledge?

Let’s say both generals have reliable timers, but not necessarily synchronized clocks. General A sends a message m like “let’s attack at noon tomorrow,” and is much shorter than the time between now and noon tomorrow. The messenger leaves General A’s camp at start time , a time known only to General A. Obviously, A immediately knows he sent m:

But A doesn’t know how much delay his message will have. Once the current time is , he knows his message has been delivered, so he knows B knows he sent it:

General B receives the message at delivery time , a time known only to her. She doesn’t know how long m was delayed, so she doesn’t know , so she doesn’t know when A will know she got m. In the worst case, the delay was but she worries it was zero, so she waits until , which is . General A can thus reason that B knows A knows B got m after a delay. In other words:

But General B isn’t sure the current time is at least until (in the worst case) it’s actually , so she has to wait another epsilon before she knows . General A knows all this about General B, so now is true. We can infer that for :

We proved earlier that the generals can’t attack together unless their plan is common knowledge, which means , but that would take infinitely long. So not even guaranteed message delivery with bounded delay is enough to solve coordinated attack.

Another indistinguishability graph

Let’s look at this from another angle, with the indistinguishability graph technique from the previous article.

Here’s a state graph, with timestamps in blue going down the left side. The black arrows are possible state transitions. In the initial state, m is unsent. In the next state, General A sends it. Then there’s a branch: m is received after zero delay or delay. Then the two branches continue separately, making a state transition after each of time passes. The first two states are connected with a green line because they’re indistinguishable to General B: she doesn’t know if A sent the message or not. After that, there are two possible states at each timestamp, and these two states are connected with red lines because they’re indistinguishable to General A: he doesn’t know if his message was received after zero delay or delay. The diagonal green lines indicate the same thing about General B: she doesn’t know if she received the message after a delay or not. (My drawing is a hybrid of a state graph and a Kripke structure.)

Like we saw in the last article, we can use properties of this graph to calculate agents’ knowledge. For example, is true in state s iff all paths from s that follow zero or one A-indistinguishable edges, then zero or one B-indistinguishable edges, end at a state where is true.

Let’s choose a particular state as a demonstration: the state where the current time is and B received m with zero delay. We can follow one A-edge, then zero B-edges, and arrive at the state labeled “sent”, where is true. Or we can follow the same one A-edge, and then one B-edge and arrive at the state labeled “unsent,” where is false. Since both “sent” and “unsent” are reachable from the current state, is false.

is false in the state marked “current state.”

But if you start from either of the states where the current time is and follow zero or one red edges, then zero or one green edges, you always arrive at states where m was sent. Thus is true in those states.

is true in the state marked “current state.”

But if you want a state where , you need to start farther away, at one of the states where the time is or later. Otherwise there exists a path that follows a red, then green, then red, then green edge and arrives at a state where is false. (I haven’t drawn a picture of this.) Continuing inductively, we can see how the graph expresses the same thing we proved before: is true only if the current time is or later.

What if the generals have synchronized clocks?

It seems weird to conclude that the generals can’t coordinate, even with bounded message delays. Isn’t it enough for A to say, “attack at noon tomorrow” at least before that time, with no acknowledgment from B? Yes, this works, but the protocol itself must be common knowledge at the start, and the generals must have perfectly synchronized clocks. If the generals have already agreed that A will send a message proposing an attack at least in the future, then when B receives “attack at noon tomorrow,” she knows:

General A sent this at least before noon (that’s the protocol)
Therefore at noon, A knows B received it
Therefore at noon, B knows A knows B received it
And so on…

At noon common knowledge is achieved, because both generals can reason about what the other knows will be true at the deadline.

Halpern and Moses don’t propose this exact idea. They instead use timestamped messages: “The time is now , let’s attack at noon.” This removes General B’s uncertainty about , so at time the plan becomes common knowledge. This is pretty much equivalent: my pre-agreed protocol and their timestamped messages both provide extra shared facts from which the two generals can reason.

Both solutions require perfectly synchronized clocks. If A and B have the slightest uncertainty about when noon arrives, they can’t become sure of the plan simultaneously, so they can never achieve common knowledge.

Weaker forms of common knowledge

This still seems weird. In practice, I can arrange to meet someone for lunch, and a Raft system can reach consensus, so why is it impossible in theory? It’s because Halpern and Moses’s definition of common knowledge is impractically strong. So they define weaker forms of shared knowledge that are achievable and suffice for consensus or lunch.

-common knowledge ( ): Instead of becoming common knowledge simultaneously, becomes “ -common knowledge” in an time window. So if message delivery is guaranteed after some bounded delay, its contents are common knowledge a bounded time after it’s sent.

Timestamped common knowledge ( ): Everyone knows at time T on their own clock. If clocks are perfectly synchronized, then is equivalent to common knowledge. If they’re synchronized within , then is . You can discard clocks and substitute something like a Raft term number for the timestamp and say, “which node is the leader in term t is timestamped common knowledge.”

Eventual common knowledge ( ): Everyone will eventually know, though not necessarily at the same time. This suffices for actions that must eventually happen everywhere, without a deadline.

These form a hierarchy: .

Internal knowledge consistency

Another solution to the paradox—common knowledge is impossible in theory but possible in practice—is “internal knowledge consistency.”

Every agent in the system has an interpretation of its observations. Formally, an interpretation is a function that takes the agent’s personal history and the current time, and outputs a set of facts the agent believes are true. For example, in the muddy children puzzle, after the children hear the father’s announcement m, “at least one of you is muddy,” each child believes it heard m. An interpretation is “knowledge consistent” if it produces only true beliefs.

A knowledge consistent interpretation must say that m isn’t common knowledge, because in reality, the children can’t achieve common knowledge of m: How do they all know that they all understand English, and they were all paying attention, etc.? But an internally knowledge consistent interpretation can say it is, so long as no observation will ever contradict this belief. This matches how we intuitively reason about knowledge and common knowledge in everyday situations.

This concludes my review of Halpern and Moses’s “Knowledge and Common Knowledge in a Distributed Environment”! But I’ll have more to say about epistemic logic and distributed systems. I’m now in a UCLA seminar with Remy Wang, reading Reasoning About Knowledge, the short book Halpern and Moses wrote with Ronald Fagin and Moshe Vardi. My goal now is to marry epistemic logic with TLA+, by analyzing a TLA+ specification’s state graph.

Images:

Review: Rethink the Linearizability Constraints of Raft for Distributed Systems

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Fri, 26 Dec 2025 23:04:21 +0000

In Rethink the Linearizability Constraints of Raft for Distributed Systems (behind the IEEE paywall, dammit), some academic researchers describe Raft optimizations that make reads and writes quicker, while preserving linearizability. Raft and linearizability are my specialities, and I’m pretty sure I see mistakes in this paper. The ideas are worth considering anyway. I recommend it, if you or a friend has an IEEE subscription.

Lower-latency writes

Classic Raft

Here’s how writes work in classic Raft, as described by Diego Ongaro in his paper and thesis:

A client sends a write command to the leader.
The leader appends a log entry, which includes the command, to its log at position i.
The leader waits for the entry to be replicated by a majority of followers, i.e. committed.
The leader advances its commitIndex to i.
The leader applies the command to its copy of the state machine.
The leader advances its lastApplied index to i.
The leader replies to the client with the command’s result.

The commitIndex and lastApplied variables are separate for concurrency’s sake. One thread is responsible for communicating with followers and advancing the commitIndex. Another thread applies committed commands and advances lastApplied. That’s how Ongaro built his reference implementation, LogCabin, and it’s a good idea for any multithreaded Raft.

Commit Return optimization

In the “Rethink” paper, the authors propose that the leader replies to the client as soon as the entry is committed, without waiting until it’s applied. They call this optimization “Commit Return.” It works for blind writes, where the client doesn’t need to know anything about the command’s result besides “it was committed.” E.g., if the state machine is a key-value store and the client sends a command like set x := 1, the leader could reply “ok” before it updates its state machine.

Commit Return doesn’t work when the client sends a command that returns a data-dependent value, like return ++x. The leader must apply the command before it knows what result to return to the client. (Can the leader know at commit-time what the return value of ++x will be? No, there might be another committed and unapplied command in the pipeline that modifies x before ++x is applied.)

I’m skeptical

The authors claim that Commit Return is a big win, because sometimes applying a command takes much longer than committing it. This is surprising to me, because like most distributed systems people I assume that network latency dominates, so committing takes most of the time. The authors argue the opposite, and they construct a benchmark where this is true, with five nodes in a single data center connected by a high-speed network. The authors’ Figure 4 shows the percentage of time the leader spends appending entries to its log, or communicating with followers, or applying commands. With 1-kilobyte commands, 40% of write command latency is attributed to applying the command:

Figure 4 in the “Rethink” paper.

I admit that applying commands could be slower than majority-replicating them in this situation. But usually, you deploy a system across multiple availability zones—if you’re going to the trouble of replicating, you might as well be truly fault tolerant. Across AZs, you’re likely to see at least 1 ms of latency. And if you care about latency, then your application is OLTP, and your commands are probably mostly much smaller than 1 kilobyte and very quick to apply.

Oddly, I see that applying takes a larger percentage of the time for smaller commands in their benchmark. I guess there’s a lot of fixed overhead per command when their system applies commands, so small commands replicate quickly and take a comparatively long time to apply. The authors say, “Apply is slow because the apply operation involves writing the state machine log to disk.” But the authors separately measure the time spent appending to the Raft log. I wonder what additional logging is part of applying in their system?

Commit Return is a useful idea, regardless. In a more normal multi-AZ deployment, where most of the latency is the network’s fault, Commit Return reduces latency a smidge for blind writes, from the client’s perspective. The bulk of the paper isn’t about Commit Return, anyway: it’s about read optimizations that are independent of Commit Return. That part of the paper is more interesting, and definitely has a mistake!

Lower-latency reads

Classic Raft

Here’s how reads work in Raft:

A client sends a query to the leader.
The leader sets the query’s readIndex to the leader’s current commitIndex.
The leader waits until its lastApplied ≥ the query’s readIndex.
The leader runs the client’s query and returns the result.

This ensures the client sees the effect of all writes that were committed before the query started.

But that’s not enough to guarantee linearizability! The problem is, a Raft leader is never sure that it’s the real leader, or if it’s been deposed by a newer one. If it’s deposed, then the newer leader is executing writes that the deposed leader doesn’t see, and the deposed leader’s reads will reveal stale data, violating linearizability.

How does a Raft leader ensure it isn’t deposed? In classic Raft, Ongaro recommends that for each read, the leader sends a message (an empty AppendEntries message specifically) to all its followers, checking that they haven’t seen a newer leader. The leader waits for a quorum of followers to reply before it runs the client’s query.

This is expensive, obviously—reads now pay the same network cost as writes—so commercial Raft implementations often use timed leader leases instead, or else they just don’t guarantee linearizability.

Unfortunately, the “Rethink” authors don’t seem to know about the deposed-leader problem at all. They claim to guarantee linearizability, but they don’t mention which, if any, technique they use to prevent reading from a deposed leader. I’ll come back to this topic and see if the paper can be saved.

Read Acceleration

As I said, in classic Raft, the leader sets a query’s read index to its commitIndex, then waits for lastApplied to reach the read index. The “Rethink” authors want to reduce this waiting period. Their ideas for this are the best part of the paper.

Immediate Read: The leader immediately runs the client’s read, and buffers the result. Then it applies the log entries between lastApplied and the read index to the result, and returns the updated result to the client. This is faster than just waiting until the applier thread applies these entries to the whole state machine, as in classic Raft, because the buffered result is small, and because the leader can ignore entries irrelevant to this result.

Figure 9 from the paper.

For example, let’s say x is zero in the current state machine, and the client sends get x to the leader. The leader has these committed and unapplied log entries:

x := 9
z := 1
z := 7

The leader runs get x on the current state machine and creates the buffered result, x == 0. It applies x := 9 to the buffered result before returning it, and ignores the entries that modify z.

Semantic-Influencing Request: Sometimes, the Immediate Read fixup process is impossible. Let’s say the client sends a SQL query like:

SELECT * FROM table WHERE x = 1

The leader runs this query and buffers the resulting rows. Now it looks in its log and sees a committed and unapplied log entry like:

UPDATE table SET x = x + 1

The buffered result has the wrong rows! It includes all the rows where x was 1 when the leader received the query, but these are not the rows where x is 1 after running UPDATE. The leader can’t fix this up as easily as the previous example. The authors call this UPDATE a “Semantic-Influencing Request.” If such a request appears in the pending log entries, the leader could try something fancy like editing the original query before running it:

-- Change the filter from x = 1 to x = 0.
SELECT * FROM table WHERE x = 0

This query selects rows that will match the client’s query, after the UPDATE is applied. The authors say the leader should estimate the cost of fixing up the query, versus waiting until it can be run normally, and act accordingly.

Interaction of read and write optimizations

The authors say that Commit Return speeds up writes, but may slow down reads in exchange, because faster writes lead to a larger gap between the leader’s commitIndex and lastApplied. They even have a chart:

The red bars are the percentage of read latency in classic Raft that’s due to waiting for lastApplied to catch up to the query’s readIndex (which was the commitIndex when the query arrived). The higher blue bars are the percentage with Commit Return. The blue/red line shows the average gap between the readIndex and lastApplied with/without Commit Return, measured in number of entries. The horizontal axis is the percentage of writes in the YCSB workload, from 90% writes on the left to 10% writes on the right.

The authors claim this chart shows that with Commit Return enabled, there is a bigger gap between the leader’s commitIndex and lastApplied. This is an unfortunate mistake. I’m certain that Commit Return has no effect on the gap or on query latency. The authors were led astray by YCSB.

Commit Return merely reduces write latency from the client’s perspective: if the client does a blind write, the server replies “ok” a bit sooner, and the client can move on to its next task. If the client is a closed loop, then it waits for each request to finish before it starts the next. Let’s say this is the client’s code:

while True:
    result1 = do_request("set x := x + 1")
    result2 = do_request("get x")

In classic Raft, set x := x + 1 doesn’t return until the leader applies it, so the leader can run get x without much waiting. (If other clients submitted writes that are committed and unapplied when get x arrives; the leader still has to wait until those entries are applied before running get x.)

With Commit Return, set x := x + 1 returns as soon as it’s committed. Thus when the client sends get x, the gap between the leader’s commitIndex and its lastApplied is bigger than without Commit Return.

It’s important to understand that Commit Return doesn’t delay the advancement of lastApplied, so it doesn’t increase the gap between commitIndex and lastApplied in general! It just returns control to the client sooner, so the client can send a higher throughput of blind writes, or start querying sooner after sending a blind write. This only matters if you’re benchmarking the system with a closed-loop workload generator like YCSB. YCSB lets the server exert backpressure on the client, so it doesn’t measure throughput or latency realistically. (This is called the coordinated omission problem.)

A closed system.

With a more true-to-life open-loop workload generator, Commit Return does not have this effect. The authors’ claim that Commit Return makes Read Acceleration more important is a misunderstanding based on this obsolete benchmark.

An open system.

Tragically, lots of distributed systems researchers still use YCSB, so I see this mistake all the time. Worse, I see reviewers, who should know better, requesting that authors add YCSB benchmarks to their evaluations. YCSB benchmarks are wrong. If you’re a researcher, please switch to an open-loop workload generator, or invent one for the rest of us to use. If you’re a reviewer, point out this mistake when you see it. Let’s all standardize on some modern, open-loop benchmarks.

What about deposed leaders?

Napolean, about to resign

The authors’ optimizations reduce latency and maintain linearizability, if the leader isn’t deposed. How do they guarantee that it isn’t?

If the leader checks a quorum of followers for each query, as in classic Raft, that gives the leader more time to concurrently apply any pending writes. It’s still possible that lastApplied hasn’t caught up to the readIndex during the quorum check—the authors constructed a scenario where applying commands is slow and communication is quick—but Read Acceleration is less useful once we take the quorum check into account.

The better way to handle a deposed leader is with a timed lease. I think the Read Acceleration techniques could be useful when paired with leases. I just hope anyone who reads this paper keeps the deposed leader problem in mind.

Conclusion

This paper is a worthwhile read, despite its flaws.

Con: I doubt that it often takes longer to apply commands than replicate them, as the authors claim.

Pro: Even so, the Commit Return optimization might be a useful idea sometimes.

Pro and con?: Commit Return doesn’t have the downside the authors think it has: they think it makes reads slower, but that’s only in the artificial environment of a YCSB benchmark. Commit Return is better than they realize.

Con: The authors’ protocol does not guarantee linearizability unless they deal with deposed leaders somehow, like with a lease.

Pro: Read Acceleration is a cool idea. It’s useful if there’s a big gap between commitIndex and lastApplied. I really don’t think this is a common scenario, but the idea has other uses. I just published a paper in SIGMOD, with my colleagues Murat Demirbas and Lingzhi Deng, that uses a similar mechanism to permit linearizable reads when there might be multiple leaders.

Images:

YCSB Is Obsolete, We Need New Benchmarks

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 21 Dec 2025 17:44:18 +0000

Let’s build a simulation of a coffeeshop. We’ll have some resources, like baristas and espresso machines. As soon as the shop opens, a fixed number of patrons—maybe ten—all line up for coffee. Whenever a patron receives her coffee, she throws it away and instantly returns to the back of the line. If the barista is slow, that’s okay: the line never grows longer than ten people.

A closed loop coffeeshop.

Oh, that’s not realistic, you say? You say that patrons arrive at random times, join the line, and leave when they get their coffee?

An open loop coffeeshop.

I agree, that is a better model of a coffeeshop. As you have guessed, the coffeeshop is a metaphor, and my actual topic is cloud databases. In the cloud, there’s no fixed number of clients. New requests are mostly triggered by external events, not by the completion of previous requests. If the database can’t keep up, requests continue to arrive and pile up or time out. But standard benchmarks like YCSB or TPC assume the first model, the unrealistic one, so they give unrealistic results.

The Difference Between Open and Closed

The first model is an example of a “closed system model” or a “closed loop”: there is a fixed number of outstanding tasks. Once the database server is saturated, a new task won’t arrive until one of the outstanding tasks finishes. The second is an “open system model” or “open loop”: new tasks arrive whenever they want to, independent of when outstanding tasks finish. In an open system, the interarrival time between tasks is usually randomly distributed. If the distribution is exponential, then the arrivals are a Poisson process.

The real world is open, but we stubbornly keep using closed loop benchmarks to test our databases!

A closed loop benchmark generates load like this:

def client_func():
    for _ in range(OPERATION_COUNT):
        start = time()
        response = do_database_request()
        end = time()
        record_latency(end - start)
        sleep(INTERARRIVAL)

threads = [Thread(target=client_func) for _ in range(THREAD_COUNT)]
for t in threads:
    t.start()
for t in threads:
    t.join()

The latency of do_database_request affects how much load the benchmark generates—the database implicitly exerts backpressure. The longer requests take, the longer the time between requests.

An open loop benchmark generates load like this:

def one_operation():
    start = time()
    do_database_request()
    end = time()
    record_latency(end - start)

for _ in range(OPERATION_COUNT):
    Thread(target=one_operation, daemon=True).start()
    sleep(INTERARRIVAL)

This code starts a new request whenever the interarrival time has passed, no matter how many requests are still outstanding. For a Poisson process, you just make the interarrival time exponentially distributed:

LAMBDA = 1000.0  # ops per second
next_time = time()
for _ in range(OPERATION_COUNT):
    next_time += random.expovariate(LAMBDA)
    Thread(target=one_operation, daemon=True).start()
    time.sleep(max(0, next_time - time()))

Is this any harder to code than the closed loop version? In 2025 we are all familiar with multithreaded or (better in this case) async programming, we can build open loop benchmarks easily. The time has come for us all to switch to open loop benchmarks.

Years Of Warnings Ignored

In the 2006 paper Open Versus Closed: A Cautionary Tale, some researchers (including Mor Harchol-Balter) benchmarked a Postgres server with TPC-W, the web commerce benchmark that’s part of the TPC suite. Like all the TPC benchmarks, TPC-W is closed loop, but the researchers also made a version that’s open loop. They compared the standard TPC-W results with their open loop version:

The closed loop benchmark can never overload the database, because it has a fixed number of threads that each wait politely for the database to respond before sending the next request. Mean response time never exceeds 2 seconds. The open loop benchmark is rude: it overwhelms the database, unfinished requests pile up and wait in the queue. Mean response time exceeds 6 seconds. This is the kind of realistic data we need to see when we test our databases, but the standard closed loop TPC won’t show it to us.

(The PELJF, PS, and PESJF lines are for different scheduling algorithms—that’s an interesting part of the paper, you should read it. Or read Murat’s summary.)

The authors list eight insightful principles about the behavior of closed and open systems. The very first is, “For a given load, mean response times are significantly lower in closed systems than in open systems.” In other words, TPC is lying about cloud database performance.

This paper should’ve set the world on fire, but years passed and we all kept creating and using closed loop benchmarks. The Yahoo! Cloud Serving Benchmark was released in 2010. It was a great innovation because, unlike the TPC family, it targeted the new generation of distributed key-value stores. Unfortunately, you guessed it: YCSB is a closed loop benchmark.

Gil Tene coined the term “coordinated omission” around 2012. It describes how closed loop benchmarks interact with the systems they’re testing to avoid measuring their behavior during overload. He said it’s “a conspiracy we’re all a part of.”

In 2017, some University of Hamburg researchers published Coordinated Omission in NoSQL Database Benchmarking, criticizing YCSB and introducing their own open loop benchmark, NoSQLMark. They point out that YCSB tried to fix the problem in 2015, but the fix is a flawed hack that doesn’t really make YCSB a reliable benchmark. (See the paper for details, or just trust me—it’s a flawed hack.) The authors’ NoSQLMark does the obvious right thing: it uses an async framework, Akka. Requests’ start times are independent of their latency, just like in the real world.

The NoSQLMark paper uses a slightly weird methodology: they try to demonstrate their benchmark’s accuracy using a partially simulated database called SickStore, which doesn’t fully prove NoSQLMark’s value in my opinion. But they also benchmark Cassandra with both YCSB and NoSQLMark and show how the results differ. This is actual science and we should all emulate it.

In 2024—look, I know you’re getting sad and tired. I’m sorry. I’m getting sad and tired too. We’re almost done. In 2024 some researchers announced KVBench. This solved one problem with YCSB: it doesn’t generate diverse workloads, like a mix of point and range queries, queries with empty results, or deletes. KVBench is a good contribution, but once again, it’s a closed loop. It can’t push a database past its limits.

Aside from research papers, some widely read bloggers have warned about coordinated omission. Ivan Prisyazhynyy wrote about it in 2021 on the ScyllaDB blog. In 2023, my colleague Murat Demirbas reviewed the Cautionary Tale paper. This year, Marc Brooker wrote that “most cloud systems are open, most benchmarks are closed.” He says that closed loop benchmarks are far too kind to systems: they reduce their load automatically when the system slows down. “The real world isn’t that kind to systems. In most cases, if you slow down, you just have more work to be done later.”

It’s Time To Take A Stand

Joan of Arc by Edward Corbould, 1890

Why do we persist in using closed loop benchmarks, even though their inaccuracy is now well-understood, and open loop alternatives exist, and async frameworks make new ones easy to write?

It reminds me of Barbara Tuchman’s diagnosis of self-defeating nations:

Wooden-headedness, the source of self-deception, is a factor that plays a remarkably large role in government. It consists in assessing a situation in terms of preconceived fixed notions while ignoring or rejecting any contrary signs. It is acting according to wish while not allowing oneself to be deflected by the facts.

Part of the problem is when peer reviewers, out of habit, ask authors to use closed loop benchmarks. A few months ago I took a small risk. My colleagues and I had submitted our paper about a Raft enhancement to a major conference, and one of the reviewers asked for some more benchmarks. They suggested YCSB or KVBench. We said “sure”—we wanted to publish the paper, and the reviewers have the final say. But then we reflected on these years of warnings about closed loop benchmarks, and we noticed that YCSB and KVBench are closed, and we changed our minds. I wrote a custom open loop benchmark in C++ using nonblocking I/O. When we submitted our revision, I explained why we didn’t follow the revision plan. The sky didn’t fall. Our revision was accepted, and it was better because it used an accurate benchmark.

I understand why reviewers ask for YCSB or KVBench or TPC, despite those benchmarks’ inaccuracy: they’re the only baseline we have. Researchers want to do science, and using established benchmarks is part of that. But established benchmarks don’t promote scientific progress if they’re wrong. The solution is to create new baselines with open loop benchmarks. There are a few good candidates. NoSQLMark is a natural successor to YCSB: it’s the same thing but open loop. Lancet is an interesting research project to measure the open loop tail latency of very fast systems, using clever sampling techniques. And someone should make a standard open loop version of the TPC suite, if that doesn’t exist already.

Let’s start a movement. As authors let’s stop using closed-loop benchmarks, and as reviewers let’s stop asking for them. Let’s pledge that from now on, we will use open loop benchmarks to test cloud databases.

LeaseGuard: Raft Leases Done Right!

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 18 Dec 2025 23:27:58 +0000

Many distributed systems have a leader-based consensus protocol at their heart. The protocol elects one server as the “leader” who receives all writes. The other servers are “followers,” hot standbys who replicate the leader’s data changes. Paxos and Raft are the most famous leader-based consensus protocols.

These protocols ensure consistent state machine replication, but reads are still tricky. Imagine a new leader is elected, while the previous leader thinks it’s still in charge. A client might write to , then read stale data from , violating Read Your Writes. How can we prevent stale reads? The original Raft paper recommended that the leader communicate with a majority of followers before each read, to confirm it’s the real leader. This guarantees Read Your Writes but it’s slow and expensive.

A leader lease is an agreement among a majority of servers that one server will be the only leader for a certain time. This means the leader can run queries without communicating with the followers, and still ensure Read Your Writes. The original description of Raft included a lease protocol that was inherited from the earlier Paxos, where followers refuse to vote for a new leader until the old leader’s lease expires. This entangles leases and elections, and it delays recovery after a crash. Besides, lease protocols have never been specified in detail, for either Raft or Paxos. For all these reasons, many Raft implementations don’t use leases at all, or their leases are buggy.

In the MongoDB Distributed Systems Research Group, we designed a simple lease protocol tailored for Raft, called LeaseGuard. Our main innovation is to rely on Raft-specific guarantees to design a simpler lease protocol that recovers faster from a leader crash.

Here’s a preprint of our SIGMOD ’26 paper. This is a joint blog post by Murat Demirbas and me, published on both of our blogs.

A huge simplification: the log is the lease

In Raft, before the leader executes a write command, it wraps the command in a log entry which it appends to its log. Followers replicate the entry by appending it to their logs. Once an entry is in a majority of servers’ logs, it is committed. Raft’s Leader Completeness property guarantees that any newly elected leader has all committed entries from previous leaders. Raft enforces this during elections: a server votes only for a candidate whose log is at least as up to date as its own. (Paxos doesn’t have this property, so a new leader has to fetch entries from followers before it’s fully functional.)

When designing LeaseGuard, we used Leader Completeness to radically simplify the lease protocol. LeaseGuard does not use extra messages or variables for lease management, and does not interfere with voting or elections.

In LeaseGuard, the log is the lease. Committing a log entry grants the leader a lease that lasts until a timeout expires. While the lease is valid, the leader can serve consistent reads locally. Because of Leader Completeness, any future leader is guaranteed to have that same entry in its log. When a new leader is elected, it checks its own log for the previous leader ’s last entry, to infer how long to wait for ’s lease to expire.

In existing protocols, the log is not the lease: instead, the leader periodically sends a message to followers which says, “I still have the lease.” But imagine a leader who couldn’t execute writes or append to its log—perhaps it’s overloaded, or its disk is full or faulty—but still has enough juice to send lease-extension messages. This lame-duck leader could lock up the whole system. In LeaseGuard, a leader maintains its lease only if it can make progress; otherwise, the followers elect a new one.

We’re excited by the simplicity of this Raft-specific lease protocol. (We were inspired by some prior work, especially this forum post from Archie Cobbs.) In LeaseGuard, there is no separate code path to establish the lease. We decouple leases from elections. The log is the single source of truth for both replication and leasing.

LeaseGuard makes leader failovers smoother and faster

Leases improve read consistency but can slow recovery after a leader crash. No matter how quickly the surviving servers elect a new leader, it has to wait for the old leader’s lease to expire before it can read or write. The system stalls as long as 10 seconds in one of the Raft implementations we studied.

LeaseGuard improves the situation in two ways. First, deferred-commit writes. As soon as a new leader wins election, it starts accepting writes and replicating them to followers. It just defers marking any writes “committed” until the old lease expires. Without this optimization, writes enqueue at the new leader until the old lease expires; then there’s a thundering herd. With our optimization, the new leader keeps up with the write load even while it’s waiting.

Second, inherited lease reads. This is our biggest innovation, and it’s a bit complicated. Consider the situation where was just elected, but is alive and still thinks it’s in charge. Neither leader knows about the other. (Yes, this can really happen during a network partition.) Raft makes sure that can’t commit any more writes, but there’s a danger of it serving stale reads. The whole point of leases is to prevent that, by blocking from reading and writing until ’s lease expires. But what if there was a way for both leaders to serve reads, and still guarantee Read Your Writes?

When was elected, it already had all of ’s committed log entries (Leader Completeness), and maybe some newer entries from that aren’t committed yet. L1 knows it has every committed entry, but it doesn’t know which ones are committed! We call these ambiguous entries the limbo region. For each query, checks if the result is affected by any entries in the limbo region—if not, just runs the query normally. Otherwise, it waits until the ambiguity is resolved. (My MongoDB colleague Lingzhi Deng is the one who saw the need for this check, which is how he became a co-author.)

Logs on the old and new leader. Entries 1-5 were committed by L₀ , and L₁ has them all, but it only knows that 1-3 are committed. It may not learn whether 4-6 are committed until it tries to commit an entry of its own.

Inherited lease reads require synchronized clocks with known error bounds, but the rest of the protocol only needs local timers with bounded drift. Our two optimizations preserve Read Your Writes and dramatically improve availability.

Transitions in the read/write availability of leaders with LeaseGuard. While the new leader waits for a lease, it can serve some consistent reads and accept writes. Meanwhile the old leader serves reads.

Here’s the whole algorithm in pseudo-Python. For more details, read the paper.


# Handle a write request from a client.
def ClientWrite(command):
    if self.state != "leader": return "not leader"
    # Create new entry, log it and record its index.
    entry = (self.term, command, intervalNow())
    index = self.log.append(entry)
    # Another thread replicates, commits, and applies the
    # command, and advances lastApplied, see CommitEntry.
    await(self.lastApplied >= index)
    if self.state != "leader":
        # Deposed, don't know if command succeeded.
        return "not leader"
    return "ok"


# Handle a read request from a client for key k.
def ClientRead(k):
    if self.state != "leader": return "not leader"
    # Last committed entry's age is calculated using
    # bounded-uncertainty clock.
    if self.log[self.commitIndex].age > delta:
        return "no lease"
    # Prevent "limbo" reads.
    if self.term != self.log[self.commitIndex].term:
        if any limbo region entry affects k:
            return "key affected by limbo region"
    return self.data[k]


# When this node learns some followers have replicated
# entries up to index i, advance the commitIndex.
def CommitEntry(i):
    if self.state != "leader": return
    if not majorityAcknowledged(self.log[i]):
        return
    # Check for past-term entry < Delta old.
    # In reality this loop is optimized away, Sec. 7.
    for e in self.log:
        if e.term < self.term and e.age < delta:
            return
    self.commitIndex = max(self.commitIndex, i)
    while self.lastApplied<self.commitIndex:
        apply(self.log[self.lastApplied+1].command)
        self.lastApplied += 1

Tests and benchmarks

When we started this research, our main goal was to publish a detailed and correct specification, so Raft implementers everywhere could implement leases without bugs. We’re TLA+ fans so obviously we specified the algorithm in TLA+ and checked it guaranteed Read Your Writes and other correctness properties. We discovered our two optimizations while writing the TLA+ spec. The inherited lease reads optimization was especially surprising to us; we probably wouldn’t have realized it was possible if TLA+ wasn’t helping us think.

We implemented the algorithm in LogCabin, the C++ reference implementation of Raft. (For ease of exposition, we also provide an implementation in a Python simulator.)

In the following experiment, we illustrate how LeaseGuard improves throughput and reduces time to recovery. We crash the leader 500 ms after the test begins. At the 1000 ms mark, a new leader is elected, and at 1500 ms, the old leader’s lease expires. We ran this experiment with LogCabin in five configurations:

Inconsistent: LogCabin running fast and loose, with no guarantee of Read Your Writes.
Quorum: The default Read Your Writes mechanism, where the leader talks to a majority of followers before running each query, is miserably slow—notice that its Y axis is one tenth as high as the other charts!
Lease: The “log is the lease” protocol with no optimizations. Its throughput is as high as “inconsistent”, but it has a long time to recovery after the old leader crashes.
Defer commit: The log is the lease, plus our write optimization—you can see that write throughput spikes off the chart at 1500 ms, because the leader has been processing writes while waiting for the lease. As soon as it gets the lease, it commits all the writes at once.
Inherit lease: LeaseGuard with all our optimizations. Read throughput recovers as soon as a new leader is elected, without waiting for the old lease to expire.

How fast does the system recover after a leader crash?

Conclusion

Until now, the absence of a detailed specification for Raft leases led to many flawed implementations: they often failed to guarantee consistent reads, had very low read throughput, or recovered slowly from a leader crash. With LeaseGuard now specified, implemented, and published, we hope it will be readily adopted to enable Raft systems to provide fast reads with strong consistency and recover quickly after a crash.

We learned yet again the value of TLA+ during this project. TLA+ is useful not just for checking the correctness of a completed design, but for revealing new insights while the design is in progress. Also, we got interested in reasoning about knowledge, also known as epistemic logic. In Raft, servers can look in their logs and know that other servers know certain facts. For example, if a leader has a committed entry, it knows any future leader knows about this entry, but it doesn’t know if a future leader knows the entry was committed. This is a different way for us to think about a distributed system: it’s not just a state machine, it’s a group of agents with limited knowledge. We’re curious about this way of thinking and plan to do more research.

Image: Horlogerie, 1765, Diderot and d’Alembert.

The Work of Art in the Age of Mechanical Creation

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 10 Dec 2025 18:55:37 +0000

Portrait of Paul Valery by Jacques-Emile Blanche

Back in 1928, Paul Valéry predicted streaming media:

It will be possible to send anywhere or to re-create anywhere a system of sensations, or more precisely a system of stimuli, provoked by some object or event in any given place. Works of art will acquire a kind of ubiquity. We shall only have to summon them and there they will be, either in their living actuality or restored from the past. They will not merely exist in themselves but will exist wherever someone with a certain apparatus happens to be. A work of art will cease to be anything more than a kind of source or point of origin whose benefit will be available and quite fully so, wherever we wish. Just as water, gas, and electricity are brought into our houses from far off to satisfy our needs in response to a minimal effort, so we shall be supplied with visual or auditory images, which will appear and disappear at a simple movement of the hand, hardly more than a sign. Just as we are accustomed, if not enslaved, to the various forms of energy that pour into our homes, we shall find it perfectly natural to receive the ultrarapid variations or oscillations that our sense organs gather in and integrate to form all we know. I do not know whether a philosopher has ever dreamed of a company engaged in the home delivery of Sensory Reality.

Paul Valéry, The Conquest of Ubiquity (La conquête de l’ubiquité)

Walter Benjamin followed up in 1935, wondering how our relationship with art will change once it’s so easily copied. In The Work of Art in the Age of Mechanical Reproduction, he analyzed what had made a work of art special in the past, and how that was changing in the 20^th century, and he forecast that original art objects would lose their value. The essay is world-famous, but his forecast was wrong: originals are now more valued than ever. Now that we’re approaching the age of mechanical creativity, I’ll try to learn from Benjamin’s error and make the opposite kind of forecast: I predict that AI art will make human art even more precious.

Walter Benjamin

What’s special about an original painting or sculpture? Benjamin says it is the object’s “aura”, which he defines in a few ways. The aura is “the unique phenomenon of a distance, however close it may be.” That is, you cannot possess or control the object, it is distant and “unapproachable.” Its aura is also its authenticity; Benjamin says, “The authenticity of a thing is the essence of all that is transmissible from its beginning, ranging from its substantive duration to its testimony to the history which it has experienced.” Maybe this is easier to grasp in the original German? I’d redefine aura as, “The special value we assign to an art object because of its unique identity.” The key is recognizing that we assign this special value. A perfect copy of the Mona Lisa produced by a Star Trek replicator, even if it were indistinguishable by carbon dating or electron microscopy, would still be a copy, valued differently from the original, only because we know which is which. If someone secretly swapped them, the aura would cling to the one we thought was original.

Benjamin says art objects first had aura because of their magical or religious meaning. I imagine Stonehenge or a statue of Athena: they are not valued for their beauty or originality, but their power. Copying was just as hard as producing an original, so the terms “original” and “copy” didn’t apply. All statues of Athena were sacred. In the secularizing Renaissance, beauty took precedence. Once the Industrial Revolution began, the Romantics responded by heroizing individual artistic creativity. As Benjamin says,

… art sensed the approaching crisis which has become evident a century later. At the time, art reacted with the doctrine of l’art pour l’art, that is, with a theology of art. This gave rise to what might be called a negative theology in the form of the idea of ‘pure’ art …

Calling it a “theology” is right: the Romantics made a real cult of “art for art’s sake” and the artistic genius. But Benjamin skips over the Romantics. He dismisses creativity and genius in a sentence as “outmoded concepts.” He thinks that the art object’s function has evolved from ritual, to exhibition, and then to politics. He’s a Marxist, after all, and he both hopes and predicts that art’s main function in the future will be making revolutionary demands. Meanwhile, he predicts the aura of the original art object will fade. “The situations into which the product of mechanical reproduction can be brought may not touch the actual work of art, yet the quality of its presence is always depreciated.” And, “that which withers in the age of mechanical reproduction is the aura of the work of art.”

The depreciated actual work of art?

Walter Benjamin was wrong. The Mona Lisa has to be one of the most reproduced artworks, but the quality of its presence has not depreciated a bit. Every year more pilgrims go to be in its presence. It’s because we can all see high-resolution color copies that the original became a cult object. It’s why we ritually gather around it, take selfies with it, make even more copies with our phones and cameras as an act of worship. Benjamin defines aura as “distance,” and the original Mona Lisa has definitely become more distant: unapproachable, roped off, guarded. Owning it is impossible to imagine—even a mediocre Leonardo sold for $450 million. In general, the more famous an object is, the more it is reproduced—fame and reproduction are nearly synonymous on the Internet—and the stronger its aura.

Maybe if Benjamin were alive today, he’d say what persists isn’t the real aura that art objects used to have; it’s just fetishism or greed. Yes, things change. The modern world is secular and scientific: I will never know what an ancient Greek felt in front of an Athena statue. And I’ll never know what it was like to see the Mona Lisa in person for the first time, before the world was tiled with copies of her. Today’s aura is different from before. But even though things change, the special value we assign to unique art objects keeps growing! More and more tourists travel to crowd around famous things, the price of famous artworks spirals upwards.

The value of an original painting or sculpture is a little beside the point, anyway. Some art forms don’t create unique objects: music, literature, photography, and movies are all exhibited by copying. Yes, there might be an original score handwritten by a composer, or an original negative shot by a photographer. They have some aura, and that’s what makes them collectors’ items, but they’re not the primary artwork. Digital photography and literature written in Microsoft Word produce no artifacts at all. For a moment, some people tried to recreate the aura of the artifact with NFTs. The moment passed, thank god, but it’s a symptom of our human need to assign aura somewhere. It turned out NFTs were unnecessary: These non-physical art forms still have aura; the aura just migrated from objects to creators.

The cult of the creative genius seemed to be the main source of aura in the 20^th century, and continues to be today. This is another way Benjamin’s forecast swerved off course. The more original and innovative the artist is, and the more they seem “pure” and uninfluenced by fame, money, or convention, the more we value their work. Think of how Picasso or Hilma af Klint are lionized, or the auteur directors of American films in the 1970s, or literary cult figures like Emily Dickinson and David Foster Wallace. “Authentic” and original musicians like Bob Dylan are celebrated ad nauseam. (Enough already, let’s worship Björk now.) Art’s aura is undiminished compared to the past—its source just changed, from religion or beauty to admiration of the artist.

(Aside: The Romantics thought of creativity as spontaneous, inspired only by the artist’s interior passion, like something from nothing. This made no sense—an artist is always determined by their environment and tradition, even if they’re rebelling against it. “Originality” is just free will, constrained to art, and it’s as nonsensical as free will in general. I think art critics now scoff at the idea of artistic originality—Dadaism wounded it and Postmodernism finished it off—but free will is still overwhelmingly popular among philosophers, which baffles me. Anyway, we don’t need to believe humans are capable of originality, or that AI isn’t—we can assign special value to human art regardless.)

What does it matter if Walter Benjamin misforecast? Yes, like most Marxists, he was too focused on economic competition among classes, he underestimated people’s attachment to ideas and rituals. It turns out people really want to treat some artworks as special, and high-fidelity copying only intensified this. We doubled down on valuing objects and artists, the opposite of what Benjamin expected. So what?

Well, now we’re entering the age of mechanical creation. We can see on the horizon AIs with an artistic creativity that matches our own. What forecast should we make about the value of human creativity in the future?

First, I have to regain your trust by showing I hate AI art as much as you do. Most of the AI art I see now is repulsive, because the AIs’ default “house styles” are tacky. If you give it a simple prompt and accept the first result, Dall-E usually makes excessively detailed airbrushed images with golden sidelight, and Gemini makes earth-toned kids’ book illustrations.

Left: art from the Dall-E home page. Right: my first Gemini image.

Even prize-winning AI art, made by experienced prompters trying their hardest, is nauseating. I hold the humans responsible: the prompters have bad taste and so do the judges.

Skilled and tasteful prompters can produce competent art that doesn’t look like AI. They make images in a variety of styles, art that fools laypeople in head-to-head contests with human art. I liked some of these images, until I found out they’re AI. So what’s that about? Why do I sometimes enjoy an image, then retch when I learn I was fooled? Why do I refuse to use image generators myself? Why do I instead spend hours searching for vintage images to illustrate my blog?

It’s because the aura of human creation is powerful. I care whether an image is created by a human, a person with a living body and a history, someone doomed to die, with limited skills and a specific intention. I care about this, even though AI art is sometimes indistinguishable to me, and I expect to be fooled much more. In fact, I think there will come an AI artist with all the creativity and intention of a Hilma af Klint: there will be a self-conscious AI artist who conceives and executes a body of work that expresses the AI’s unique experience, unprompted by a human. I might appreciate that art, knowing how it was made. But for no reason inherent in the art itself, I will always distinguish between human and mechanical creation.

So here’s my prediction: just like we still value original art objects, no matter how faithfully they can be copied, we will also value human creativity, no matter how creative AI becomes. Once we can’t tell the difference, we’ll make systems for tracking the provenance of human creations, just like we do for objects. We’ll sometimes fall for AI forgeries, and sometimes catch them. But we’ll always assign special value to art we believe was human-created. Aura is not a property of objects or of creations, it’s something we give them to fulfill our own need. Whenever aura is undermined by technological change, it doesn’t wither. We find a way to create it again.

How I Write a Related Work Section with Zotero and Overleaf

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 02 Dec 2025 18:56:30 +0000

This week I’m writing a Related Work section for a research paper in a field outside my specialty. Usually I write about distributed systems, and I already know most of the papers I need to cite on a specific topic. But now I’m targeting a software engineering conference, so I’m rolling all over the Internet like Katamari, accumulating a big ball of citations that may eventually form a bibliography.

Here’s what I’ve learned about efficiently assembling a Related Work section for a new-to-me topic. I’ll show you how to use Overleaf and Zotero, and … I’ll announce a new Zotero plugin I built which makes my job easier, and perhaps yours.

Find Papers

LLMs in Deep Research mode make a good start—this is available with a subscription to Gemini, ChatGPT, etc. I started by uploading a draft of the paper I’d written so far. A prompt like this partly corrects the LLM’s bad instincts:

I’ve uploaded a draft of a research paper I’m writing for [insert conference]. Please look for papers in peer-reviewed journals and conference proceedings on the subject of [describe the subject]. I’m looking for this because I want papers to cite in the “related work” section of this paper. Don’t tell me about the paper I uploaded, I wrote it, I already know about it. Just use it as an example of the topic I’m researching. I want you to find me research papers. They’re probably PDFs on the internet. It’s helpful to search scholar.google.com, dblp.org, or researchgate.net. If it’s not a research paper, or if I wrote it, don’t tell me about it. Find me research papers that I can cite in the research paper that I will write.

Maybe these instructions are needlessly grouchy; you can guess I’d had a few rounds of disappointing results before I wrote them.

Besides using AI, I followed my own advice and plugged keywords into Google Scholar, Arxiv.org, DBLP.org, etc. You know which sites are best for you.

Collect Papers in Zotero

Once I had opened a hundred tabs, I skimmed the papers’ abstracts and imported the promising ones into a Zotero collection with the Zotero Connector for Chrome. Zotero has a lot of great features for assembling a bibliography. One of them is, papers’ citation numbers are clickable:

This makes it easy to open a paper, skim its intro, and find its citations to earlier, more foundational papers. Eventually you find the ur-paper (or some survey paper) which is an authoritative citation on some topic. I wish Zotero also helped you find and download cited papers, but this is easy enough to do myself. (Unless you can tell me about a plugin that does this, or build one for me?)

Fill In Missing Metadata

The Zotero Connector for Chrome usually fills in a paper’s metadata—authors, date, publication, and so on—when I import a paper from the web. But sometimes a paper lands in Zotero with bad or missing info, if I downloaded it from an obscure site. This means I can’t generate a bibliographical reference for it. To my surprise, I didn’t find an automated solution, so I built a Zotero plugin over the weekend called Metadata Search. The plugin searches just CrossRef.org and DBLP.org for now (pull requests welcome), and shows a list of results.

For example, here I have a paper in Zotero, with only a title and incomplete authors list. My plugin finds more info on DBLP.org:

I check some boxes for the fields I want to update, and the plugin copies the values into Zotero.

Bibliography and Citations

Now, since I’m writing my paper with LaTeX, I need to generate a BibTeX bibliography file and cite particular entries in the text of my paper. There are two paths.

One path is to install Better BibTeX for Zotero. This has two killer features. First, it creates stable citation keys for papers. Even if I update the paper’s metadata (e.g. change its author or year), Better BibTeX ensures that the citation key doesn’t change. So if I first write a citation like \cite{2020_davis}, and then discover the paper was published in 2021, I can fix the date without breaking the cite directive. Its second killer feature is auto-export: I can tell it to export a Zotero collection as a BibTeX bibliography, and it will re-export to the same file each time the collection changes. This seems like a great labor-saving device for people writing LaTeX on their own machines.

The second path is to write an online shared doc in Overleaf. This is what I do, because it allows Google Docs-style collaboration with co-authors. Overleaf has a Zotero integration which dumps your whole Zotero catalog into the bibliography. If you have a huge catalog, like me, I recommend you create a Zotero “group library” and tell Overleaf to import only from there. Regardless, only the papers you actually cite will appear in the rendered PDF. You can refresh the Overleaf bibliography from Zotero whenever you want.

Here’s a demo of the whole workflow:

This Senior Staff Engineer Vibe-Coded for the First Time, What Happened Next Will Shock You

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 19 Nov 2025 02:53:40 +0000

Actually it won’t shock you. Spoiler alert: Coding with AI is a skill. It’s annoying but efficient. My prior experience handwriting code helped.

By Gemini, obviously. This is the first and only post I’ll ensloppify with AI “art.”

I started programming seriously when I was in college, in the late 90s. Basic autocomplete in Microsoft Visual C++ was the state of the art back then. I spent the next couple decades meticulously laying out my code like a 16th-Century typesetter, crafting each line with an artisan’s pride.

When I switched from engineering to research I stopped coding regularly. Now I have spurts of programming when I prototype a new algorithm or benchmark it, and between spurts I go months without significant time in an IDE. Last year, IDEs didn’t change by the month. This year, each month is a revolution. I wanted to experience the new reality for myself. So, for two recent personal projects, I tried going all-in on AI coding.

As I’ll describe below, my first attempt was frustrating because I didn’t embrace the vibe-coding lifestyle. I chose the wrong model, then micromanaged it, trying to make it write the code I would’ve written. The LLM implemented the wrong features, or wrote buggy code, or correct code that I thought was ugly. On my second attempt, I used the right model and treated it like an adult: I gave it high-level instructions and let it write all the code and run its own tests. I didn’t type a single character of code—I hardly even looked at it—and I had an enjoyable day building a useful app.

Act I: The Senior Staff Engineer is a Control Freak

Gemini doing a decent impression of the Blondie comic strip style.

For my first vibe-coding exercise, I turned a Python script into a web app and deployed it to the cloud. The script was something I’d developed over a few years to optimize the schedule for a twice-annual Zen retreat. My fellow Zen students suggested I turn it into an application they could use without me. I thought this was a perfect use case for AI. I opened VS Code and accepted the default model, GPT-5 mini, and conversed with it about my needs.

The model started off strong: it made useful suggestions for deploying to Google cloud. It proposed a mix of Google services that would save money and pay only for the resources I needed. It insisted I needed some complex web of Google accounts, roles, and auth tokens—I kept pushing back and asking it to simplify, but in the end it was right. I eventually followed all its instructions, after discovering there was no simpler alternative.

When it came time to write code, GPT-5 mini wrote a good first web page and Flask backend. It was much quicker than my own work would’ve been, and the web page was nicely styled.

I didn’t care about the code style of the generated HTML, CSS, and JavaScript, but I’m a stickler for Python style and I made the mistake of reading its code. I hated it.

GPT-5 mini really likes to write useless comments:

# Return the square of x.
def square(x: int) -> int:
    ...

It also likes to write “defensive” code that only makes the program less predictable:

# Return the square of x.
def square(x: int) -> int:
    if x is None:
        print("x wasn't set")
        return 0

    return x * x

I griped in copilot-instructions.md that it shouldn’t do this, but it seemed to have no effect. (Edit: that’s because VS Code agent instructions should be in .github/instructions. When I first booted the agent it suggested that we create copilot-instructions.md so I figured wrongly that it knew where its own instructions file belongs.) So I started including in my prompts, “Just write the code, no comments, no error-checking.” Hilariously, it then wrote comments like:

def square(x: int) -> int:
    # Don't check whether x is None, just let the exception be thrown.
    return x * x

This is high-school-sophomore-style coding. Someone who’s worked on large codebases for years knows that comments are tech debt and should be minimized. They should be addressed to a reader in the far future, not to the most recent code reviewer. Defensive coding like if x is None makes the code more complex, not more resilient: error-handling should be consolidated at a layer of the application that knows how to respond. I’m afraid that GPT has either seen too much sophomoric code from StackOverflow in its training set, or else OpenAI hired naïve coders to provide feedback, and they encouraged this sort of thing during RLHF.

I got tired of fighting with GPT-5 mini over style and started coding small Python changes myself. But when I was ready to deploy to the cloud I turned to it for help again. We got the job done faster together, but I wasn’t happy with my colleague. It hallucinated all sorts of command line options to the gcloud CLI. I got myself stuck in an exasperating loop, where GPT suggested a command, I pasted it into the terminal, the command failed because of GPT’s mistakes, and I pasted the error back into GPT for diagnosis. I’d become the dumbest agent in a system of dumb agents.

Act II: The Senior Staff Engineer Learns to Chill Out and Enjoy the Vibes

Gemini still doesn’t know how many arms humans have.

For my second attempt, I shamelessly embraced the vibe-coding lifestyle. I used AI to build an AI agent. The thing I wanted to build was an AI coach, who would look at my long-term goals (in some PDFs), my task list (using the Remember the Milk API), how I use my time (from the Toggl Track API), and what I’ve accomplished lately (in a text file where I track accomplishments). I’d read an inspiring article about building agents and thought, this looks surprisingly straightforward, I’ll make an agent do it for me.

(Yes, I’m telling an AI to tell an AI to tell me what to do. A few months ago this was futuristic.)

In the weeks since my previous session, Microsoft overhauled the coding agents in VS Code. Now there’s a clear separation between a planning agent and a coding agent. This made my second vibe-coding project go more smoothly. Plus, instead of accepting the default model, I chose Claude Sonnet 4.5, based on rumors that it’s good.

This might get confusing, since there are two LLMs in this story, so from now on:

Claude is the LLM I used to write code for an “AI coach”.
GPT is the LLM that the AI coach used as its backend.

I began with Claude in planning mode and described what I wanted: an agent written in Python, using the GPT API, plus various APIs to track my work. Its main task would be to suggest what I should work on for the next 45 minute pomodoro. I wanted to chat with the agent in my terminal. Claude and I had a 20-minute conversation about my requirements. It asked me, should auth tokens be stored in the macOS keychain or a text file? How often should it refresh its data sources, such as Remember the Milk and Toggl? Whenever I start chatting with the coach, should it remember our past conversations? How should the coach prioritize tasks—due date only, or also priority, or how well they align with my long-term goals? Whenever I answered Claude’s questions, it asked me four or five more. After a few rounds of this, I clicked the button VS Code provided me, which says, “start implementing.” Claude stopped asking questions and started generating dozens of files. I don’t know if the “planning mode” agent would ever have stopped asking for more details, or how I should decide when we’ve talked enough.

Claude worked for a few minutes autonomously. I occasionally had to click “ok” when it wanted to run a shell command or fetch a page of documentation from the web. The wait wasn’t long enough for me to work on something else with any focus, so I watched YouTube in 30-second slices between AI interactions. It seems helpful to have something mindless to do in these gaps—listen to music, watch climbing or cooking videos, practice tying knots… Claude generated 30 Python files and 2800 lines of code, and spat out a summary of its work. It claimed it had built OAuth flows for RTM and Toggl, and a context-management system that stored recent messages in SQLite and asked GPT for summaries of older messages, to limit the size of the context window. I resisted the temptation to read the code: I just tried it out.

> python3 main.py
✓ All services authenticated
ℹ Refreshing all data sources...
✓ Data refresh complete

╔══════════════════════════════════════════════════════════════════╗
║              Professional Coaching Agent (GPT-5.1)              ║
╚══════════════════════════════════════════════════════════════════╝

Type your questions or commands. Use /help for available commands.

You: /help

┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command        ┃ Description                          ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /refresh       │ Refresh all data sources             │
│ /refresh-rtm   │ Refresh Remember The Milk tasks only │
│ /refresh-pdfs  │ Refresh PDF documents only           │
│ /refresh-toggl │ Refresh Toggl Track entries only     │
│ /refresh-logs  │ Refresh accomplishments log only     │
│ /help          │ Show this help message               │
│ /quit          │ Exit the agent                       │
└────────────────┴──────────────────────────────────────┘

You: what should i work on next?

Coach: Given what’s on your plate and that you’ve already spent a lot of time in “AI coding” and “driver testing paper”, I’d suggest one of these for your next 45-minute block:

**1. Review my current MongoDB development plan**
- Due: tomorrow (2025-11-19)
- Why: Directly tied to your long-term goals (publishing pipeline, collaborations, synced clocks, etc.).
- 45-minute plan:
  - 10 min: Skim the dev plan PDF and your recent accomplishments.
  - 25 min: Write/update concrete 2025–mid-2026 goals (papers, talks, collaborations) in a doc.
  - 10 min: Turn those into 3–5 actionable RTM tasks (e.g., “Draft driver testing paper intro,” “Outline synced-clocks survey sections”).

**2. Prep for “Meet about leases in new architecture”**
- Due: today
- Why: High-leverage research conversation; you just finished your leader leases paper, so consolidating your thinking is valuable.
- 45-minute plan:
  - 15 min: Jot down what you want from the meeting (questions, decisions, next experiments).
  - 20 min: Review any notes on leases + new architecture, and your “Leases”/“Data movement” tasks.
  - 10 min: Draft a short agenda and 2–3 concrete proposals to discuss.

If you feel a bit fried from heavy cognitive work, a lighter option:

**3. Shut down lease benchmark EC2 instances (prep + checklist)**
- Due: 2025-11-30
- Use 45 minutes to:
  - List all benchmark-related resources.
  - Decide what to snapshot/keep.
  - Draft the shutdown steps so the actual execution is quick later.

Which of these matches your current energy:
- (A) reflective/planning (dev plan),
- (B) focused technical prep (lease meeting), or
- (C) lighter operational cleanup (benchmark EC2)?

This was an impressive debut. Most of the coach’s ideas were appropriate, and Claude’s code for reading the data sources and connecting to GPT all worked the first time.

As I used the coach more, I found some bugs. E.g., Claude had only stubbed-out the code for updating RTM tasks. If I asked the coach to “put off that meeting for 2 days,” it crashed. I was smarter now than in my previous attempt, though. Instead of copying error messages from the terminal to my chat with Claude, I used VS Code’s builtin terminal, so Claude could look at the terminal and see the error messages itself. I simply told Claude, “Look at the terminal and fix this bug,” and with little further intervention it found the stubbed-out code, looked up the RTM API docs, and implemented the missing feature. I maintained my disciplined laziness, never typing or even reading any code. It only took a few hours of testing the coach and asking Claude for bugfixes or changes, and the thing I’d envisioned was fully built.

Lessons learned

Vibe-coding is a skill.
Avoid reading and writing code??
Claude Sonnet is far better than GPT-5 mini, obviously. In general, it’s important to learn models’ differences.
Try to remove yourself from the agentic loop: give the AI the tools and permissions it needs to test its code and diagnose its mistakes autonomously.
Eventually, you might need to intervene, especially (I found) when the AI doesn’t know how to use a particular API and keeps hallucinating options. Fall back to searching Google and StackOverflow, like you did last year.
I need a very wide monitor. At my age even 27" isn’t enough to display the agent chat, file explorer, code, etc., all large enough for my eyes, so I’ve upgraded to an ultrawide.

I’m reminded of a more abstract lesson, the one from William Gibson: “The future is already here, it’s just not evenly distributed.” Sometimes the new technology works as advertised! Sometimes it doesn’t. Revolutionary new time-saving technologies take practice to use well.

Back in 2005, my mind was blown watching David Heinemeier Hansson implement a blogging engine in 15 minutes with Ruby on Rails. Until I saw the demo, I wouldn’t have believed such velocity was possible. Rails was so much more concise than anything before, and so was Ruby. And the code editor DHH used, TextMate, was so tricked out with templates and macros, it practically wrote the code for him. But when I tried Rails and TextMate myself, things didn’t go so well. The new technologies really were transformative—looking back, it’s clear that DHH was demonstrating the future of web development—but it took time for me and the world to catch up.

Aerial Silks in the Gunks, part 2

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 07 Oct 2025 17:13:59 +0000

October 6, 2025. Kelsey Roman performing with aerial silks while hanging from the West Trapps cliff in the Gunks. We photographed during sunset and moonrise. Rock guide Dustin Portzline managed Kelsey’s safety rigging and filmed behind-the-scenes footage.

I photographed this with a flash while hanging from a static line. I used a big wall harness for safety, a bosun’s chair for comfort, and a Peak Design thingy that’s now out of stock for clipping my camera to my harness.

Versions of The Eternaut

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 07 Oct 2025 13:17:09 +0000

In late-1950s Buenos Aires, on a quiet night, Juan is in his attic, playing cards with his close friends. The radio plays in the background. Juan’s wife and daughter are downstairs. On their suburban street, snow begins to fall. The music on the radio turns to static. Out the window, Juan sees cars crashing, pedestrians dropping dead: the snow is glowing, toxic, everyone it touches dies in seconds. It’s the prelude to an alien invasion. Juan’s family and friends are lucky: the windows were all closed, and Juan’s modern house is tightly sealed. They fashion hazmat suits from spare parts and venture out to gather supplies for their survival.

Over the next few days they’ll witness mass death, anarchy, waves of attacks by various weird aliens and crazy weapons, the destruction of the Argentine army, and perhaps the end of humanity. In the story’s final moments, Juan tries to steal an alien ship, but presses the wrong button and gets teleported to another dimension, doomed eternally to wander space and time, searching for his wife and daughter.

The Eternaut (El Eternauta in the original Spanish), written by Héctor Germán Oesterheld, was serialized in an Argentine sci-fi magazine from 1955 to 1957. It’s drawn by Francisco Solano López in a pulpy style, though with more artistic brushwork and composition.

Oesterheld wrote the story in the aftermath of Juan Perón’s overthrow and an airstrike against a pro-Perón rally in Buenos Aires’s central square, which killed hundreds of civilians. I don’t know much about Argentina’s history, and I’m confused about Oesterheld’s opinion of Perón and the coup. Everyone has to agree that the bombing of the Plaza de Mayo was an atrocity, I assume. But in The Eternaut, it’s the good guys who bomb a plaza, trying to destroy the alien headquarters.

The themes of The Eternaut seem universal enough to be reused by any political faction. On one side of the conflict is a classic 50s hero, a square-jawed family man, with his brave and competent friends, a physicist and a metalworker. On the other side, inhuman and implacable enemies who bombard Buenos Aires with toxic snow and lasers, then use mind-control devices to enslave the survivors. They could represent the military forces that bombed civilians and overthrew Perón, or the repressive regime that followed, or the foreign imperialists who tried to manipulate Argentine politics from afar, or any foe of middle-class liberal patriotism.

The comic was published in installments, and you can see the seams between them when reading it as a book: each episode begins with a bit of recap and ends on a cliffhanger. Too many times, a major character is presumed dead and reappears improbably. But the story is among the most gripping I’ve ever read. The alien assault is relentless, the velocity of events is frantic and disorienting, the body count is in the billions, and the only predictable plotline is that all human resistance will be crushed. The story’s end is abrupt and shockingly inconclusive. We never actually see the invaders, or know for certain why they attacked and what they will do with the handful of survivors, if there are any.

Eternaut 1969

In the 60s and 70s, Oesterheld moved left. He wrote biographies of Che Guevara and Eva Perón, and a book titled 450 Years of War Against Imperialism, and sequels to the Eternaut, which I hope are translated to English someday. Meanwhile, Argentina descended into its worst era of dictatorship and violence, known as The Dirty War. Oesterheld and his family joined a rebel group. He was “disappeared” and presumably killed by the government around 1977.

In 1969, Oesterheld rebooted the Eternaut with a new artist, Alberto Breccia, serialized in an Argentine gossip magazine. Breccia’s art is like a bad ayahuasca trip. Weird shapes and gloopy lines fill the background, the characters’ faces are lost in murk, the action is often illegible.

The content has darkened, too. In the 1950s, Oesterheld’s main characters are a plucky band of friends, and the Argentine army is led by stoic professionals. In 1969, everyone is paranoid. Juan observes that as soon as they rescue a young woman, the men see each other as rivals for her. The army is led by a brute who murders his lieutenant for dissenting.

In the original Eternaut, “the North” is a distant ally attacking the aliens with nuclear missiles. In the reboot, the northern powers have negotiated a separate peace with the aliens and abandoned the South, without even a warning.

Eternaut 1969 alienated its audience. Oesterheld’s resentful politics and Breccia’s avant-garde art led to the series’s early cancellation. In the last few pages the comic rushes to its conclusion, summarizing the plot in a few paragraphs and some vague illustrations. The work is a sad failure, comprehensible or interesting only to an Eternaut completist.

The Eternaut on Netflix

The original Eternaut and Eternaut 1969 were republished, including English translations, in 2015 and 2020. I think the Eternaut comics influenced Netflix’s superb TV adaptation of the novel Station 11: the Station 11 show includes images of people in improvised hazmat suits walking around snowy Chicago, and a knife fight with a hazmat-suited intruder, that seem more inspired by scenes in The Eternaut than by Emily St. John Mandel’s novel. Maybe Station 11’s success and the reissue of the comics led Netflix to greenlight a 2024 adaptation of The Eternaut. The first season aired last year, and another season is in production.

Oesterheld’s heirs required that any adaptation must be in Spanish and set in Buenos Aires. Netflix complied, but moved the action from the 1950s to the present day. The show has the luscious visuals, desaturated palette, stylish soundtrack, and tight plotting you’d expect from a prestige sci-fi series these days. It borrows and expands the ideas of the 1950s comic, combining them with the paranoid atmosphere of Eternaut 1969. Unfortunately, some of the original Eternaut’s attention to detail is lost. Oesterheld knew enough science to make the alien snow a consistent and believable phenomenon. His heroes, a band of can-do tinkerers, build various filters, respirators, and airlocks to survive. These are described with some specificity, in the Popular Mechanics spirit of the era. On Netflix, though, the snow seems to kill or spare people based on the writers’ mood and the needs of the plot. Oesterheld’s snow blocks radio signals, neatly isolating the main characters and driving the plot, whereas on Netflix the snow is accompanied by some implausible super-EMP that not only knocks out the power grid but drains cell phone batteries specifically, sparing old cars and flashlights.

I’m not mad that Netflix is unfaithful to the comic, though. We’ve had seventy years to add ideas and overtones atop the foundation Oesterheld laid. In that time, Argentina passed through several coups, dictatorships, republics, another Perón presidency, and finally a stable democracy. Some of the genres that Oesterheld drew from, like alien invasion and post-apocalyptic survival, were new in the 1950s and are well-developed today. Perhaps most important, we’ve all recently lived through a pandemic. Like Juan when he first saw the snow, we didn’t know if the disaster was natural or a bioweapon. Like him, we were trapped inside until we jury-rigged masks so we could venture out and gather supplies. We personally saw the camaraderie and paranoia, the selfishness and courage that humanity displays in such a crisis. The Netflix writers mixed all these ingredients into a bigger, more complex story than Oesterheld’s original.

Ridicullissima

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 29 Sep 2025 15:56:22 +0000

September 28, 2025. Chris climbing Ridicullissima, a tall classic 5.10d in the Gunks. I photographed him while hanging from a rope anchored near the top of the route. At Dustin Portzline’s suggestion, I experimented with anchoring the other end of the rope to the ground, a hundred feet to the side of the climb. I attached myself to the rope with two Gri Gris and attempted to position myself in space by individually lengthening or shortening the two segments. It was awkward; I have a lot to learn. At least having two attachments prevented me from spinning.

Behind the scenes photo by Dustin Portzline:

Eye Opening Ceremony at the New Paltz Zen Center

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 28 Sep 2025 22:03:24 +0000

September 27, 2025. A ceremony in which the New Paltz Zen Center was officially made a Zen temple. Jo An sensei brushed open the eyes of the Manjushri Bodhisattva statue on our altar. The abbot of our mother temple the Village Zendo joined and gave his blessing. A big step for my little community.

Images © A. Jesse Jiryu Davis

How To Belay With Prescription Glasses Or Sunglasses

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 22 Sep 2025 12:35:40 +0000

If you want to wear sunglasses and/or prescription glasses while belaying a climber, especially outdoors on multipitch climbs, I hope this post helps you cobble together a system that is functional and fairly robust.

Glasses

I bought my first pair of Roka glasses a couple years ago and they transformed my life. Before, I’d worn:

prescription glasses to see far,
prescription sunglasses to see far in sunlight,
contacts to see far without my glasses falling off while I run, bike, or climb,
contacts plus sunglasses to run, bike, or climb in sunlight.

Then I replaced all those configurations with Rokas. I have a pair of transition glasses (aka photochromic, they darken in sunlight), with Roka’s special rubber that grips my nose and ears. I can run, bike, and climb with them and they don’t slip, even when I sweat. They’re an all-purpose vision device: I put them on my face and I can see.

You don’t need to spend $400 on a pair of Rokas though; the rest of this post applies to any (sun)glasses. I just had to tell you how much I like Rokas.

Belay glasses

Before I got my Rokas, I used to climb and belay wearing contacts. That worked fine with belay glasses except in sunlight, when I wished I had belay sunglasses. Now, though, I wear Rokas all the time, and my belay glasses fight with my Roka glasses for nose-space.

YY Vertical Clip-Ups are unique: they’re built to work with your existing glasses. They clip to the top of your glasses’ frame, and they can flip up (so you can see normally) or down (so you can look upward through the prisms).

They’re a bit heavy, so even the special Roka rubber can’t keep my glasses from slipping on a sweaty day when the Clip Ups are attached. YY Vertical includes a pair of ear hooks to attach to your glasses so they embrace your head more ardently. I lost mine. In Europe, you can order replacements from YY Vertical. Here in the US, I replaced them with these from Amazon.

Combined with my glasses, they look like this:

They slip on and off the arms in a few seconds, so you don’t need to look dorky when you’re not climbing. The whole package looks like this:

The Clip Ups pop on and off the glasses in a second. I put them on while I belay and put them away in their case clipped to my harness when I climb. Now I can see far, straight ahead or up, in sunlight or shade!

Reinforce the tips

The four rubber tips that keep the Clip Ups gripping your glasses are not well-secured to the Clip Ups. You should reinforce them with Super Glue as soon as you buy your Clip Ups, otherwise the tips will fall off almost immediately and become micro-trash. I learned this too late and lost three of the four. Just like the ear hooks, YY Vertical sells replacements only in Europe. I replaced mine with the smallest size from this box of rubber tips; but I wish I hadn’t had to waste the rest of the box.

Upgrade the case

All belay glasses, including Clip Ups, are sold with a flimsy case and a flimsy carabiner that will soon break, falling off your harness when you’re five pitches up and tumbling into the void. I guess a robust case would double the total cost of the belay glasses and give us all sticker shock, but it’s a shameful waste.

I replaced the Clip Ups case with this “tactical wallet with coin purse” by OneTigris, plus a full-strength climbing carabiner:

OneTigris makes gear for guys who want to dress up like commandos and go to January 6th riots, I think. Apparently those guys need tactical wallets with coin purses so they can buy an ice cream cone after they survive the apocalypse. OneTigris stuff is silly but well-built, and this wallet fits the Clip Ups snugly and has a carabiner loop that seems durable. It has a water-resistant zipper that keeps dust out. There are a few little pockets where you could slide a chapstick or something, too.

After years of faffing around, I think I found a belaying setup that works for me. Maybe it will work for you, too!

How To Check All Your Feeds With One Tool

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 08 Sep 2025 01:36:07 +0000

That’s right, I’ve found it. The holy grail, the one feed to rule them all, the aggregator for your whole Internet. None of these walled-garden rent-seeking websites want you to have it, but I’ll share it with you.

It’s your bookmarks.

I have a long and often-changing list of websites I want or need to check:

Facebook, mainly for my local climbers’ group.

You Need A Budget to track my spending.

Threads. I miss Twitter’s golden age. Since I quit Twitter, I hoped Mastodon would become something. It didn’t, but I keep hoping Threads will become something.

Bluesky. Since Threads is a bust, I keep hoping this will become something.

Feedbin for the blogs I follow via RSS. This is the actual answer to, “What do I do without Twitter?”

Instagram. Technically it’s the worst photo site, but all the photographers are still here?

Glass, a technically good photo site with hardly any photographers.

Discord. I hate it, but I use it for talking about distributed systems with non-MongoDB colleagues.

My Google Docs spreadsheet of friends. A nerd’s way to maintain social ties. For each friend, I track how recently we’ve interacted, and a target frequency. Overdue friends turn red.

Lobste.rs. Hacker News without the preening fools, yet.

I put bookmarks to all these sites in a folder called “check” in Chrome’s bookmarks bar. Here’s how it looks in the bookmarks manager:

When I want to spend ten or twenty minutes just goofing around on the Internet, I right-click the “check” folder and select “Open All”, and the pages open in tabs:

This isn’t just a convenient way to goof off, it’s a behavior-modification tool for my future self. When I decided a couple years ago to get back into RSS, I added Feedbin to the folder. When I decided to give up on Mastodon, I removed it from the folder so I wouldn’t keep trying to make Mastodon happen. That bookmark is gone now, and so is my frustration.

When I instituted my friends spreadsheet, I added it to the folder. When some colleagues started using Discord, I added Discord to the folder. Now my slightly addictive websites like Instagram are paired with my regular chores, like You Need A Budget. I influence my future attention by curating the bookmarks in my “check” folder.

If there’s a site I want to check frequently, e.g. to see if a conference has posted next year’s dates, or whether anyone has replied to my question on an obscure forum, I add it to the folder. It doesn’t matter if the site has no way to subscribe by RSS or email. And I don’t need to use an unreliable third-party website change tracker like VisualPing, which has trouble determining if the site really changed in a way that matters to me. I just toss the site in my “check” folder, so I’ll automatically open it the next time I want to goof off. I keep checking the site until I see the change I’m hoping for, or give up. Either way, I just delete the bookmark from the folder.

How My Blog Handles Math and Images in HTML, Atom, and Email in 2025

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 31 Aug 2025 08:43:59 +0000

I want to explain computer science on my blog, and show photos, and I want every article to look exquisite. My goal is to display math and images (especially SVGs) as beautifully as possible for people who read my articles on my site, and via an Atom feed and email. This is hard with today’s technology, plus I don’t have real frontend or design skills.

I’ve recently started a series of articles about epistemology and distributed systems, which includes diagrams and equations that broke my existing publication system. I came up with a new stack of kludges that works for me. I’ll write them down here so I remember, and perhaps you’ll learn something you can use.

Throughout this post, you can refer to this map:

My site generator

I use Hugo. I learned about Hugo early, since Steve Francia was my boss at MongoDB when he created it, but I resisted using it for some time. It’s not as extensible as the Python-based static site generators, and it relies on Markdown, which is a hacky format. But the various Python generators were all much too slow for me, so eventually I switched. Hugo is super fast enough: even now that my site has 700 articles and thousands of images, it does a full rebuild in under a second.

Images and previews

While I’m drafting an article, I run the Hugo server and preview the article locally. Whenever I save a change to the article, Hugo rebuilds it and auto-reloads it in the browser. For some reason, Hugo doesn’t properly reload when I change an image, even though it’s perfectly aware that the image has changed. I fixed this by creating this template:

{{/* This is layouts/_default/_markup/render-image.html */}}
{{/* Email-safe output for Atom/Kit */}}
{{/* .IsBlock is available if configured: gohugo.io/render-hooks/images/ */}}
{{- if .IsBlock -}}
<img
  src="{{ .Destination | safeURL }}"
  style="display: block; max-width:100%"
  alt="{{ .Text }}"
  title="{{ .Title }}">
{{- else -}}
<img
  src="{{ .Destination | safeURL }}"
  style="display:inline;width:auto;height:auto;vertical-align:middle"
  alt="{{ .Text }}"
  title="{{ .Title }}">
{{- end -}}

This template interprets Markdown images, like ![](foo.jpg). If it’s running inside Hugo’s development server, then it adds a cache-busting fingerprint to the image URL. Now whenever the contents of foo.jpg change, Hugo creates a copy of the file with its hash included in the filename, like foo-xxx.jpg. Hugo updates the HTML page to point to foo-xxx.jpg, so my browser will definitely load the newest version of the image. When I change foo.jpg again, its hash is now different, so it’s named like foo-yyy.jpg, and the browser reloads it properly.

This shouldn’t be necessary—this is what ETags are for—but Hugo needs this kludge to properly refresh the preview when I change an image.

Publishing an article

I manage a variety of tasks with some PyCharm custom shortcuts and an ever-growing Python script called blog. Working together, PyCharm and my script can do some advanced tricks, like pasting an image from the macOS clipboard into a Markdown file, or creating a photo gallery, or starting up the Hugo development server and opening the browser to a preview of the draft I’m currently editing.

The blog script works especially hard when I finish a draft and publish it. I’ll describe the publication pipeline below.

SVGs

I love to draw diagrams in Excalidraw and export them as SVGs like this:

An intricate diagram. I drew this for an article about knowledge.

I’ve learned that mail readers (even web-based ones like GMail) won’t show SVGs, so I need to give them a fallback. Remember, the article pipeline is HTML → Atom → email, so my Atom feed must provide that fallback, which is then passed through Kit to email subscribers.

When I publish a draft, my Python script creates a PNG for every SVG in the article (or a JPEG, if the PNG is too many bytes). I have a template that converts every SVG in the feed to a <picture> that prefers the SVG but falls back to the PNG or JPEG. This template is called from the template that generates the whole Atom feed.

Math

A well-rendered formula is beautiful, like this Riemann sum:

I use both block formulas like the one above, and also inline formulas, like . By default, Hugo renders math with client-side Javascript called MathJax, but I want to show math on the web using the modern HTML <math> tag instead. I override Hugo’s default with this template. Hugo requires it to be called render-passthrough.html, but don’t be fooled: it’s not for a generic “passthrough” feature, it’s for rendering math specifically. The template also sets a variable hasMath to true, so this other template knows to include special math CSS.

HTML <math> tags may not display in Atom readers or email, so once again my Python publication script has a job to do. When I publish an article, the script searches it for formulae, and creates an SVG for each. The best way I found to do this was install the MathJax library on my laptop and execute it at build time to produce an SVG. Then my Python script produces a PNG for each formula’s SVG.

The filenames for the SVGs and PNGs include a hash of the formula itself. Imagine the “xxx” in the filenames above are 64-bit hashes. On the web, you see <math> tags rendered by your browser, but in Atom (and therefore in email), you see the SVG, or fall back to the PNG. This template replaces <math> tags with <picture> tags in the Atom feed, using the hashes of the formulas to determine the correct image filenames.

Block and inline images

I didn’t use many inline images until I started writing a lot of math formulae. But now, since inline formulae become inline images in Atom and email, I need inline images to render correctly. At the start of this journey I was using Mailchimp’s feed-to-email automation, and Mailchimp seems unable to show both inline and block images in emails from feeds. By default, inline images are okay, but block images expand to a monstrous width. If you select Mailchimp’s option “Resize RSS feed images to fit template”, then block images look good, but inline images are stupidly converted to blocks, which is abominable.

This template adds CSS styles to images in the Atom feed to make them behave. It also transforms block images to inline images, wrapped in paragraphs. This didn’t actually work, though: Mailchimp mangled my images regardless.

{{/* this file is render-image.rss.xml, it's called rss but it makes
     email-safe output for Atom/Kit. Put a block image in a "p" and
     mark inline for proper fill in Atom-to-email campaign */}}
{{- if .IsBlock -}}
<p>
<img src="{{ .Destination | safeURL }}"
     style="display:inline;max-width:100%;
            width:100%;height:auto;margin:1em 0"
     alt="{{ .Text }}" title="{{ .Title }}">
</p>
{{- else -}}
<img src="{{ .Destination | safeURL }}"
     style="display:inline;width:auto;height:auto;
            vertical-align:middle"
     alt="{{ .Text }}" title="{{ .Title }}">
{{- end -}}

I switched from Mailchimp to Kit (formerly ConvertKit) for my Atom-to-email automation. Mailchimp did some funny business with image widths that I couldn’t defeat, but Kit worked well from the start. I didn’t experiment to check if the template above is actually necessary with Kit, but at this point I was getting tired of futzing with my blog.

(Note: this template is only executed for Markdown images like ![](foo.jpg), not handwritten HTML <img> tags in Markdown. I’ll have to remember that as I write.)

Conclusion

The feeding machine in Charlie Chaplin’s “Modern Times”

What a tzimmis. What a kludge. Why is all this necessary in 2025? I guess the future is already here, it’s just not evenly distributed: feed readers are behind the curve, and email readers farther behind, because of security concerns or other issues I don’t understand. So I have to build a complicated pipeline to transform math into SVGs, and SVGs into other formats, to ensure graceful degradation. Furthermore, Hugo is fast but hard to customize, and it takes a precarious stack of templates to (usually) produce the HTML that I want, where I want it. These templates use regexes to parse HTML, which will inevitably break on some page someday. And finally, RSS-to-email automation is a dark art.

I hope all this is useful to someone else. Maybe you don’t need these hacks. Maybe a basic Hugo setup works fine for your blog, or you don’t mind if your images don’t appear exactly right in all channels. Good for you. But there may come a time when you need more. I will friend you, if I may, in the dark and cloudy day.

Knowledge and Common Knowledge in a Distributed Environment, Part 2

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 25 Aug 2025 10:42:25 +0000

Odin, god of wisdom.

This is the second in my series of articles about Knowledge and Common Knowledge in a Distributed Environment, a beautiful and difficult 1990 paper about distributed systems and epistemology. So far, I analyzed the “muddy children puzzle,” I defined levels of knowledge, and I used these levels of knowledge to analyzed the Raft protocol (which was published long after this paper).

Now, the moment you’ve been waiting for: coordinated attack!

Coordinated attack

Chinese general Guan Yu.

Two generals are encamped on hilltops, on either side of the enemy army. If they attack simultaneously, they’ll win, otherwise they’ll fail. They have no initial plan, so they send messengers back and forth, to try to agree on a time to attack. Unfortunately, messengers can be unpredictably slow, or they can be caught by the enemy. How can the generals agree on a time?

(The paper’s authors, Halpern and Moses, call this problem “coordinated attack” and they say it was introduced by Jim Gray in 1978. The puzzle may be older than that. Sometimes it’s called “Chinese generals.” The “Byzantine generals problem” is a more complex version that Lamport described later.)

As you probably know, the generals can never agree. If general A sends a message saying, “attack at dawn,” he doesn’t know whether B received it, so he risks attacking alone. So perhaps the protocol is, “attack if you proposed a time and received acknowledgment”? But if B receives and acknowledges the message, she doesn’t know whether A got her acknowledgment, so she also risks attacking alone. And so on. No protocol in this asynchronous system can guarantee they attack simultaneously.

Halpern and Moses (citing earlier work) use a many scenarios argument to prove that coordinated attack is impossible. If A sent the message and hasn’t observed an acknowledgment, there are many scenarios that fit his observations:

The message was lost
The message is still in transit
The message was received and the acknowledgment was lost
The message was received and the acknowledgment is still in transit

General A’s knowledge is the set of facts that are true in all the possible scenarios that match his observations.

So A knows his message was sent, but that’s it. If B receives the message and sends an acknowledgment, she also has four scenarios she can’t distinguish. The generals can’t be sure they’ve agreed to attack unless A knows that B knows that … infinitely. In other words, their attack time must be common knowledge, and common knowledge is impossible in an asynchronous system.

(Yes, this is related to FLP impossibility.)

Definition of knowledge

Here’s my favorite part of the paper. It begins with a mathematical model of a distributed system, which will be familiar to people who use PlusCal, TLA+, or other formalizations. There are some processors (aka nodes or agents) called p₁, p₂, …, p_n. A run of the system is just like a TLA+ behavior: a sequence of events starting at time 0 and continuing infinitely (perhaps reaching a final state and staying there forever). The system is characterized by the set of all possible runs, R. This is just like a TLA+ state graph. A point (r, t) is a moment in a run r at time t. At every point, each processor has its own local history, the events it has observed, including its own actions. E.g., a Raft node knows the actions it’s taken and the messages it’s received, but it has no direct knowledge of other nodes’ histories. A protocol is a function of a processor’s local history: each processor deterministically chooses its next action based on its observations so far.

A processor p_i knows a fact at point (r, t) if is true at all points of R that are indistinguishable from (r, t). By indistinguishable points, I mean all points (r', t') where p_i’s local history is the same as at (r, t). For example, if I observed my partner Jennifer go out the front door, and I haven’t seen her come back in, then I know she’s gone, based on my local history of observations. She might have taken the car or gone for a walk—since I haven’t looked in the parking lot, those two scenarios are indistinguishable to me. But “Jennifer’s gone” is true in both scenarios, so I know that fact.

In Raft, if the leader has sent a log entry to both followers and only one follower has acknowledged it, then the leader can’t distinguish between the scenarios where one or two followers received the entry. But “the entry is majority-replicated” is true in both scenarios, so the leader knows that fact.

A processor’s view of the system is a function of its history. The view could just be the identity function—the processor’s view is its history, i.e. the sequence of all its actions and observations. Or the processor’s view could be a summary of its history, e.g. the Raft election protocol requires a node to remember only the most recent vote it cast, not the sequence of all its past votes. In TLA+ we usually have some variables for each node, which are updated when nodes take actions or receive messages: these are the nodes’ views, their summaries of their local histories!

Halpern and Moses more formally define a processor’s view of the system, and how that relates to levels of knowledge, like distributed knowledge and common knowledge, in Section 6 of the paper. It’s great, go read it.

Indistinguishability graph

As a TLA+ guy, I usually think of a state as an assignment of values to variables, and the state graph connects states with directed edges representing possible state transitions. Let’s consider the muddy children from the previous episode. Say there are two children. Our variables are:

a, b: Whether child a and/or b is muddy.
m: Whether the father has announced, “At least one of you is muddy.”
q: The number of times the father has asked, “Can you prove you’re muddy?”

The initial state is:

Then the father makes his announcement (m becomes true), and then he starts asking his question (q increases). We know at least one child is muddy; let’s say that’s child a and the maybe-muddy one is b. Here are the states and transitions:

How can we use this graph to represent what the children know? For any two states, let’s add a non-directed edge between them if some child has the same view in both states, i.e. the states are indistinguishable to that child. And let’s label the edge with the name of the child who can’t distinguish the states:

Child b can’t distinguish if it’s muddy or clean in the initial state, so there’s a blue “b” edge between the first two initial states. The two states are distinct in a’s eyes, because it can see whether b is muddy, so we don’t label the edge “a”. After the father announces “at least one of you is muddy,” b still doesn’t know if it’s muddy, because it sees that a is muddy so a might be the only muddy child. Therefore the two next states are also indistinguishable to b. Finally the father asks, “can you prove you’re muddy?” As I explained in the previous article, a and b now know if they’re muddy, so the third states are distinct to both of them and there’s no blue edge.

Can we do the same for Raft? The real Raft TLA+ spec has dozens of variables. Let’s simplify. Let’s say there’s one log entry, there’s a permanent leader, and there is a set F of two followers named f₁ and f₂. Here are our variables:

r₁, r₂: Follower f₁ or f₂ has received the entry (“r” for “receive”).
a₁, a₂: The leader received an acknowledgment from f₁ or f₂ (“a” for “acknowledge”).

At first all the variables are false. Then f₁ or f₂ receives the log entry, and some sequence of receiving and acknowledging leads to the final state, where both followers have received and acknowledged the entry. Here’s a state-transition graph, false is white and true is red:

A state transition graph.

In the last article I defined two facts:

(phi): the log entry exists
(psi): the log entry is majority-replicated

A follower knows if it received the entry, and the leader knows if it knows at least one follower knows , since the leader plus one follower is a majority of the three-node replica set. In other words:

Let’s analyze this with the graph. A follower f knows , if is true in all the states indistinguishable to f from this state. The only variable in f₁’s view is r₁, so all states with the same r₁ value are indistinguishable to f₁. For f₁’s indistinguishability graph, I’ll draw edges between the nodes where r₁ is true, and edges between the nodes where r₁ is false:

The follower f₁'s indistinguishability graph.

The graph for f₂ is the same, but flipped vertically.

The follower f₂'s indistinguishability graph.

The leader doesn’t know whether a follower has received the entry, so states that differ only by their r₁ or r₂ values are indistinguishable to the leader. The leader knows what acknowledgments it received, so it can distinguish states by their a₁ or a₂ values.

The leader's indistinguishability graph.

Putting it all together:

The combined indistinguishability graph.

As I said earlier, “the leader knows ” is equivalent to “the leader knows that at least one follower knows .” We can evaluate a property like this with the indistinguishability graph. The leader knows f₁ knows at a state S, if every path from S along zero or one leader edges, then zero or one f₁ edges, leads to a state where f₁ knows :

This is a graph-style way of saying:

the leader knows f₁ knows in state S, if
in all states T indistinguishable (to the leader) from S,
f₁ knows in all states indistinguishable (to f₁) from T !

The same goes for states where the leader knows f₂ knows :

Putting it all together:

As you’d expect, the leader knows in states where a₁ or a₂ is true; i.e. states where one or the other follower has acknowledged the entry. But it’s cool to see how it can be expressed as a graph query. This seems ripe for automatic verification. Halpern and Moses describe how various levels of knowledge, such as distributed knowledge, or “everyone knows,” or “everyone knows that everyone knows,” are properties of paths through the indistinguishability graph.

I’ll stop here and let my brain cool off. Next time: agents in a distributed system can achieve common knowledge if they have reasonably reliable clocks.

Odin and other Norse images by Gerhard Munthe.

Speak Up If You're Stuck in a Zazen Rut

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 19 Aug 2025 11:13:09 +0000

Certain topics seem off-limits in Zen, like: What’s the goal of meditation? How do I get better at it? What is everyone else experiencing when they sit? I spoke at the New Paltz Zen Center about getting stuck at a plateau in meditation skill, and about Zen communities’ reluctance to discuss peak experiences, progress, and goals. Here’s the video and a written version of the talk.

There’s a lot we don’t talk about in Zen.

In my experience, we hardly ever talk about what our daily zazen is like. What’s it usually like for you? Is it groggy, dull, joyful, peaceful, deep, distracted? How long do you usually go between thoughts? Do you usually hear every sound outside your window, or do you kind of space out most of the time?

We also don’t describe the deepest experiences we’ve ever had. And we hardly talk about progress either: have you gotten better at meditation over the years? In fact, I get the sense a lot of people are allergic to two words I just used: “progress” and “better”!

I haven’t read anywhere that we’re not supposed to talk about this. Not that I remember. But I’ve picked it up from hearing what other people say or don’t say. Maybe I just made an assumption that these topics are off-limits, and never questioned it.

When we first learn to meditate, there’s a lot of talk about the contents of our zazen and how to do it better. Like whenever I’ve taught beginning instruction, I meditate with people for a few minutes and then I ask them, “What happened? What was that like?”

But then that kinda … dies? I don’t ask experienced meditators, “What was that last half-hour like for you?” They don’t ask me either.

Maybe you’re all talking about it in private, in dokusan. Dokusan is like sex, I don’t get to see what other people are doing. Maybe you’re always asking the teachers, “I can’t stop worrying about work. How do I reliably achieve samadhi?” There’s nothing stopping me from asking that in dokusan. I mean, nothing overt. But I don’t. Over my years in Zen I’ve come to hide my curiosity, partly because I’ve heard our community’s silence on certain topics and I’ve become silent about them too.

Lately I’ve been talking with some of my fellow students, and they mostly agree: we hesitate to talk intimately about meditation, and it’s a relief when we occasionally break the silence. Some of the core questions we have about our practice—over time we get the message that we’re not supposed to ask. Our strongest spiritual desires—we’re not supposed to express them. Our greatest accomplishments, we’re not supposed to celebrate them. I often feel this way, and I’m not the only one.

So today is an experiment. I’m going to break some silences and see what happens.

One of the deepest meditations I ever had was in late August 2004. I was 23 years old, it was the last night of my last sesshin at the end of a year living at Yokoji monastery, off the grid high up in the mountains in Southern California. I knew that in a few days this monastic life would end, I would fly to New York City and start a new life of some sort. So, on the final night of sesshin, after everyone else had left the zendo, I stayed. The jikido had blown out the candle—we had to be very careful at Yokoji not to burn the monastery down—and all the lights were off—electricity at night at Yokoji was from a couple of car batteries charged from a solar panel during the day, so we used it very sparingly—except there was a dim yellow light in a sconce on the zendo wall. I was sitting in that pool of dim warm light, with the dark room around me and the dark night outside, and the cacti and the pine trees and mountains around me.

For a little while, I stopped. I barely had any thoughts. I heard every single cricket chirp, without missing one, for minutes upon minutes. There was nothing wrong with the world, and nothing wrong with me. For the first time in years, I wasn’t worrying about myself, wasn’t trying to fix myself. I had no preferences. Normally, I’d be impatient to leave the zendo, go down to sit on the tree stump at the bottom of the hill and smoke a cigarette in the moonlight, then read in bed for a few minutes and do all the little indulgences that would comfort me at the end of the day.

But this night, I had no preference. I could leave the zendo, or I could stay all night listening to the crickets. Everything was ok, I had finally stopped trying to fix my life.

Eventually I did leave the zendo of course, smoked my cigarette in the moonlight, and a few days later I flew to New York City and I’ve never had a period of zazen quite like that again. And I was pretty angry about that for a while. I gave a year of my life to Zen, and the second I leave the monastery, my samadhi is all shot and I can’t get back to that peaceful place anymore. I blamed Zen, and I blamed myself. It was actually pretty reasonable to blame myself—I was smoking pot and getting drunk every night, I was a mess, I wasn’t creating the conditions for a strong Zen practice and I knew it. But I couldn’t help remember that one night and wish I was there again.

So this is the trap of being attached to an experience, and comparing the past and the present, right? Zen texts warn us about this trap all the time. But on the other hand, that comparison told me something about how I was living; it was a signpost. I knew I was capable of that kind of samadhi, and the farther my zazen got from the experience of that night, the more clear it was that something was going wrong with how I was living in general.

At the beginning of that year at the monastery, when I arrived at Yokoji, there was another young guy my age, Wes, who showed up the same day. I was very impressed with all the monastics at Yokoji, of course, I wanted to be just like them and I wanted them to approve of me. But I kind of hated Wes. He was maybe a year younger than me, big and goofy and enthusiastic. Once he came into the monastery kitchen, took a gallon of milk from the fridge and chugged the entire thing in one go, right in front of everybody.

My worst fear was that the legit Zen monks at Yokoji would see me and Wes as the same. I had to distinguish myself in their eyes from this milk-chugger. I had to be better than him, more sophisticated. One day, Wes and I and a few of the monks were hanging out in the kitchen, and Wes said that the day before, he’d been concentrating on Mu, and he had some kind of mind-blowing experience where everything was dark and empty and Mu was everything and he fell into it. I was so jealous. How could big, goofy Wes have achieved this state already, after a few weeks, and not me!

That was more than 20 years ago, and I still remember that moment in the kitchen, listening to Wes, feeling my envy. It got deep into me and grew into something bigger, something that lasted. I think this is one of the reasons we don’t generally talk about our zazen experiences in public. What if other people compare themselves to us? What if they feel jealous, or on the other hand, what if their zazen is better and they feel arrogant?

A subtler danger is that, by describing something aloud, I make it into a thing. I fix that memory, like a sticking a needle through an insect specimen. I took that risk by describing my samadhi to you just now. Now my memory of that final night has been permanently changed by describing it to you. It’s changed and it’s hardened.

And you know what? Life goes on. I get jealous sometimes, or self-satisfied other times, and my practice continues and I get over it. I have nice experiences, and then they end, and I’m disappointed, and I get over that too. Jealousy and comparison and attachment to past experiences, these are just natural human feelings, and I don’t think we have to be so concerned about protecting ourselves from them. The benefits of talking about practice openly might outweigh these risks.

Sometimes I have arguments with Dogen. The arguments are a bit one-sided, because I’m angry and he isn’t, plus he’s been dead for 800 years. He sits there in his black robe. He has a pale round baby face and a little rosebud mouth, like in that one painting I’ve seen of him. He’ll start with some innocuous instruction, like: “Sit stably in samadhi.”

I say: I’ve been trying for half my life and hardly ever achieved stable samadhi. The best sit I ever had was after a year in a monastery, do I have go back to the monastery?

Dogen: There is no reason to leave your own seat at home and take a meaningless trip to the dusty places of other countries. The treasure house will open of itself, and you will be able to use it at will.

Me: I’ve been waiting more than 20 years and the treasure house seems mostly closed. I definitely can’t “use it at will,” I have so little control. Time swiftly passes by and opportunity is lost. I need to change something before I waste any more time.

Dogen: How can we distinguish practice from enlightenment? The Vehicle of Reality is in the Self. Why should we waste our efforts trying to attain it?

Me: At this point I’ve long since given up on “attaining enlightenment.” I’m just trying to be a bit more enlightened from moment to moment, and I’m looking for a meditation practice that supports me better. Surely it’s ok to experiment a bit and see if something works better?

Dogen: Learn to step back, turning the light inwards, illuminating the Self. Doing so, your body and mind will drop off naturally, and Original Self will manifest. If you wish to attain suchness, practice suchness immediately.

Me: This is beautiful. I really mean it, Dogen, I appreciate your beautiful poetic writing. But it’s not very helpful for me. Can you break it down at all?

Dogen: Think of not-thinking. How do you think of not-thinking? Beyond thinking. This is the essential way of zazen. The zazen which I am talking about is not step-by-step meditation. It is simply the dharma gate of peace and comfort. It is the practice-enlightenment of the ultimate Way.

Me: But it doesn’t feel like “the dharma gate of peace and comfort” most of the time. Instead, it’s been a dharma gate of struggle and frustration.

At this point Dogen sits in serene and irritating silence.

I’ve felt quite stuck in my zazen. Some retreats are deep, some are just “whatever,” some I spend in awful anxious obsession. I can’t predict or control it. And my half-hour morning meditation has just been groggy mind-wandering; I wonder if it’s worth it or if it would be better to just sleep in.

I’ve been on this disappointing plateau for many years, and I stopped asking teachers about it at some point. I got embarrassed to admit how stuck I was. I teach meditation to other people, I give dharma talks, I’ve been doing this so long, I’m ashamed to admit how little I still understand about how my mind works and how to guide my own zazen. And also I’d gotten the impression that everything I’ve just said isn’t Zen—I’m not supposed to evaluate my progress, I’m not supposed to want progress, I’m not supposed to want my zazen to be a certain way!

So this June, my partner Keishin and I did a weeklong retreat with a company called Jhourney, which is trying to teach people to reach specific states in meditation. This company was founded a few years ago in San Francisco by some young guys who’d been practicing jhana meditation. The jhanas are levels of meditative absorption, they’re described in a few suttas of the Pali canon, written 2000 years ago. There are eight levels of jhana, give or take. In the Mahāsaccaka Sutta, Buddha says,

I recall once, when my father was working, and I was sitting in the cool shade of a rose-apple tree, then—quite secluded from sensuality, secluded from unskillful mental qualities—I entered and remained in the first jhana: rapture and pleasure born from seclusion, accompanied by directed thought and evaluation. Could that be the path to Awakening? Then following on that memory came the realization: That is the path to Awakening.

In recent years some Western Buddhists have become very interested in bringing the jhanas to the West and learning or rediscovering how to enter these states reliably. And these young San Francisco guys had been studying under some teachers, comparing notes with each other about what sort of meditation techniques worked more or less well to enter the jhana states, and they decided to create a company to teach the jhanas to others. That’s Jhourney.

So, doing this Jhourney retreat was a chance to get out of my zazen rut. Jhourney is fairly scientific. They say a certain percentage of people on their retreats reach the first jhana or the second jhana. They run experiments with different methods for teaching the jhanas, and they measure their retreats’ outcomes and try to optimize the path to the jhanas. That seemed really attractive to me, just a fresh way of doing things. It seemed like the opposite of Zen. Plus, I could walk into a room of strangers and admit how frustrated and stupid I feel, because I didn’t care what they thought of me.

The Jhourney retreat was, indeed, the opposite of Zen. We were handed a 120-page instruction book the first night of retreat. Most of the retreat was unscheduled free time, but we had a lot to do in that time. We had to read most of the instruction book in the first few days. The book told us to try a bunch of experiments on ourselves and write down the results. Like, when I recite a metta mantra, does it produce a feeling somewhere in my body? What’s that like? What about if I visualize a tree of compassion growing from my chest, or if I just smile? How is it different to sit or lie down, to be silent or listen to music, to sit for 45 minutes, or an hour, or more? We tried all these experiments in our meditations, figured out what seemed to work to produce a peaceful, openhearted, effortless feeling. That feeling is the ember that can grow into a roaring fire, and that’s the first jhana. Once we could produce that feeling, we kept refining and experimenting and writing down the results.

We had regular interviews with “coaches,” asking what we tried, how well it worked, and what we were going to try next. One time I meditated with my coach for 20 minutes, narrating my experience moment by moment, while she directed me on how to use my attention and my imagination to steer the meditation.

The retreat was mostly silent. But partway through, the retreat managers split us into groups of four to have dinner together and discuss our experiences with the other participants. We were encouraged to say what we’d experienced, whether it was a jhana or something else, and swap tips about how to go deeper in meditation. I have to admit I was really apprehensive about this—what if someone else has gone much deeper than I have, will I be really jealous and disappointed? What if that person is some meditation noobie who doesn’t deserve to succeed more than me, someone who got to some profound jhana from beginner’s luck? If that happens I’ll feel like a fool. It’ll be Wes in the kitchen all over again. Luckily I had reached Jhana One a couple times before this dinner, and that was on par with the other people, so I felt good about myself. But I noticed how the fear of being jealous was a big deal for me!

Talking in detail about what we’d tried and how it was going, it was helpful. And it brought us together. On sesshin, it’s easy for me to assume that some people have it all figured out. If you’re sitting still and upright, I’m guessing you’ve got zazen figured out. I am also sitting still and upright, and my zazen is a catastrophe, but I assume I’m the only one who’s struggling. But if we talk about it, and I hear that you’re struggling too, I’m not alone anymore.

On the Jhourney retreat, the coaches set up laptops in a separate room, where we could go watch videos of past participants describing the jhanas they reached. I did not go watch the videos, again because of my fear of jealousy. And also because, towards the end of the retreat, I was enjoying meditation so much I didn’t want to spend my free time doing much else.

One of the goals of the Jhourney retreat is to enter a jhana. The jhanas for most people go in a certain order, starting at Jhana One, and indeed the jhana that I entered matched the description of Jhana One. I was following the instructions: I relaxed and I smiled a little bit, I welcomed all feelings and thoughts that arose. Whereas usually in zazen I try to drop my thoughts, and let feelings come and go, with jhana meditation I welcomed each thought and feeling, I invited them to join my meditation. I focused on a feeling in my body, Jhourney calls this an “openhearted feeling,” for me it’s like a bittersweet heartache. It was the bittersweet joy and relief at finally being compassionate with myself.

I sat like this for an hour and a half, occasionally shifting position, but I didn’t want to get up. At some point there was a warm achy swelling feeling in my heart, on the actual left side of my chest. I understood for the first time why all our old phrases for emotions talk about the “heart lifting” or “heart bursting,” for the first time I noticed that these feelings are literally centered in my heart. My whole body tingled, I felt joyful and excited, my heart raced, and I felt like I was being lifted off the floor by the lightness of my joy.

Of course, as soon as this happened, I started to wonder: How do I make this last? This is clearly Jhana One, how do I progress through this to Jhana Two? And I lost it, like a kid who gets so excited the first time they balance on a bike that they fall right off. I entered Jhana One a few times over the retreat, and I found it fascinating and exciting. After so many years, something totally new was happening in my meditation.

So what? So I figured out how to make a nice tingling feeling, big whoop.

Jhourney has a slogan, something like: come for the jhanas and stay for the transformed relationship with your emotions.

Well, now that I’m back home, my zazen is transformed. I’ve gone from sitting half an hour each morning to a full hour. And the hour feels easy. Instead of resisting distracting thoughts, or coldly watching them pass by, I actually embrace them as a welcome addition to the whole messy total. Like, if I catch myself thinking about rock climbing, I just think, “I love rock climbing,” and welcome it in to quietly join the meditation. If I have a Radiohead song stuck in my head, I think, “May Radiohead be happy,” and I welcome Thom Yorke and the band to join the meditation. They usually quiet down after a while. But I’m not trying to make them be quiet, I’m welcoming them in.

Zazen isn’t a battle against my mind anymore, so I enjoy it, and find it much less effortful. I haven’t returned to Jhana One, which is a little disappointing, I was hoping to continue to later jhanas, which sound much more valuable than just tingly excitement. That’s ok, I’ll keep working on it and trying things.

Since jhanas have gotten more attention in the West in the last few years, there’s been a common reaction among Western Buddhists: trying to achieve a jhana state is striving, and it will only reinforce the ego. It’s counterproductive, it’s a symptom of the exact thing Buddha said is the source of suffering: thirst, trishna, the desire for things to be different from how they are.

It’s funny when Zen people have this reaction. It’s funny because jhana in Pali is the same as the Sanskrit word dhyana, which in Chinese is pronounced Chan, which in Japanese is pronounced Zen. We are the jhana school. It’s funny, too, that some Zen people are allergic to saying there are levels of meditation, when our koan curriculum is hundreds of ordered, numbered, pass-fail exams. Is it more dangerous to have meditation goals than koan goals?

The key to entering jhana is, in my experience, to very gently try, and not be too attached to the outcome. It’s like pissing—it’s learning a way of relaxing and letting in the joy, or letting out the piss or whatever.

The zazen instructions I’ve read have said to relax, of course, but the missing part was for me to enjoy letting the mind be as it is. The stereotype is that jhana meditation is about striving for a goal. But it’s paradoxical, like so many things in Buddhism: the more I accept my mind as it is, the more I welcome all my thoughts and emotions as they naturally arise, the less effort I make to change how I feel, the more easily I fall into a jhana.

After the Jhourney retreat I had dokusan, and a VZ teacher told me that the way I’m meditating now, after the retreat, sounds perfectly compatible with Zen. I won’t name the teacher in case they’re listening to me now and changing their mind, concluding that in fact I’ve fallen into heresy and they disown me. But I also think it’s compatible with Zen —I think the content is mostly the same, but the teaching method is totally different, and it’s what I needed to get out of my rut. That’s just upaya, skillful means: the message was better tailored for me.

Right now, I think there’s a middle path. On one side of the path, you can fall into obsessive desire to achieve something in meditation. You can compare yourself with others and get jealous. You can get attached to an experience in the past and disappointed by your experience now. That’s a danger that lots of Zen texts warn us against, and it’s a real danger. But also, I’ve found that I can get jealous and attached and disappointed, and it isn’t fatal. Eventually I get over it.

But what about the danger on the other side of the middle path? What about giving up, deciding that I’m not supposed to want to get better at meditation, that I can’t get better at meditation, that my groggy mind-wandering is the best there is? I’ve fallen down on that side of the path and spent years there, wasting thousands of hours sitting on the cushion with very little benefit. I now think it’s helpful to evaluate my meditation. After a period of zazen, do I feel relaxed, happy, focused? Do I enjoy it, do I look forward to it? Is it getting easier, more enjoyable, over the years? How does the rest of my day go after an hour of zazen in the morning?

There are dangers in asking these questions. I could get so focused on the relative that I forget the absolute—like, thinking that if my zazen doesn’t feel good, it isn’t the manifestation of Buddha mind. I can hear Dogen saying, “You are endowed with the essential functioning of the Buddha Way, why pursue worthless pleasures that are like sparks from a flint?” And that’s a great point, Dogen, we should remember that this mind has always been the Buddha mind, there is nothing to strive for and no one who can attain it. But also, it’s helpful for me to set some goals, and evaluate my progress, and distinguish what works and what doesn’t. It’s like the foot before and the foot behind in walking.

Besides, the old man backs me up! Buddha said in the Kalama Sutta, that we should evaluate various practices and behaviors for ourselves. If they lead to harm and suffering, abandon them; if they lead to welfare and happiness, enter and remain in them. Don’t just follow tradition or a teacher or scripture.

Why did it take me so long to admit my frustration with meditation, and try something new? I take responsibility for this, I’m a perfectionist, and I care what other people think of me. I get embarrassed if I don’t understand something, so I hide it.

I also think that, for someone like me, there are some dangers lurking in Zen specifically. A lot of the old koans look like competitions between monks. They’re one-upping each other, they’re pretentious, they’re quoting sutras and Chinese literature, they’re hitting each other. Nansen killed a cat because no one could answer his question. Gutei cut off his servant’s finger because his servant didn’t understand. And then the commentaries compare them to generals and swordsmen, or tigers and dragons. I like this fierceness, but it can make Zen into a competition where I don’t want to admit my weakness. I think this competitive, even violent streak, has passed from China through Japan and the samurai era all the way to today. Even at the fuzzy cuddly Village Zendo, we have this aspect. So if you’re like me, check yourself: are you stuck in a rut, because you’re afraid to admit when you don’t get it?

When I was embarrassed to talk openly with teachers, at least I had spiritual friends. The Buddha was a big fan of spiritual friendship. In the Kalyāṇamitta Sutta there’s a story where Buddha’s student Ananda says, “I’ve heard that good friends are half of the spiritual life.” Buddha replies, “No, Ananda, no, that’s not true at all. Good friends are the whole of the spiritual life. Someone with good friends can expect to develop and cultivate the noble eightfold path.” In other Pali suttas the Buddha says that monastics and householders all benefit from spiritual friendship, from kalyanamitta, that if a friend has attained a greater understanding on some subject of the dharma, we should engage our friend in discussion to learn what they know.

For me, spiritual friendships with my peers has been essential. With teachers it’s hard for me to really express my doubt and confusion, to say, “I hate this entire book of 100 koans,” “My zazen is a catastrophe,” “I wasted every minute of that sesshin thinking about work,” “I passed that koan by imitating a car horn, but I still don’t understand it.” But with my close friends it’s easy to be real. It’s a relief to admit what a mess my practice is, how I doubt myself and the practice. I find out that most people are going through this kind of thing, at least the ones I talk to, and I’m not alone and there’s nothing wrong with me. And once I’ve calmed down a bit, hope returns. I can imagine ways to change my practice, to improve it.

Let me close with some caveats and clarifications and qualifications about the jhanas, and about Zen silences. I’m not saying everyone should practice jhana meditation. I am saying that if you feel stuck, try something new, maybe try some techniques from other traditions. I’m not claiming that the teachers and ancestors created all these Zen silences deliberately. I think they mostly arise subtly and implicitly in the community. I’m not claiming these silences are absolute: sometimes we break them, different people are bolder or more cautious. I’m not even claiming they’re all worthless and we should break them: We should be cautious talking about our practice with others, we should be mindful of Right Speech. We should be careful talking about goals.

Here’s what I’m saying: Real Zen is freedom. Freedom to break silences, or maintain them, according to circumstances. It’s a human instinct to conform to the group, but Zen is liberation from that conformity, freedom to go with the flow and to go against it.

As for me, I’m going to talk more with my teachers, more with my friends, about how my practice is really going. Intimately, in detail. I won’t pretend anymore, or hide my confusion and my curiosity.

I hope you all talk more openly with your spiritual friends, and trust your own wisdom, especially when it doesn’t seem to conform.

There’s a story about when Buddha was dying, he was talking with his beloved student Ananda, who was probably panicking a bit at the thought of life without his teacher. Ananda asked how the sangha should continue without the Buddha. Buddha said everyone in the sangha must be islands unto themselves, or perhaps lamps unto themselves. The Pali word dipa is the same for island and lamp, so maybe he’s saying we must all be our own light to illuminate the path, or all be our own islands, safe in the flood of samsara.

He said, in the Mahaparinibbana Sutta:

I am frail, Ananda, old, aged, far gone in years. This is my eightieth year, and my life is spent. Even as an old cart, Ananda, is held together with much difficulty, so this body is kept going only with supports.

Therefore, Ananda, be islands unto yourselves, refuges unto yourselves, seeking no external refuge; with the truth as your island, the truth as your refuge, seeking no other refuge.

Photos © A. Jesse Jiryu Davis.

Knowledge and Common Knowledge in a Distributed Environment, Part 1

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 13 Aug 2025 11:24:35 +0000

Saraswati, goddess of knowledge

We usually reason about distributed systems by asking, what can one process know about the state of the system? E.g., if a majority of Raft nodes tell the leader that they have replicated a log entry, the leader knows the entry is durable. Therefore the leader can take various actions, like replying to the user who submitted the log entry. On the other hand, if a Raft follower hasn’t heard from the leader in a while, it might suspect the leader has crashed, but it can never know for sure. A process must somehow gather enough information to deduce the facts it must know to take the correct action. I think about knowledge all the time when I’m analyzing a distributed algorithm.

On the other hand, my main tool for checking the correctness of an algorithm is TLA+ and the TLC model-checker. There’s no notion of knowledge in a TLA+ specification, just a mindless state machine obeying if-then instructions. When I read a TLA+ spec I might think, “In this state, the leader knows the log entry is majority-replicated.” But that’s anthropomorphism. The spec just says: if this, then do that.

I’m studying the 1990 paper, “Knowledge and Common Knowledge in a Distributed Environment”, by Joseph Halpern and Yoram Moses. They bring a theory of knowledge and belief to distributed systems. It’s a hard paper for me, so I’ll blog it as I read, a few sections at a time. My goals are to understand the material and explain it to you. I also want to explore a new direction: integrating knowledge into TLA+. This might be a dead end, we’ll see!

Muddy Children

The paper begins with the muddy children puzzle, apparently a classic of epistemology:

Imagine n children playing together. The mother of these children has told them that if they get dirty there will be severe consequences. So, of course, each child wants to keep clean, but each would love to see the others get dirty. Now it happens during their play that some of the children, say k of them, get mud on their foreheads. Each can see the mud on others but not on his own forehead. So, of course, no one says a thing. Along comes the father, who says, “At least one of you has mud on your head,” thus expressing a fact known to each of them before he spoke (if k > 1). The father then asks the following question, over and over: “Can any of you prove you have mud on your head?” Assuming that all the children are perceptive, intelligent, truthful, and that they answer simultaneously, what will happen?

Halpern and Moses introduce this puzzle to warm up the reader’s thinking about knowledge in distributed systems. When is it important for one process to know what another process knows? What is “common knowledge” and when is it needed?

The solution to the muddy children puzzle is: the first k - 1 times the father asks the question, all children simultaneously reply “no”: they can’t prove they’re muddy. The k^th time, the muddy children say “yes” and the clean children say “no.” But if the father hadn’t announced “at least one is muddy” at the start, the muddy children would never figure out they’re muddy. Even when k > 1, and all the children can see with their own eyes that at least one is muddy, the father’s announcement of this fact is crucial.

This result is well-known to logicians, so the paper doesn’t explain it in depth. But I will. Let’s work the problem for k from 1 to 3. (The number of clean children is actually irrelevant.)

Here are our variables:

k: the number of muddy children.
m: the father’s announcement, “at least one of you is muddy.”
q: the number of times the father has asked, “can you prove you’re muddy?”

One muddy child

The first time the father asks the question, the muddy child sees that all the other children are clean. Since the father announced m, the muddy child knows it must be the muddy one, and says “yes.” Incidentally, the muddy child knows k = 1. The other children see one muddy child. They don’t know whether they’re also muddy (they don’t know if k is 1 or 2), so they say “no”.

The purpose of the father’s announcement m is clear here: without it, the muddy child doesn’t know if the reason it sees only clean children is because k = 0, or because k = 1 and it has mud on its face.

Two muddy children

Here’s where things get freaky. The father announces m, which all the children can already see is true, and yet his announcement is necessary for solving the puzzle.

When the father asks his question the first time, can Child a answer “yes”? It sees that Child b is muddy. It considers some possibilities:

a could have mud on its own face or not.
b could think it has mud on its own face, or not.

These two pairs of possibilities lead to four possible worlds in a’s mind:

But the father’s announcement m is “common knowledge”: that is, everyone knows that k ≥ 1, and everyone knows that everyone knows … (ad infinitum) … that k ≥ 1. So a knows b knows the world where all children are clean is impossible:

Now the father asks both children, “Can you prove you’re muddy?” Since a could be muddy or clean, depending on which world is real, it replies “no.” Child b replies “no” for the same reason (swapping a and b).

Child a hears b’s answer, and now a has learned something. Child a knows that b sees a muddy child! If b hadn’t seen a muddy child, then b would’ve said “yes”, because of m. This eliminates the other world where b sees no muddy child:

All worlds where a is clean are now eliminated, so the next time the father asks his question, a replies “yes”: it knows it’s muddy. Child b does the same.

So the father’s announcement of m is crucial to a, even though a knows k ≥ 1 already: m ensures that a knows b knows k ≥ 1.

Three muddy children

I’ll work this problem similarly, but without diagrams. At the start, before the father announces m, a sees 2 muddy children, so:

Initial condition, before father announces m:
a knows:
  k ∈ [2, 3]
  b knows:
    k ∈ [1, 3]
    c knows:
      k ∈ [0, 3]

The text above is my attempt to represent statements like, “a knows k is between 2 and 3 inclusive, and a knows b knows k is between 1 and 3 inclusive,” and so on. Why does a know b knows that? Because:

a knows b sees mud on c (definitely 1 muddy child)
b could think its own face is muddy (maybe 1 more muddy child)
b (in a’s mind) might see mud on a (maybe 1 more muddy child)

Hence b (in a’s mind) could think there are 1, 2, or 3 muddy children.

How could a think b thinks c thinks there are 0 muddy children, even though a knows c sees the mud on b’s face? Well, if a is clean (in a’s mind), and b thinks its own face is clean (in a’s mind), then b thinks c might see no muddy children (still in a’s mind). So a thinks it’s possible for b to think c sees no muddy children, although a knows that’s wrong!

But then the father announces m. Now it’s common knowledge that k ≥ 1:

After father announces m:
a knows:
  k ∈ [2, 3]
  b knows:
    k ∈ [1, 3]
    c knows:
      k ∈ [1, 3]

The father asks the question, “can you prove you’re muddy?”, once:

After father announces m, and q=1:
a knows:
  k ∈ [2, 3]
  b knows:
    k ∈ [2, 3]
    c knows:
      k ∈ [1, 3]

Child c says “no,” which must mean it sees a muddy child. Child a knows b knows c is muddy as well, so a knows b knows k ≥ 2.

The father asks a second time:

After father announces m, and q=2:
a knows:
  k ∈ [3, 3]
  b knows:
    k ∈ [2, 3]
    c knows:
      k ∈ [1, 3]

Since b said “no,” and a knows b knew k ≥ 2, a knows b sees 2 muddy children. Child a also knows b is muddy, so a knows k ≥ 3. Since a only sees 2 muddy children, a knows the third muddy child is itself and k = 3. The third time the father asks, a says “yes.” The other muddy children are reasoning identically to a, so they also say “yes.”

The battle of wits in The Princess Bride.

k muddy children

We’ve seen that for each k from 1 to 3, when the father asks the question for the k^th time, all muddy children answer “yes” and the others answer “no.” Let’s say a sees 3 muddy children and assumes it has no mud on its own face, i.e. it assumes k = 3. If the father asks 3 times and all children still answer “no,” then a knows its assumption was false, so a knows it’s muddy and k = 4. Child a then correctly answers “yes” to the 4^th question. Same for the other children. And so on for all k > 4, inductively.

A hierarchy of states of knowledge

We’ve seen a weird phenomenon, where the father’s announcement of a fact that everyone already knows somehow gives the children useful information. I explained to myself and to you how this works in the muddy children puzzle. Halpern and Moses explain it in general, by defining a hierarchy of states of knowledge. (I think this is based on prior work in philosophy, and the authors are porting it to computer science.) To begin, they introduce the notation:

This is read, “i knows .” Some agent (or process or whatever) i knows a fact (Greek letter “phi” for “fact”). I’ll discuss the authors’ definition of knowledge later, it’s terrific, stay tuned! For now, let’s just say an agent’s knowledge can depend only on the agent’s local history, i.e. its initial state and the actions it’s taken and observed. Also, knowledge is always true. A belief can be false, but if an agent knows , then is a true fact.

Here’s the authors’ hierarchy of knowledge, from weakest to strongest:

“the group G has distributed knowledge of ”

A fact is distributed knowledge if someone with a global view could infer from everything known by every agent in some group G, even if no individual agent in G knows.
“someone in G knows ”

Defined as: for some i in G.
“everyone in G knows ”

Defined as: for all i in G.
“ is -knowledge in G”

Everyone in G knows that everyone in G knows that … everyone in G knows , where “everyone in G knows that” is repeated k times.
“ is common knowledge in G”

Defined as, for all k ≥ 1.

The authors point out, using this framework, that when the muddy children puzzle begins, is true. E.g., if there are two muddy children, then everyone knows m. If there are three muddy children, everyone knows that everyone knows m, because everyone sees at least 2 muddy children, so everyone knows everyone else sees at least 1 muddy child. But to solve the puzzle, they must upgrade their knowledge from to , which is what the father’s announcement does. (The father’s announcement goes farther, making m common knowledge, but all he must do is upgrade m by one level, to .)

This is a useful way to think about nodes in a distributed system: each has limited knowledge, but there is distributed knowledge implicit in the whole system. To correctly take certain actions, nodes need a certain level of knowledge or higher. Nodes exchange messages to promote their knowledge up the hierarchy. Each level in the hierarchy implies all the lower levels:

Knowledge hierarchies in Raft

Halpern and Moses don’t talk much about actual distributed protocols, but I kept thinking about Raft—how does the Raft protocol look if it’s recast as a flow of knowledge?

Let’s say a Raft leader creates a log entry. I’ll call the fact that the entry exists , and right now only the leader knows .

The leader sends the log entry to its two followers, but they haven’t acknowledged it yet:

Now everyone knows , but the leader doesn’t know any follower knows . Let’s say G is the set of all nodes and F is the set of followers. Using Halpern and Moses’s notation:

Let’s call the fact “the entry is durable” , the Greek letter psi. This fact is true if the entry is replicated to at least a majority of nodes, so it’s certainly true if it’s replicated by all nodes—that is, if everyone knows then is true. But currently only God can see that all nodes have the entry, so is distributed knowledge.

Then the leader receives an acknowledgment.

We can say the leader knows if it knows any follower knows :

(This is true because the leader + one follower is a majority. If there were more than 3 nodes we’d need a different rule.) Now has been upgraded, from distributed knowledge to something that someone knows:

The leader tells the followers that the entry is committed.

Now has been upgraded again, from a fact that someone knows to a fact that everyone knows.

But (according to the Raft paper) followers don’t tell the leader which entries they know are committed, so doesn’t become something that everyone knows that everyone knows, much less common knowledge.

Onward

This covers the first three sections of the paper. We’ve seen Halpern and Moses’s knowledge hierarchy, and how it’s useful for analyzing the muddy children puzzle and Raft. Next, we’ll briefly visit the famous Byzantine Generals, who are still trying to decide when to attack their common enemy. After that we’ll get into the meat of the paper, which offers a remarkably satisfying definition of knowledge.

Lakshmi and Saraswati

Optimizing a Meditation Retreat with a SAT Solver

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 29 Jul 2025 22:00:52 +0000

Well, a SAT solver can optimize the retreat schedule. A meditation retreat as a whole is a living creature, beyond any individual’s control. But I can provide a schedule that functions smoothly and reduces distractions. I used Python and integer programming, a technique for mathematical optimization.

The problem

The Village Zendo holds two weeklong Zen meditation retreats every summer, and one every winter. An essential part of Zen retreats is interviews (“dokusan” or “daisan”) between students and teachers, where students can ask questions or present their understanding of koans. At a Village Zendo retreat, we want each student to see one teacher per day. We assign students to groups, and call each group out of the meditation hall to see a teacher at the start of a meditation period. Certain students serve as jishas; they’re in charge of calling a group of students and escorting them to the door of the interview room. The students in the group line up, and enter the room one at time for a short conversation with the teacher.

Besides ensuring one interview per student per day, there are many other rules and goals. Dear reader, if you’re a programmer, you are probably already beginning to model this as an optimization problem. The objects in the system are:

Students. Some of the students are jishas, and some have other jobs which affect the optimization problem.
Teachers. This summer there were 8 teachers for 35 students.
Rooms. This summer there were 2 interview rooms. Our abbot used his own room for interviews.
Groups. Each student belongs to one group.
Slots. There are time slots for interviews in the morning, afternoon, and evening meditation periods.

The challenge is to assign students to groups, then schedule jishas to bring groups to teachers during slots, so that all groups see a teacher once a day. Ideally, a group sees a variety of teachers during the week, and the workload is spread evenly among the teachers, and evenly among the jishas.

For the last few years my fellow Zen student, the mathematician Volker Kenko Ecke, had been manually solving this problem for each retreat with a big spreadsheet. He assigned students to groups, groups to slots, etc. by hand, then wrote formulas to check that all the rules and goals were satisfied. It was impressive, but I felt the itch to automate it. This is the difference between mathematicians and programmers.

Translating the problem into SAT

Last summer I offered to write a program. This was not a problem for LLMs. I vaguely knew that a SAT solver was the natural tool. I chose Google’s open source CP-SAT solver, which solves integer programming problems: optimization problems involving assigning integers to variables, constrained by simple formulas. It has a nice Python API.

To translate the interview scheduling problem into Solverese, I made two big binary matrices. One is group assignments: the columns are groups, the students are rows, and there’s a 1 wherever a student is assigned to a group and 0 everywhere else.

students = [
    Student(last="Davis", first="Jesse", dharma_name="Jiryu"),
    Student(last="Cohen", first="Steven", dharma_name="Jindai"),
    # ... and so on ...
]

MAX_STUDENTS_PER_GROUP = 5
groups = [Group(number=i) for i in range(ceil(len(students) / MAX_STUDENTS_PER_GROUP))]

from ortools.sat.python import cp_model
model = cp_model.CpModel()

# Matrix of bools, each is true if a student is part of a group.
group_assignments = {
    (st, g): model.new_bool_var(f"{st.display_name} group {g.number}")
    for st in students for g in groups}

As you can see, you create a variable with model.new_bool_var. Later I’ll show you how these are used in constraints like “each student is in exactly one group.”

Besides the two-dimensional group assignment matrix, the other binary matrix has four dimensions: teachers, jishas, groups, and slots. If a certain jisha takes a certain group to see a certain teacher at a certain time slot, then that cell of the matrix has a 1, otherwise 0. I called these combinations shifts.

# Matrix of bools, each is true if a teacher, jisha, and group is assigned
# to a slot. # Key is a Shift object, value is a variable to which the
# solver assigns a value.
shifts = {Shift(t, j.display_name, g, s): model.new_bool_var()
    for t, j, g, s in product(teachers, jishas, groups, slots)}

Creating a schedule is equivalent to assigning 1s and 0s to the cells in these two matrices, in a way that satisfies all the constraints and optimizes some goal. So I started adding constraints.

Expressing constraints

First, a student can only be in one group. That means the sum of group assignments along a student row is 1:

# Each student is in 1 group.
for st in students:
    model.add(sum(group_assignments[(st, g)] for g in groups) == 1)

And each group can only have MAX_STUDENTS_PER_GROUP members:

for g in groups:
    # Enforce MAX_STUDENTS_PER_GROUP.
    model.add(sum(group_assignments[(st, g)] for st in students)
              <= MAX_STUDENTS_PER_GROUP)

Giving interviews is tiring for teachers, they should only have one shift per day. For each teacher/day combination, I create an integer variable with model.new_int_var, constrain it to equal the number of shifts for that teacher on that day, and also constrain it to be less than or equal to one:

days = list(sorted(set(s.day for s in slots)))
for t in teachers:
    n_teacher_shifts = sum(
        v for s, v in shifts.items() if s.teacher == t)
    for d in days:
        shifts_today = model.new_int_var()
        model.add(shifts_today == sum(
            v for s, v in shifts.items()
            if s.teacher == t and s.slot.day == d))
        model.add(shifts_today <= 1)

There are a dozen other rules, and they’re all similarly easy to express. For example:

A teacher can only see one group per slot.
The abbot prefers to do interviews only in evening slots.
The abbot has a particular jisha; other jishas are free-floating.
Jishas should only do one shift per day.
Each group should get one interview per day, and see each teacher only once during the retreat.
Since there are two interview rooms, so only two teachers can do interviews per slot (but the abbot uses his own room).
Some students are monitors. At least one monitor should be in the meditation hall at all times.
… and so on.

Once I banged the model into shape, I told the solver to find a solution:

solver = cp_model.CpSolver()
# Determinism.
solver.parameters.random_seed = 1
# Stop trying after about 5 minutes - but use a virtual clock
# that always stops after the same number of operations.
solver.parameters.max_deterministic_time = 300
solver.parameters.num_workers = 1
status = solver.solve(model)

The “determinism” arguments ensure I get the same solution each time I run the same code, which helps me debug.

Refinements

Once the program basically worked, I generated a PDF of the schedule and the group assignments, and made some improvements. For example, could I reduce the jishas’ workload? A jisha leaves the meditation hall for two reasons: if she leads a group of students to see a teacher, and also if her own group is called to see a teacher. On past retreats, a jisha might lead Group 1 to Teacher A, while her own Group 2 was seeing Teacher B. This led to hectic improvisation; not very peaceful. I called these “jisha conflicts” and told my program to eliminate them. This was a bit complex. Recall that every “shift” is a combination of teacher, jisha, group, and slot. So for every shift S1, and every other shift S2 with the same slot but a different group, either S1’s jisha isn’t in S2’s group, or S2’s group isn’t doing interview during this slot.

# Avoid jisha conflicts: when a jisha is serving one teacher while
# their group is called # for another. To prevent conflicts, if a
# jisha j is assigned to a shift it implies # j isn't in groups seen
# by other teachers in that slot.
for shift in shifts:
    for t, j, g in product(teachers, jishas, groups):
        if g == shift.group:
            continue
        # Another shift with the same time slot but different group.
        other_shift = shifts[Shift(t, j.display_name, g, shift.slot)]
        # If shift is true, then other_shift is false or shift's jisha
        # isn't in other_shift's group.
        j_in_other_group = group_assignments[
            (name2student[shift.jisha_name], g)]

        model.add_bool_or([
            j_in_other_group.negated(),
            other_shift.negated()
        ]).only_enforce_if(shifts[shift])

This was slightly mind-bending to code, but it only took a few hours of thinking and reading the CP-SAT docs. It eliminated a historical source of stress for the jishas.

I wondered, could I simplify jishas’ lives even more? For each jisha I calculated the number of extra trips she took out of the meditation hall: i.e., the number of times she led out a group of students besides her own. I told the model to minimize the sum of the squares of the extra trips, so it would try to both reduce the workload and spread it evenly. It’s about 20 lines of code, and it reduced the average jisha’s time out of the meditation hall from 3 trips to 0 or 1. This was another big improvement over past retreats.

As I said, a SAT solver can’t optimize your meditation, but it can help create good conditions!

Minimizing changes

I’ve been on dozens of Zen retreats, and I’ve been involved in organizing some of them. Constant change is the norm, just like Buddha taught. People join late, leave early, get sick. And maybe we realize partway through the retreat that we should change the program, add a new constraint or goal we didn’t think of until we put the schedule to use. Once my program had made a schedule, I knew it wouldn’t be “final.”

So, each time the program runs, it checks the time. If the retreat hasn’t started, it can make a new schedule from scratch, freely reassigning groups and shifts. Once the retreat starts, the groups are permanent, and the program must do its best to meet its goals without changing group assignments. Shifts are malleable so long as they’re in the future, but the program knows it can’t change the past. So if we suddenly decided some group of students must see the abbot, the program can satisfy that only by changing shifts starting after the current time.

Retrospective

I’ve scheduled four weeklong retreats with this program, improving the code each time. (I put a version with participants’ names redacted on GitHub.) I’ve learned what questions to ask the registrar, the abbot, and the retreat managers before the retreat begins, so I create a schedule that fits their requirements. I still have to personally edit and run the code—this is a script, not a software product—but it’s becoming routinized.

The program generates a nicely formatted PDF of the schedule, that we can print out and post on the wall at the retreat center:

The PDF includes a timestamp and a fingerprint (a hash of all the variable assignments), to avoid confusion when there are multiple printed copies floating around among the retreat staff.

The program also generates a PDF of group assignments, and a card with each student’s individual schedule. It also outputs a text file with statistics, like how many shifts each teacher and jisha has, so that the retreat managers and I can evaluate the quality of the solution.

The smartest decisions I made were the first ones: to encode the schedule as two boolean matrices and use a SAT solver. Now, I hardly ever have to think, “How do I solve this problem?” I only have to figure out, “How do I express this constraint or goal in terms of CP-SAT?” Furthermore, I spend little time checking the solution. If I code a new constraint, then I spend a few minutes checking that the latest schedule satisfies it as I expect, but after that I mostly trust the program. Even when I make a consequential change, like adding a constraint or removing a day, I’m confident the program will rearrange the puzzle pieces to fit. If I had tried to write my own scheduling algorithm, changes would be much harder and riskier.

I was particularly satisfied last week, when a teacher got sick and canceled on the first day of retreat. With the old handwritten spreadsheet, I think it would’ve been hard to adapt the schedule, and there wouldn’t be enough time to make a really good one. But now that it’s automated, I simply deleted the sick teacher and reran the program. It found no solution, so I relaxed a constraint—now some teachers will have to take two shifts per day—and voilà, a perfectly good new schedule popped out 5 minutes later.

Aerial Silks in the Gunks

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 28 Jul 2025 15:10:02 +0000

July 26, 2025. Kelsey Roman performing with aerial silks suspended from the Nectar Vector climb on Millbrook, a cliff in the Gunks. Rock guide Dustin Portzline is above, managing the rigging and holding a parasol.

Images © A. Jesse Jiryu Davis

RIP Sojourner

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 07 Jul 2025 22:16:48 +0000

Sojourner Truth Armstrong Davis, Jennifer’s and my Chinese dwarf hamster, died last night. She had lived with a cancerous affliction of the womb, a malady common to her kind, for several months. We had expected her demise soon after the first signs—hamsters rush headlong through all activities, and life itself—but she endured for a season, bearing her disease with the same insouciance she displayed in all circumstances.

Soji joined our household a year and a half ago, leaving behind her infant siblings at PetCo to commence a solitary existence. Experience had informed us that sisterly companionship harbored risk, so Soji’s house was a hermitage from the day of her arrival. Her society consisted chiefly of two humans, who held her in the deepest affection. Whether our fondness was reciprocated in equal measure is a matter of speculation; yet she would settle contentedly into our hands, emitting her customary clicks and chirps, as if to signify approval.

The original Sojourner Truth was born a few towns distant from us, and in her sixtieth year commenced a tour of the nation, speaking against slavery. Sojourner the hamster was also an outspoken creature. Most afternoons she stationed herself within her modest wooden dwelling, and from this redoubt delivered an oratory of squeaks, clucks, and chirps loud enough to be heard from the adjoining room.

Unlike her namesake, Soji addressed an audience of none. No observer, however, would ever mistake her isolation for loneliness. She was happier than any hamster we had known before, for which the likeliest explanation was her assured seclusion. She was a connoisseur of aloneness. Assured that her territory was uncontested, and eternally unencumbered by any relative or rival, she spent her days in appreciation of her boundless solitude.

Images © A. Jesse Jiryu Davis

From Python Programmer to Distributed Systems Researcher in 10 Years Without a PhD

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 30 Jun 2025 22:29:35 +0000

Phil Eaton wrote “From Web Developer To Database Developer In 10 Years” recently. I was inspired to write my own version.

In high school, I wanted to be a modern dancer, with a company like Pilobolus or Streb. But I also kind of wanted to be a programmer. Pixar’s Toy Story, the first 3D-animated feature film, was released when I was a high school sophomore. I was fascinated by this simulated world, revealed by simulated light to a simulated camera. How did they make it? I somehow learned (however I learned things before I had Internet access) that 3D graphics were rendered with linear algebra and C++. So I got books about both from my neighborhood library in Shaker Heights, Ohio. I don’t recall much about the linear algebra book, but the C++ book is a core memory: a giant glossy volume, with a copy of Borland Turbo C++ for DOS on a CD in a slip attached to the inside back cover. I installed it on the family computer, a Tandy with a 386 processor, which looked like this:

When I installed Turbo C++, the IDE looked like this:

I taught myself basic C++, but on my own I didn’t manage to learn 3D graphics.

Meanwhile, I reached my senior year of high school, and auditioned for the prestigious dance program at Ohio State University. I was admitted to the waitlist, but that was a superficial victory. I saw in that audition another man my age who was a true dancer. He was unselfconscious, emotional, natural, free. I would never dance like that. By the time I got a letter from OSU, saying I’d been accepted from the waitlist, I’d already decided to major in computer science and do 3D graphics for a living.

(Is this what really happened? It’s been 28 years. This is how I remember and tell my life story now.)

I got my CS degree from Oberlin College with a focus on 3D graphics. For some reason I didn’t apply to Pixar. Why didn’t you follow your dream, young Jesse? I ended up in Austin, Texas, working for a tiny company that made flight safety analysis software. At least I was doing 3D graphics: our software turned data from jets’ flight recorders into 3D animations, so airlines could review near-accidents and adjust their procedures.

Let’s fast-forward. I bounced around from a Zen monastery to an NYC education startup, then freelanced for a few years, and joined MongoDB in 2011 (“10gen” at the time). I’d abandoned 3D graphics by now. I’d learned Python, and I used MongoDB at one of my freelance gigs—it was so much more fun than SQL, I thought the idea might have legs. MongoDB was unique in NYC at the time: the only company building technically demanding open-source software in the city. Google and Amazon hadn’t yet opened offices here. MongoDB hired me as a developer advocate promoting MongoDB to the Python community. They called me a “Python evangelist”—my mom loved the title, she said it sounded like I was a snake handler.

Handling serpents at the Pentecostal Church of God, by Russell Lee.

I made myself a minor celebrity in the Python world, speaking at PyCon many years in a row and publishing a lot of popular Python content on this blog. But mostly I was an engineer. I worked on PyMongo, I created the async alternative Python driver Motor, and for a while I led development of our C and C++ drivers. I was promoted a few times, to Staff Engineer. After 7 years on the Drivers Team, I hit a plateau—my learning had slowed and I didn’t know how to get another promotion. So I decided to switch teams. I rotated through three MongoDB teams, trying each one for a month or two, until I settled on the Replication Team, which maintains MongoDB’s consensus protocol, deep in the core database server.

The Replication Team transformed my career, because they read and wrote research papers about distributed systems. Before, I had deepened my knowledge by studying programming languages. Now, I was learning about much higher-level concepts: consensus, fault tolerance, causality, isolation and consistency. The team met regularly to read papers in our field, both new and old. I got excited and read more papers on my own, and joined an external distributed systems reading group. I also met TLA+ for the first time. With two colleagues, I published my first research paper, about testing the conformance between TLA+ specifications and implementations. (I’m still researching this topic, 5 years later.) I made a plan to read dozens of papers and a few textbooks; my boss and grandboss agreed that if I completed the curriculum I’d designed for myself, I’d be promoted to Senior Staff Engineer, and soon enough this actually happened.

In 2022 I took most of a year off to climb, and moved to New Paltz, a world class climbing mecca. While I was away, I considered my reentry. How could I contribute best to MongoDB? And how could I keep my calendar clear of team standups and planning meetings and all the other cruft that would prevent me from running out to the cliff when the weather was good? MongoDB had a small research group, almost entirely in Australia at the time, led by Michael Cahill. I asked to join it when I returned, working four days a week at four fifths of my previous salary, and Michael assented. I was gambling with my career: I only had a BA, I had only published one paper, and my primary mentor would be on the other side of the world. On the other hand, I could use the writing and speaking skills I’d learned as a “Python evangelist,” and devote myself to the distributed systems theory I enjoyed. I returned to MongoDB in December 2022 as a Senior Staff Research Engineer.

How did I get this coveted position—the freedom of academia with an industrial salary? I didn’t have an advanced degree, but I had some advantages. My previous boss supported my decision to transfer. I’d been at MongoDB for 11 years, a long time at any company, especially a fast-growing one like ours. The one paper I’d published was good, and accepted by a top conference. I’d not only read a lot of papers, I’d spread my knowledge by leading discussions of them in reading groups. And I came to Michael with ideas he liked, such as predictive auto-scaling or improving the consistency of MongoDB’s secondary reads.

So that’s the story: that’s how I progressed from Python programmer to distributed systems researcher in ten years, without a PhD. Since then, a lot has happened. Michael Cahill retired soon after I joined the research group, and I was unmentored for a year. Luckily I was able to hire a famous expert, Murat Demirbas, to lead MongoDB’s distributed systems research. I’m still finding my stride in my new career, but at least I have guidance.

Taming Google Scholar Alerts

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 23 Jun 2025 17:09:48 +0000

Since I joined MongoDB’s Distributed Systems Research Group a few years ago, I’ve relied on Google Scholar alerts to tell me when an author in my specialty publishes a new paper, or (much more often) when a new paper cites an author I’m interested in. Google Scholar can send me email alerts, but their frequency isn’t configurable: daily only. And if the same new paper is included in multiple alerts (e.g., it cites several authors I follow), Google Scholar won’t deduplicate it. It’s common for me to get a half-dozen emails in the morning about overlapping sets of papers.

I’ve made a little Google Apps Script automation that solves these problems for me. Now, I’ve configured a GMail filter so my daily Google Scholar alerts skip my inbox and don’t bother me. Once a week, my script summarizes and deduplicates all the alerts, then deletes them from my GMail archive. This turned a daily chore of cleaning up my inbox into a much easier weekly chore: skimming the titles of this week’s papers, in a single email.

Image by Edward Bird.

Gunks, June 2025

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 23 Jun 2025 10:13:58 +0000

Climbing in the Gunks on June 21 and 22, 2025.

Images © A. Jesse Jiryu Davis

Genuine Climbing retreat for women with cancer

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 15 Jun 2025 07:53:34 +0000

June 2025. A climbing retreat in the Gunks for women with cancer led by Anyssa Lucena of Genuine Climbing. Most of the women had never climbed outside, or never climbed at all. Their awe of the rock reminded me of when I fell in love with climbing, years ago. Eric Bates and a small crew were shooting a documentary of the retreat. He made Five Dot Fun, one of my favorites; I was excited to meet him and watch him work.

Shot on a mix of digital and Portra 400.

Images © A. Jesse Jiryu Davis

Opus 40

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 14 Jun 2025 10:20:00 +0000

Opus 40, a sculpture park in Saugerties, New York, hand-built by Harvey Fite over 37 years from local bluestone.

Images © A. Jesse Jiryu Davis

Can We Rely On Timers For Distributed Algorithms?

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 05 Jun 2025 10:45:23 +0000

Distributed systems people are suspicious of timers. Whenever we discuss an algorithm that requires two servers to measure the same duration, the first question is always, “Can we rely on timers?” I think the answer is “yes”, as long as you add a safety margin a little over 1/500^th.

Example: Leader Leases Rely on Timers

Leader-based consensus protocols like Raft try to elect one leader at a time, but it’s possible to have multiple leaders for a short period. (In theory, in a Raft group of 2f+1 servers, as many as f can be leaders at once!) In this situation, you risk violating read-your-writes:

A client updates some data on the current (highest-term) leader.
That leader majority-replicates the change and acknowledges it to the client.
The same client reads stale data from an old leader that hasn’t stepped down.

A Raft leader must always suspect that it’s been deposed, so it acts paranoid: for each query, it checks with a majority to confirm it’s still in charge (Raft paper §8). This guarantees read-your-writes at a high cost in communication latency.

A leader lease guarantees a single leader, but it relies on timers. In Diego Ongaro’s thesis, he proposes a simple lease mechanism for Raft: The leader starts a timer at time t, and sends heartbeat messages to all its followers. Once a majority has responded, the leader knows they won’t vote for another leader until t + election timeout * ε, where ε is the maximum rate of clock drift. Here’s Figure 6.3 from Ongaro’s thesis:

At MongoDB we made a more sophisticated lease algorithm that improves availability. Stay tuned for our research paper. But just like Ongaro’s, our algorithm depends on timers.

Why We Distrust Timers

In the classic asynchronous system model, servers have no clocks and delays are unbounded. FLP impossibility says that in this model, no consensus algorithm is guaranteed to make progress. I believe this means no protocol can both 1) always eventually elect one leader and 2) prevent multiple concurrent leaders. This is what makes Raft leaders paranoid.

But if we relax the model slightly—allowing bounded timer inaccuracy—we can design more efficient protocols. In the timed asynchronous system model, servers use timers with bounded drift. This is, as I’ll argue below, a more realistic model, and it enables leader leases: a majority promises not to elect another leader for a duration. The leaseholder can serve reads without checking with peers.

At MongoDB Distributed Systems Research Group, we’re developing lease protocols for Raft and MongoDB. So I spent a few days researching this question: how reliable are timers?

Timer uncertainty

With leader leases, the leader starts a timer for some duration d, and sends a message to its followers telling them to start timers for the same duration. Leases guarantee consistency so long as the leader believes its timer expires before any of its followers believe theirs do. I formulated this as the Timer Rule:

If server S₀ starts a timer t₀,
then sends a message to server S₁,
which receives the message and starts a timer t₁,
and S₁ thinks t₁ is ≥ d * ε old,
then S₀ thinks t₀ is ≥ d old.

We should make the safety margin ε large enough to guarantee this, but not unnecessarily large (which would hurt availability). Here are some sources of uncertainty:

Clock frequency error, a.k.a. “drift”

Any server’s timer depends on a quartz oscillator (“XO”) on its motherboard, even if the server is a VM. All XOs are manufactured with some inaccuracy, and their speed is affected by age and temperature. XOs slow down if they’re too cold or too hot! Cloud providers control temperature fairly well in their data centers and they swap out components periodically, but not all servers are so well cared for.

NTP clients (ntpd or the more modern chronyd) measure oscillator drift over time (even across reboots) and compensate for it, disciplining the oscillator to near-perfection. In the last couple years, cloud providers have achieved clock synchronization within tens of microseconds, implying minuscule clock drift. However, for maximum safety let’s assume that NTP isn’t functioning at all. The servers’ timers are undisciplined and freely drifting.

It’s weirdly hard to find the model numbers for XOs commonly used in servers, but I think I found some examples (1, 2, 3). They usually advertise a maximum drift of ±50 ppm (parts per million) over a vast range of operating temperatures. (See this formula for the accuracy of XOs in general.) This matches the 50 ppm value that the AWS ClockBound engineers consider worst-case.

Clock slewing

When ntpd or chronyd make routine clock adjustments, they slew it gradually to the correct time instead of stepping it discontinuously. The max slew rate for ntpd is 500 ppm hardcoded, for chronyd it’s 1000 ppm configurable.

Every few years the authorities announce a leap second to account for recent unpredictable changes in the earth’s rotation. It is announced 6 months ahead. Leap seconds will probably be canceled starting in 2035, in favor of a leap hour every few thousand years. If there are any leap seconds before 2035, different time providers may unfortunately implement them differently. AWS and GCP are likely to slew the clock for 24 hours (a 7 ppm slew rate) and Azure may ignore the leap second, introducing a 7 ppm discrepancy between timers in Azure and non-Azure during the 24 hours. But we can ignore leap seconds; we only care about the max slew rate, regardless of the reason for slewing.

Bottom line: the max slew rate is 1000 ppm.

VM interruptions

Hypervisors have paravirtual clocks: the VM’s clock is a passthrough to the host clock, so when the VM wakes from a pause its clock is still up to date. AWS and GCP use the KVM hypervisor, which has kvm-clock. Azure uses Hyper-V, which has VMICTimeSync. Xen is going out of style, but it has something poorly documented called pvclock. If a VM pauses and resumes, its paravirtual clock won’t be affected. If it’s live-migrated to a different physical host, then presumably the accuracy of its timers across the migration depends on the clock synchronization between the source host and target host. Major cloud providers now sync their clocks to within a millisecond, usually much less.

Martin Kleppmann warns about checking a timer and then acting upon the result: your VM could be paused indefinitely between the check and the action. But his article is about mutual exclusion with a lease, and we’re just trying to guarantee read-your-writes. For us, the server only needs to be a leaseholder sometime between receiving a request and replying.

So what’s ε?

Ideally, clock slewing cancels out frequency error to approximately zero. After all, the NTP client slews the clock in order to get it back in sync. But let’s pessimistically assume that clock slewing adds to frequency error. The sum of the errors above is 50 ppm (max frequency error) + 1000 ppm (chronyd’s max slew rate) = 1050 ppm. If two servers’ clocks are drifting apart as fast as possible, that’s 2100 ppm, or a little over 1/500^th. So if we set ε = 1.0021, we’ll definitely obey the Timer Rule above. For example, if election timeout is 5 seconds (MongoDB’s default in Atlas), this means waiting an extra 11 ms to be sure.

How important is ε?

We could probably get away with ignoring drift, and set ε = 1, and still never violate the Timer Rule. Recall the scenario I care about: S₀ starts a timer and sends a message to S₁, which starts its own timer. S₁’s timer mustn’t expire before S₀’s. There’s some builtin safety already, because S₁’s timer will start after S₀’s. The delay depends on network latency and processing time on each server; it will probably be much larger than timer inaccuracy.

If we rely on leases for consistency, then even if S₁ thinks its timer expires too soon, it’s still hard to observe a consistency violation. There’s only a short time while S₀ thinks it still has a lease, but S₁ thinks S₀’s lease expired. During this window all the following must happen:

S₁ runs for election, and wins.
S₁ receives a write command, majority-replicates it, and acknowledges it to the client.
S₀ receives a query for some data that was just overwritten on S₁.
S₀, thinking it still has a lease, runs the query on stale data and replies.

This sequence would violate read-your-writes. It’s hard to imagine all those events in the milliseconds or microseconds between the two timers’ expirations. It seems more likely to me that the Timer Rule is violated, not because ε is a smidgen too small, but because some misconfiguration makes a timer wildly inaccurate.

In MongoDB’s version of Raft, the violation is even less likely, because the MongoClient will know that S₀ has been deposed. You have to use multiple MongoClients, or restart your client application, to lose this information and observe the consistency violation. See my causal consistency article for a technique to handle this case.

In summary, a user would have to be very unlucky to observe a consistency violation with leases: they’d need a Raft group with two leaders due to a network partition, but the user can talk to both sides. They’d have to use two clients (to disable MongoClient’s deposed leader check), write to the new leader, and quickly read the overwritten data from the old leader. The old leader would have to somehow fail to step down, while the new leader won election surprisingly quickly. And finally, the lease mechanism would have to fail because clock frequency error was worse than ε.

In meta-summary, I think distributed systems implementors should rely on timers more. Whenever I discuss leases with engineers, the first question is, “Can we rely on timers?” It’s good to ask this question, but sometimes we have to take “yes” for an answer.

Images: Wellcome Collection.

Conformance Checking at MongoDB: Testing That Our Code Matches Our TLA+ Specs

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 03 Jun 2025 09:03:01 +0000

Cross-posted from the MongoDB engineering blog.

At MongoDB, we design a lot of distributed algorithms—algorithms with lots of concurrency and complexity, and dire consequences for mistakes. We formally specify some of the scariest algorithms in TLA+, to check that they behave correctly in every scenario. But how do we know that our implementations conform to our specs? And how do we keep them in sync as the implementation evolves?

This problem is called conformance checking. In 2020, my colleagues and I experimented with two MongoDB products, to see if we could test their fidelity to our TLA+ specs. Here’s a video of my presentation on this topic at the VLDB conference. (It’ll be obvious to you that I recorded it from my New York apartment in deep Covid lockdown.) Below, I write about our experience with conformance checking from 2025’s perspective. I’ll tell you what worked for us in 2020 and what didn’t, and what developments there have been in the field in the five years since our paper.

Agile modelling

Our conformance-checking project was born when I read a paper from 2011—“Concurrent Development of Model and Implementation”—which described a software methodology called eXtreme Modelling. The authors argued that there’s a better way to use languages like TLA+, and I was convinced. They advocated a combination of agile development and rigorous formal specification:

Multiple specifications model aspects of the system.
Specifications are written just prior to the implementation.
Specifications evolve with the implementation.
Tests are generated from the model, and/or trace-checking verifies that test traces are legal in the specification.

I was excited about this vision. Too often, an engineer tries to write one huge TLA+ spec for the whole system. It’s too complex and detailed, so it’s not much easier to understand than the implementation code, and state-space explosion dooms model checking. The author abandons the spec and concludes that TLA+ is impractical. In the eXtreme Modelling style, a big system is modeled by a collection of small specs, each focusing on an aspect of the whole. This was the direction MongoDB was already going, and it seemed right to me.

In eXtreme Modelling, the conformance of the spec and implementation is continuously tested. The authors propose two conformance checking techniques. To understand these, let’s consider what a TLA+ spec is: it’s a description of an algorithm as a state machine. The state machine has a set of variables, and each state is an assignment of specific values to those variables. The state machine also has a set of allowed actions, which are transitions from one state to the next state. You can make a state graph by drawing states as nodes and allowed actions as edges. A behavior is any path through the graph.

This diagram shows the whole state graph for some very simple imaginary spec. One of the spec’s behaviors is highlighted in green.

The spec has a set of behaviors B_spec, and the implementation has a set of behaviors B_impl. An implementation refines a spec if B_impl ⊂ B_spec. If the converse is also true, if B_spec ⊂ B_impl, then this is called bisimulation, and it’s a nice property to have, though not always necessary for a correctly implemented system. You can test each direction:

Test-case generation: For every behavior in B_spec, generate a test case that forces the implementation to follow the same sequence of transitions. If there’s a spec behavior the implementation can’t follow, then B_spec ⊄ B_impl, and the test fails.
Trace-checking: For every behavior in B_impl, generate a trace: a log file that records the implementation’s state transitions, including all implementation variables that match spec variables. If the behavior recorded in the trace isn’t allowed by the spec, then B_impl ⊄ B_spec and the test fails.

Here are the state spaces of a spec and an implementation. Non-conforming behaviors are highlighted in red:

Both techniques can be hard, of course. For test-case generation, you must somehow control every decision the implementation makes, squash all nondeterminism, and force it to follow a specific behavior. If the spec’s state space is huge, you have to generate a huge number of tests, or choose an incomplete sample.

Trace-checking, on the other hand, requires you to somehow map the implementation’s state back to the spec’s, and log a snapshot of the system state each time it changes—this is really hard with multithreaded programs and distributed systems. And you need to make the implementation explore a variety of behaviors, via fault-injection and stress-testing and so on. Completeness is usually impossible.

We found academic papers that demonstrated both techniques on little example applications, but we hadn’t seen them tried on production-scale systems like ours. I wanted to see how well they work, and what it would take to make them practical. I recruited my colleagues Judah Schvimer and Max Hirschhorn to try it with me. Judah and I tried trace-checking the MongoDB server (in the next section), and Max tried test-case generation with MongoDB Mobile SDK (the remainder of this article).

Trace-checking the MongoDB server

For the trace-checking experiment, the first step Judah and I took was to choose a TLA+ spec. MongoDB engineers had already written and model-checked a handful of specs that model different aspects of the MongoDB server (see this presentation and this one). We chose RaftMongo.tla, which focuses on how servers learn the commit point, which I’ll explain now.

MongoDB is typically deployed as a replica set of cooperating servers, usually three of them. They achieve consensus with a Raft-like protocol. First, they elect one server as the leader. Clients send all writes to the leader, which appends them to its log along with a monotonically increasing logical timestamp. Followers replicate the leader’s log asynchronously, and they tell the leader how up-to-date they are. The leader keeps track of the commit point—the logical timestamp of the newest majority-replicated write. All writes up to and including the commit point are committed, all the writes after it are not. The commit point must be correctly tracked even when leaders and followers crash, messages are lost, a new leader is elected, uncommitted writes are rolled back, and so on.

RaftMongo.tla models this protocol, and it checks two invariants: A safety property, which says that no committed write is ever lost, and a liveness property, which says that all servers eventually learn the newest commit point.

Judah and I wanted to test that MongoDB’s C++ implementation matched our TLA+ spec, using trace-checking. Here are the steps:

Run randomized tests of the implementation.
Collect execution traces.
Translate the execution traces into TLA+.
Check the trace is permitted by the spec.

The MongoDB server team has hundreds of integration tests handwritten in JavaScript, from which we chose about 300 for this experiment. We also have randomized tests; we chose one called the “rollback fuzzer” which does random CRUD operations while randomly creating and healing network partitions, causing uncommitted writes to be logged and rolled back.

We added tracing code to the MongoDB server and ran each test with a three-node replica set. Since all server processes ran on one machine and communicated over localhost, we didn’t worry about clock synchronization: we just merged the three logs, sorting by timestamp. We wrote a Python script to read the combined log and convert it into a giant TLA+ spec named Trace.tla with a sequence of states for the whole three-server system. Trace.tla asserted only one property: “This behavior conforms to RaftMongo.tla.”

Here’s some more detail about the Python script. At each moment during the test, the system has some state V, which is the values of the state variables for each node. The script tries to reconstruct all the changes to V and record them in Trace.tla. It begins by setting V to a hardcoded initial state V₀, and outputs it as the first state of the sequence:

\* Each TLA+ tuple is
\* <<action, committedEntries, currentTerm, log, role, commitPoint,
\*   serverLogLocation>>
\* We know the first state: all nodes are followers with empty logs.
Trace == <<
  <<"Init",                               \* action name
    <<"Follower","Follower","Follower">>, \* role per node
    <<1, 1, 1>>,                          \* commitPoint per node
    <<<<...>>,<<...>>,<<...>>>>,          \* log per node
    "">>,                                 \* trace log location (empty)
\* ... more states will follow ...

The script reads events from the combined log and updates V. Here’s an example where Node 1 was the leader in state V_i, then Node 2 logs that it became leader. The script combines these to produce V_i+1 where Node 2 is the leader and Node 1 is now a follower. Note, this is a lie. Node 1 didn’t actually become a follower in the same instant Node 2 became leader. Foreshadowing! This will be a problem for Judah and me.

Anyway, the Python script appends a state to the sequence in Trace.tla:

Trace == <<  
  * ... thousands of events ...  
    <<"BecomePrimary",                  * action name for debugging  
    <<"Follower","**Leader**","Follower">>, * role per node  
    <<1, 1, 1>>,                        * commitPoint per node  
    <<<<...>>,<<...>>,<<...>>>>,        * log per node  
    * trace log location, for debugging:  
    "/home/emptysquare/RollbackFuzzer/node2.log:12345">>,  
  * ... thousands more events ...  
>>

We used the Python script to generate a Trace.tla file for each of the hundreds of tests we’d selected: handwritten JavaScript tests and the randomized “rollback fuzzer” test. Now we wanted to use the model-checker to check that this state sequence was permitted by our TLA+ spec, so we know our C++ code behaved in a way that conforms to the spec. Following a technique published by Ron Pressler, we added these lines to each Trace.tla:

VARIABLES log, role, commitPoint
\* Instantiate our hand-written spec, RaftMongo.tla.
Model == INSTANCE RaftMongo
VARIABLE i \* the trace index

\* Load one trace event.
Read == /\ log = Trace[i][4]
        /\ role = Trace[i][5]
        /\ commitPoint = Trace[i][6]

ReadNext == /\ log' = Trace[i'][4]
            /\ role' = Trace[i'][5]
            /\ commitPoint' = Trace[i'][6]

Init == i = 1 /\ Read
Next == \/ i < Len(Trace) /\ i' = i + 1 /\ ReadNext
        \/ UNCHANGED <<i, vars>> \* So that we don’t get a deadlock error in TLC

TraceBehavior == Init /\ [][Next]_<<vars, i>>

\* To verify, we check the spec TraceBehavior in TLC, with Model!SpecBehavior
\* as a temporal property.

We run the standard TLA+ model-checker (“TLC”), which tells us if this trace is an allowed behavior in RaftMongo.tla.

But this whole experiment failed. Our traces never matched our specification. We didn’t reach our goal, but we learned three lessons that could help future engineers.

What disappointment taught us

Lesson one: It’s hard to snapshot a multithreaded program’s state. Each time a MongoDB node executes a state transition, it has to snapshot its state variables in order to log them. MongoDB is highly concurrent with fairly complex locking within each process—it was built to avoid global locking. It took us a month to figure out how to instrument MongoDB to get a consistent snapshot of all these values at one moment. We burned most of our budget for the experiment, and we worried we’d changed MongoDB too much (on a branch) to test it realistically.

The 2024 paper “Validating Traces of Distributed Programs Against TLA+ Specifications” describes how to do trace-checking when you can only log some of the values (see my summary at the bottom of this page). We were aware of this option back in 2020, and we worried it would make trace-checking too permissive; it wouldn’t catch every bug.

Lesson two: The implementation must actually conform to the spec. This is obvious to me now. After all, conformance checking was the point of the project. In our real-life implementation, when an old leader votes for a new one, first the old leader steps down, then the new leader steps up. The spec we chose for trace-checking wasn’t focused on the election protocol, though, so for simplicity, the spec assumed these two actions happened at once. (Remember I said a few paragraphs ago, “This is a lie”?) Judah and I knew about this discrepancy—we’d deliberately made this simplification in the spec. We tried to paper over the difference with some post-processing in our Python script, but it never worked. By the end of the project, we decided we should have backtracked, making our spec much more complex and realistic, but we’d run out of time.

The eXtreme Modelling methodology says we should write the spec just before the implementation. But our spec was written long after most of the implementation, and it was highly abstract. I can imagine another world where we knew about eXtreme Modelling and TLA+ at the start, when we began coding MongoDB. In that world, we wrote our spec before the implementation, with trace-checking in mind. The spec and implementation would’ve been structured similarly, and this would all have been much easier.

Lesson three: Trace-checking should extend easily to multiple specs. Judah and I put in 10 weeks of effort without successfully trace-checking one spec, and most of the work was specific to that spec, RaftMongo.tla. Sure, we learned general lessons (you’re reading some of them) and wrote some general code, but even if we’d gotten trace-checking to work for one spec we’d be practically starting over with the next spec. Our original vision was to gather execution traces from all our tests, and trace-check them against all of our specifications, on every git commit. We estimated that the marginal cost of implementing trace-checking for more specs wasn’t worth the marginal value, so we stopped the project.

Practical trace-checking

If we started again, we’d do it differently. We’d ensure the spec and implementation conform at the start, and we’d fix discrepancies by fixing the spec or the implementation right away. We’d model easily observed events like network messages, to avoid snapshotting the internal state of a multithreaded process.

I still think trace-checking is worthwhile. I know it’s worked for other projects. In fact MongoDB is sponsoring a grad student Finn Hackett, whom I’m mentoring, to continue trace-checking research.

Let’s move on to the second half of our project.

Test-case generation for MongoDB Mobile SDK

The MongoDB Mobile SDK is a database for mobile devices that syncs with a central server. (Since we wrote the paper, MongoDB has sunsetted the product.) Mobile clients can make changes locally. These changes are periodically uploaded to the server and downloaded by other clients. The clients and the server all use the same algorithm to resolve write conflicts: Operational Transformation, or OT. Max wanted to test that the clients and server implement OT correctly, meaning they resolve conflicts the same way, eventually resulting in identical data everywhere.

Originally, the clients and server shared one C++ implementation of OT, so we knew they implemented the same algorithm. But in 2020, we’d recently rewritten the server in Go, so testing their conformance became urgent.

My colleague Max Hirschhorn used test-case generation to check conformance. This technique goes in the opposite direction from trace-checking: trace-checking starts with an implementation and checks that its behaviors are allowed by the spec, but test-case generation starts with a spec and checks that its behaviors are in the implementation.

But first, we needed a TLA+ spec. Before this project, the mobile team had written out the OT algorithm in English and implemented it in C++. Max manually translated the algorithm from C++ to TLA+. In the mobile SDK, clients can do 19 kinds of operations on data; six of these can be performed on arrays, resulting in 21 array merge rules, which are implemented in about 1000 lines of C++. Those 21 rules are the most complex, and Max focused his specification there. He used the model-checker to verify that his TLA+ spec ensured all participants eventually had the same data. This translation was a gruelling job, but the model-checker caught Max’s mistakes quickly, and he finished in two weeks.

There was one kind of write conflict that crashed the model-checker: if one participant swapped two array elements, and another moved an element, then the model-checker crashed with a Java StackOverflowError. Surprisingly, this was an actual infinite-recursion bug in the algorithm. Max verified that the bug was in the C++ code. It had hidden there until he faithfully transcribed it into TLA+ and discovered it with the model-checker. He disabled the element-swap operation in his TLA+ spec, and the mobile team deprecated it in their implementation.

To test conformance, Max used the model-checker to output the entire state graph for the spec. He constrained the algorithm to three participants, all editing a three-element array, each executing one (possibly conflicting) write operation. With these constraints, the state space is a DAG, with a finite number of behaviors (paths from an initial state to a final state). There are 30,184 states and 4913 behaviors. Max wrote a Go program to parse the model-checker’s output and write out a C++ unit test for each behavior.

Here’s an example unit test. (It’s edited down from three participants to two.) At the start, there’s an array containing {1, 2, 3}. One client sets the third element of an array to 4 and the second client removes the second element from the array. The test asserts that both clients agree the final array is {1, 4}. The highlighted lines are specific to this generated test. The rest of the code is the same for all tests.

TEST(Transform_Array)
{
  size_t num_clients = 2;
  TransformArrayFixture fixture{test_context, num_clients, {1, 2, 3}};

  fixture.transaction(0, [](TableRef array) {
    array->set_int(0, 2, 4);
  });
  fixture.transaction(1, [](TableRef array) {
    array->remove(1);
  });

  fixture.sync_all_clients();
  fixture.check_array({1, 4});

  fixture.check_ops(0, {ArrayErase{1}});
  fixture.check_ops(1, {ArraySet{1, 4}});
}

These 4913 tests immediately achieved 100% branch coverage of the implementation, which we hadn’t accomplished with our handwritten tests (21%) or millions of executions with the AFL fuzzer (92%).

Retrospective

Max’s test-case generation worked quite well. He discovered a bug in the algorithm, and he thoroughly checked that the mobile SDK’s Operational Transformation code conforms to the spec. Judah’s and my trace-checking experiment didn’t work: our spec and code were too far apart, and adding tracing to MongoDB took too long. Both techniques can work, given the right circumstances and strategy. Both techniques can fail, too! We published our results and lessons as a paper in VLDB 2020, titled “eXtreme Modelling in Practice.”

In the subsequent five years, I’ve seen some progress in conformance checking techniques.

Test-case generation:

Model Checking Guided Testing for Distributed Systems. The “Mocket” system generates tests from a TLA+ spec, and instruments Java code (with a fair amount of human labor) to force it to deterministically follow each test, and check that its variables have the same values as the spec after each action. The authors tested the conformance of three Java distributed systems and found some new bugs. Their technique is Java-specific but could be adapted for other languages.
Multi-Grained Specifications for Distributed System Model Checking and Verification. The authors wrote several new TLA+ specs of Zookeeper, at higher and lower levels of abstraction. They checked conformance between the most concrete specs and the implementation, with a technique similar to Mocket: a human programmer instruments some Java code to map Java variables to spec variables, and to make all interleavings deterministic. The model-checker randomly explores spec behaviors, while the test framework checks that the Java code can follow the same behaviors.
SandTable: Scalable Distributed System Model Checking with Specification-Level State Exploration. This system is not language-specific: it overrides system calls to control nondeterminism and force the implementation to follow each behavior of the spec. It samples the spec’s state space to maximize branch coverage and event diversity while minimizing the length of each behavior. As in the “Multi-Grained” paper, the SandTable authors wisely developed new TLA+ specs that closely matched the implementations they were testing, rather than trying to use existing, overly abstract specs like Judah and I did.
Plus, my colleagues Will Schultz and Murat Demirbas are publishing a paper in VLDB 2025 that uses test-case generation with a new TLA+ spec of MongoDB’s WiredTiger storage layer, the paper is titled “Design and Modular Verification of Distributed Transactions in MongoDB.”

Trace-checking:

Protocol Conformance with Choreographic PlusCal. The authors write new specs in an extremely high-level language that compiles to TLA+. From their specs they generate Go functions for trace-logging, which they manually add to existing Go programs. They check that the resulting traces are valid spec behaviors and find some bugs.
Validating Traces of Distributed Programs Against TLA+ Specifications. Some veteran TLA+ experts demonstrate in detail how to trace-log from a Java program and validate the traces with TLC, the TLA+ model-checker. They’ve written small libraries and added TLC features for convenience. This paper focuses on validating incomplete traces: if you can only log some of the variables, TLC will infer the rest.
Smart Casual Verification of the Confidential Consortium Framework. The authors started with an existing implementation of a secure consensus protocol. Their situation was like mine in 2020 (new specs of a big old C++ program) and so was their goal: to continuously check conformance and keep the spec and implementation in sync. Using the new TLC features announced in the “Validating Traces” paper above, they toiled for months, brought their specs and code into line, found some bugs, and realized the eXtreme Modelling vision.
Finn Hackett is a PhD student I’m mentoring, he’s developed a TLA+-to-Go compiler. He’s now prototyping a trace-checker to verify that the Go code he produces really conforms to its source spec. We’re doing a summer project together with Antithesis to thoroughly conformance-check the implementation’s state space.

I’m excited to see growing interest in conformance checking, because I think it’s a serious problem that needs to be solved before TLA+ goes mainstream. The “Validating Traces” paper announced some new trace-checking features in TLC, and TLC’s developers are discussing a better way to export a state graph for test-case generation. I hope these research prototypes lead to standard tools, so engineers can keep their code and specs in sync.

MongoDB and CS Education for All, Part 2

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 27 May 2025 22:19:52 +0000

In 2016 and 2017, I photographed, reported, and wrote a two-part series on MongoDB’s sponsorship of computer science education in NYC public schools. Those articles were eventually deleted from MongoDB’s engineering blog, so I’m reposting them here, eight years later. Part 1 is here, and Part 2 is below.

Investing in CS4All: One Year Later

When a couple of New York City high school teachers partnered with MongoDB to teach computer science, did they succeed? Their curriculum was untested, and they were teaching in difficult districts where most students are from poor and minority families. I talked with these two teachers, Jeremy Mellema and Timothy Chen, back in September, when they had completed a summer fellowship at MongoDB and had just started teaching their curriculum; at the end of the academic year this spring, I visited Jeremy and Tim again to find out the result.

Their successes were sparse and partial. They discovered that their students’ poor reading skills were a barrier to learning to code, and that teaching new coders how to solve problems is, itself, an unsolved problem. With a coarse unit of iteration—a school semester—it is painfully slow to experiment and find teaching methods that work. But even partial wins make a difference for individual kids, and the support of professional engineers at companies like MongoDB can be a powerful accelerant.

What Engages Students

Jeremy’s main struggle was to get his students excited about code. He was assigned to teach a computer science class at Bronx Compass High School in the fall, using the curriculum he wrote during his fellowship at MongoDB last summer. In the beginning he spent too much time lecturing. “It felt weird,” he said. “It should be more like, ‘Let’s get down and dirty,’ and not, ‘Let’s have me talk to you.’” Even when his students did get their hands on computers, the first exercises were simply retyping Python scripts from a textbook. The payoff, watching a script run without throwing an exception, was hardly satisfying to them.

Things started to click when he introduced Python turtle graphics, which gave the class more obvious evidence of their accomplishments. It also allowed Jeremy better opportunities to motivate and engage his students directly. “Some days I would challenge them to see who could make the craziest drawing,” Jeremy says. He would tell his students, “That’s so cool. I only made a star. You definitely beat me today.”

Jeremy teaches both history and computer science, and he finds that some of his lowest-performing history students are his best CS students. “It’s satisfying to see them in their element,” he says. In Jeremy’s view a computer science class can touch a student’s intellect just as deeply as history. “People are multifaceted. You’re not only who you are when you’re in my history class.”

Jeremy’s computer science class was cancelled this spring; the students at Bronx Compass High School are behind on history credits and there are only three history teachers on staff. For now, computer science is merely an elective, so Jeremy is back teaching history full-time. “I really miss teaching CS,” he says.

If he resumes the course, Jeremy thinks it must be livelier. He is reconsidering his use of the videos from How To Think Like a Computer Scientist, which he studied last summer on the recommendation of his mentor Shannon Bradshaw, MongoDB’s vice president of education. The content helped Jeremy train to teach CS, but when he showed the videos to his kids they were bored. Jeremy hopes to make new videos that will draw them in. His students from the fall semester say he should get their advice. Otherwise, they warned, “you might do something that you think is cool but it’s actually super corny.”

A Head Start

Although there is no computer science elective this semester, some students are pursuing the topic in other ways. A young woman from his class in the fall, Tatyana Camacho, now interns for the high school’s IT department. I had quoted her in my previous article, and Jeremy tells me she loved it. She commanded him to show her father in the next parent-teacher conference: “You need to show my dad that I’m one of the advanced students.”

Jeremy still runs the afternoon Computer Club. I visited the club to meet a student, Daniel Rodriguez, who was tinkering with an Arduino and a circuit board that the school provided. “I don’t have the ability to get this equipment otherwise, in my predicament,” says Daniel. He starts his Arduino projects by copying examples. The wiring is easier than the coding for him, he told me, “especially because I’m not the best speller in the world.” Once he has the example working he modifies it to his own taste. Most recently, he wanted to show a message but, with only LEDs, he can’t display much. He researched Morse Code and made a light flash the code for “HELLO”, like any programmer demonstrating a system for the first time.

“Most people think that once you plug something in, that’s it, it works,” says Daniel. “But I’m the person that makes the circuit run. I tell people, ‘I made it do that.’ And seeing them fascinated by what I did, it makes me, in turn, fascinated by what I’m doing.”

Daniel has to return the Arduino at the end of the year. Next year he’ll go to a trade school for electricians. Working with the Arduino will give him an advantage, he hopes, and it seems plausible to me. As he finishes school and starts work as an electrician the world will be changing around him: smart appliances and programmable components will be everywhere. An electrician who loves to code will have a big head start.

Making Anything They Want

Timothy Chen teaches in Hell’s Kitchen, at Urban Assembly Gateway School for Technology. I visited his class in May to see how his students had progressed since I last saw them in September. They were involved in a multi-week project called the AP Create Task, part of a national Advanced Placement exam. “They are allowed to make literally anything they want, in any language,” says Tim. Students submit their code and a one-minute video of the program in action, and they may describe their project either in writing or in audio narration. I was surprised by how 21st Century the test is, and how accommodating it could be to students with a deficit in reading and writing. It must be remarkably difficult, however, to score fairly.

The Create Task is many students’ first time scoping and integrating a sizable project, and there were some flameouts. One young man tried to make a maze game drawn with ASCII characters; it proved too ambitious and he ran out of time. Tim isn’t supposed to help students define the scope of their projects, but if they announce they’re going to tackle something difficult he will push them to list all the components. In the best case, they realize they don’t know how to do most of the project and choose something simpler.

One of Tim’s students, Jasean Maxwell, was building a Go Fish card game in Python, and she was having trouble integrating the pieces. “It has to be a certain order and it’s hard to make that order when you don’t know, really, what you’re doing. I’m struggling, putting it all together.”

Another student, Cecilia Gonzalez, was writing a Choose Your Own Adventure game. She says the AP Create Task encourages students to work in pairs. “We work sort of together but not exactly.” Each must create at least one significant part of the program independently. Cecilia’s game is based on a monster of urban legend called The Rake, which comes closer when you think about it. The game begins by asking questions such as the player’s name and height. She told me the player’s answers will determine “some things that are going to happen,” but she didn’t give away any spoilers.

When Tim began teaching the class in September, he hadn’t written the ending yet, either. His greatest fear was his students would learn the curriculum faster than he could write it. By May it was clear that wasn’t a problem. “Some of the students can’t read very well, and that was a big barrier because all the things I made were text,” he says. “Everything just took longer than expected.”

Problem Solving

How do you teach problem solving? This is Tim’s great unanswered question from the year. Perhaps if high school computer science were taught like math, as a series of small problems with only one right answer each, then how to solve those problems wouldn’t be such a mystery. But high school CS is taught like art class. Tim’s students invent new projects and somehow solve the unpredictable problems that arise in them.

Tim speculates that he would learn problem solving himself by watching an experienced programmer solve a new problem, hit roadblocks, and overcome them. Indeed, that is how I have taught problem solving to MongoDB interns. Together, we attack problems without knowing the answers beforehand. It requires an entire summer of one-on-one collaboration. “I don’t think that model works very well with the kids,” says Tim, “especially if they are not very good with sitting still for an extended period. I’m not sure how to reach them.”

Tim, like Jeremy, wants to make more multimedia to reach students despite their poor reading skills. “I want to rethink how it should be done before I start this time. I kind of jumped into it too quickly.” Tim’s main goal is to give kids the chance to fall in love with programming and continue on their own. Many other goals are still out of reach: students at his school score low on the AP test, and few of them are likely to get a college degree in CS or be professional coders. Still, Tim hopes that a more varied course, with audio and video, could bring students farther. “The big hurdle for everyone is teaching problem solving. If I can get that, everything else is easy. I’m still trying to figure out how to do that.”

The View From 2025

After I wrote the 2017 followup you just read, I lost track of the two teachers, Jeremy Mellema and Timothy Chen. MongoDB had donated money and teacher training to the CS4All program in 2016 and 2017, but I don’t know if we continued after that. CS4All’s champion at MongoDB, our VP of Education Shannon Bradshaw, left the company in 2020.

It’s 2025 now and CS4All is supposed to achieve its goals this year: every NYC public school student should have at least one “meaningful CS experience” every 3 years throughout their school career: either a computer science class or 11+ hours of CS instruction within some class. The plan was, instead of hiring specialist teachers, to train 5000 existing teachers in computer science, throughout all 1800 schools.

The Research Alliance for New York City Schools at NYU published dozens of reports on CS4All’s progress over the last decade, culminating in a final report this December. There’s good news and bad news. On the one hand, “Student participation in CS has expanded exponentially: About 54,000 students participated in a CS unit or course during the first year of the initiative; this grew to nearly 170,000 in 2022-23—an astounding 200 percent increase. (Preliminary data show CS reaching nearly 200,000 in 2023-24.) The number of schools offering CS courses—and the total number of CS courses available—have also increased dramatically. The percentage of schools offering CS each year tripled (from 15% to 45%), while the number of CS courses increased nearly 10-fold, from 534 to over 5,000. Altogether, by 2022-23, 93 percent of schools offered students at least one CS experience at some point over the previous three- or four-year time period (corresponding to the three- or four-year grade bands).” However, by 2023 only 38% of students had had a CS experience in the last 3 years. Girls participated less than boys, and Black and Latino students less than White and Asian—these disparities were worse among older students, but CS4All narrowed disparities over the course of the decade.

According to the report, about 3000 teachers were trained, compared to the goal of 5000, and this wasn’t enough to meet CS4All’s goals. Not all trained teachers actually taught CS. A quarter of them left the NYC public school system some time after training, and many who remained stopped teaching CS. Computer science classes competed on the schedule with other subjects, and teachers often didn’t have enough time to prepare CS lessons. At the high school level, NYC had trouble finding qualified specialist teachers—even in the current tech industry downturn, I imagine there are few programmers who’d accept the salary and conditions of NYC public schools.

The city is still committed to some sort of CS education for all students, and now New York State has announced required CS curriculum for all students and CS training for all teachers. It seems like the obstacles I observed almost ten years ago were a difficulty for the program’s whole history, but it nevertheless made progress and had a statewide and national influence.

MongoDB and CS Education for All, Part 1

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 27 May 2025 12:18:34 +0000

MongoDB’s Investment in CS Education

Until last year, Jeremy Mellema was a history teacher. Now, he’s teaching computer programming. When I visited his class in the Bronx this month, he had 30 students with 30 Macbooks, completing exercises in Python. They had just finished a lesson on data types, and now they were tackling variables. In Jeremy’s class, the first variable assignment is:

tupac = "Greatest of All Time!!"

Initiative

A year ago, New York City mayor Bill de Blasio announced Computer Science for All, an $80 million public-private partnership. The goal is to teach computer science to every student at every public school. But first, the schools need curriculum, and 5000 teachers need training.

Here at MongoDB, our VP of Education Shannon Bradshaw oversees MongoDB University, which trains IT professionals to use MongoDB. When he heard about CS4All, he wanted us to contribute. He proposed that we set aside budget for two paid fellowships, and recruit public school teachers to spend the summer with us. We would develop them as teachers, and help build curricula they could take back into schools this fall. MongoDB staff would share our expertise, our office space, our equipment, and the MongoDB software itself.

Shannon pitched his proposal to the company like this: “As many of us know, it’s still unusual for students to encounter computer science, let alone databases, in their classrooms before entering college. I believe this absence directly contributes to the gender and racial disparity we see today across our industry.” The CS4All project improves access to these subjects for many more students in our city, and MongoDB could be part of it from the beginning.

Introducing Tim and Jeremy

The teachers we hired for the summer are at opposite ends of a spectrum. At the more technical end is Tim Chen. He majored in Math and took some CS classes in college. When he applied to work for the NYC Department of Education, there was a job opening for a Tech Teacher. “I didn’t know what ’tech’ meant at the time,” he said, “but I lucked out because it was teaching really basic software engineering.” His first year teaching, he taught the 9th grade curriculum for the Software Engineering Program. Now, he’s an experienced CS teacher, and his students this fall are comparatively advanced programmers. He spent his summer at MongoDB outlining an ambitious two-year curriculum in Python and Javascript.

Jeremy Mellema, on the other hand, was teaching 10th grade world history when he was tapped to teach a CS class last year. He took an intensive training with the NYC Department of Education’s Software Engineering Program, where he spent a day apiece working in Python, HTML, CSS, JavaScript, Scratch, and Arduino. That fall was a struggle. “It was pretty overwhelming,” he said. Jeremy is a skilled teacher, but on the topic of software, he “just didn’t know a whole lot.”

Jeremy joined us this summer to meet software engineers and learn to code more like a professional. His presence is particularly valuable because he’s not a specialist: he’ll ensure the curriculum he builds can be taught by other teachers who aren’t already coders. Shannon assigned him the book How to Think Like a Computer Scientist, and he spent the summer turning its contents into something high schoolers can learn from. He told me, “I was excited to actually get my hands dirty with what it means to be a programmer and learn how to do things besides basic Scratch programming, drag and drop, and to see how you actually use it in the real world.”

Foreshadowing

Now that school is back in session, I’m watching Tim and Jeremy’s progress with fascination. Their students come to CS from a different angle than I did: I have the typical background of a computer programmer in this country, with college-educated parents and all the privileges that typically pave the way to a career in software. I taught myself to code in high school and got a Bachelor’s in CS from Oberlin College. But the students in Tim and Jeremy’s classes nearly all come from low-income families and qualify for federal meal assistance. These students have self-selected into the computer track, and they are mostly boys; but there are a half-dozen girls in both classes. The majority are Black or Hispanic, and Jeremy’s class also includes many recent immigrants from the Caribbean and Bangladesh. If young people like them succeed as software engineers, it will go a long way to addressing the inequalities of our industry and our society as a whole.

But the goal for Tim and Jeremy this year is more modest. They will prove and refine their curricula in their classrooms; then the materials can be used by any teacher in the New York City public schools. I visited their classes to watch their plans being put to the test.

Jeremy

When I saw Jeremy at Bronx Compass High School, he’d had the students for one week. He was nervous the month before about how class would go: “I’m always afraid it’s going to bomb, or the students won’t find it interesting.” But the kids were hacking enthusiastically. Watching them reminded me of my own joy when I first learned to code.

Class time is spent working independently while Jeremy checks in with each student. The kids present a huge range of skills: some have taken computer science classes for several years, and others have just moved to New York and are getting their first exposure to CS. Jeremy worries about challenging all his students according to their level.

An advanced student, Tatiana, is able to use this wide skill range to make the class more effective for the novice students. I noticed her sitting sideways in her chair so she could coach the students to her side and behind her. “It’s nice to help people who struggle with coding because it’s like a mixture of math and grammar,” she said. “When you write ‘print’, if you write it with a capital P your code won’t work. It’s like teaching them a new language.” Tatiana plans to keep coding after high school: “I like making things work the way I want them to work.”

Since my formal CS education began in college, I’ve never sat in a high school programming class. To my surprise, 30 students can be trusted to stay on task, even when they’re on laptops with the whole Internet in reach. Jeremy doesn’t mind when they take detours to the Web. He said, “If they just need YouTube to play music to drown out their classmates that’s fine.” I saw his student Amaury with headphones on and YouTube playing in the background in order to concentrate, the same way I use music at the MongoDB office. Jeremy gives him space to work in his own style: “It’s fine. As soon as I ask Amaury a question, his headphones are off and he answers.”

I asked Amaury to take off his headphones for a minute and tell me how the class was going. “It’s cool learning how a computer works using just a bunch of freaking inputs. Put in a couple lines of code and you can have the computer do crazy stuff.” Then his headphones went back on and he got back to hacking.

By the end of this year, Jeremy hopes his kids will advance from sandboxed programming environments to real-world tools—that they’ll install Python on their computers and code on their own. “The kids I’ve seen really do well,” he said, “this will open the door for them to take the world by storm. There’s a lot of really smart and talented kids who are not in an advantaged place. This will put them at an advantage.”

Tim

Tim Chen teaches a few blocks from the MongoDB office, at Urban Assembly Gateway School for Technology on the west side of Manhattan. His students have specialized in CS more than Jeremy’s, and he has more time with them: 45 minutes a day, 5 days a week. His curriculum is accordingly more ambitious. This year he’s teaching introductory programming in Python, aligned with the upcoming version of the Advanced Placement test in Computer Science Principles. Beginning with Python makes entry to the subject easy for Tim’s students, but by the end of the year they’ll cover nearly every topic of a first semester college class. Next year they’ll switch to Javascript. They’ll build web applications with MongoDB, Express, and Node. These three technologies (along with Angular.js) make up the famous MEAN Stack, which Tim learned this summer from MongoDB University.

When I asked Tim how his curriculum performed in its first week, he said there’d been no surprises yet—but he’s still in the process of writing it. Only the first semester’s material is complete. “My greatest fear is that they’ll move faster than I can create it,” he said. Still, he expects his students to be forgiving if they hit any bumps. “I told them this is my first year trying this out, so we’re going to try things, and if it doesn’t work out we’re going to try different things. They’re cool with it.”

Scaling up

Teaching one class in Hell’s Kitchen and one in the Bronx is only the start. Tim wants to create a curriculum that any teacher can pick up and deliver to their students. Jeremy has the same goal, and because of his background as a history teacher, rather than a computer scientist, he is focused on making his course effective for teachers like him.

For MongoDB’s VP of Education Shannon Bradshaw, this teacher-training is the most unexpected, and the most inspiring, aspect of our involvement. “Most of the people who will teach computer science in the New York City public schools are transitioning from another discipline,” he said. “They’re never going to be hardcore software engineers. They’re professional teachers; that’s what they want to do.” But in nine years, computer science will be a core class in New York City, and the existing staff will have to teach it. “If we can contribute to figuring that out, how to make this transition as effective as possible for computing education in New York City and maybe elsewhere in the country, then we’ve gone miles beyond where I imagined us going.”

Images © A. Jesse Jiryu Davis

Liza Mills

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 22 May 2025 17:38:37 +0000

Gunks climber Liza Mills on the routes Lisa and The Sting with her husband Jordan and their cat Dwayne. The Sting ends with a dramatic dyno to a ledge; Liza latched it and sent the route on her first try of the day, but I’m including a photo of her falling on a repeat attempt, for spice.

Shot on 35mm Portra 400 with a Canon EOS10, which has a rather fast motor drive for sports photography.

Images © A. Jesse Jiryu Davis

Jesse's Notes from the Antithesis BugBash Conference

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 20 May 2025 22:33:34 +0000

Antithesis has pioneered “autonomous testing”, a term they coined for their platform, which combines a deterministic hypervisor with coverage-guided fuzzing and fault injection. My company MongoDB was one of their first customers—I evaluated their platform for MongoDB in 2020. At the time we had several very rare replication bugs, including one that manifested once in many years of stress testing (tens of thousands of hours of compute), and Antithesis found them reliably in about 24 hours.

Antithesis was in quiet mode until mid-2024 when they suddenly turned on the marketing switch, started blogging, and announced the BugBash conference. This is a small (about 200 attendees) conference with invited speakers (they accepted lightning talk proposals). MongoDB’s distributed systems researcher Will Schultz gave a lightning talk on Spectacle, and our new Atlas Federation engineer Stephanie Wang gave a regular talk on lessons learned from her time building a cloud DB at MotherDuck.

The Antithesis CEO Will Wilson calls this “not a software testing conference, but a software correctness conference”. Of course there was a fair amount of Antithesis marketing and education, but mostly the talks were about non-Antithesis software correctness topics.

Antithesis is posting and summarizing the conference videos on their blog, two have been posted as of 2025-05-24.

Talks

I attended almost all of the talks. Here are my notes from the ones I saw.

Will Wilson

Why is software buggy? He’s the Antithesis CEO. He offered various theories about why we tolerate bugs, including my favorite, “it’s economically optimal to tolerate medium-quality software in most circumstances”, and his own favorite, “software engineering is a young discipline whose major innovations in quality are yet to come—soon we hope”.

Zac Hatfield-Dodds

Why is everything under-tested? He’s now at Anthropic, but he has a longer history as the author of the Python property-based testing library Hypothesis. I covered his work when I was a blogger for the Python Software Foundation. He described property-based testing and its advantages, and theorized that the main barrier to adoption is awareness and education. There has been a major uptick in Hypothesis usage recently.

Zac said something like, “coverage-guided fuzzing reduces the time to find a bug from exponential to polynomial”. This sent me on a side-quest, not yet completed, about the efficiency of randomized testing.

Stephanie Wang

Reliability lessons from building a cloud data warehouse from 0 to 1. Stephanie was at MotherDuck until very recently, now at MongoDB on Atlas Data Federation. She described the challenges of layering a cloud DB on top of the DuckDB embedded columnar database.

Ben Eggers

You should test in prod. This is like Netflix’s “Chaos Engineering”, though I guess that term is passé for obscure factional reasons. Eggers describes testing in prod at OpenAI: how to convince your manager, what to test, how to test it, how to do so safely. You should do it during waking hours, warn everyone ahead of time, and have a concrete hypothesis that you’re testing. E.g., don’t test “what happens when I take down this database?” Test, “I think when I take down this database that service X will go offline but service Y will stay up.”

Kyle Kingsbury

Jepsen 17: ACID jazz. Standard Jepsen talk: he tested a few distributed services and found some interesting consistency violations. Extremely entertaining as always.

Ryan Worl

The Multi-Tenant Bill of Rights: observability for all. An argument for investing in observability, with the twist that it helps “noisy neighbor” interference among tenants.

Mitchell Hashimoto

Can we test it? Yes, we can! Hashimoto created Vagrant, Terraform, and other cloud infrastructure tech, now he’s made a surprising pivot and develops a terminal emulator called Ghostty. He talked about refactoring code to make it unittestable and how to test GUIs.

Lawrie Green

Autonomous testing masterclass: How to succeed in software testing without really trying (or, a guide to autonomous testing). Principles for using Antithesis. Don’t test specific sequences of steps, test software properties and let Antithesis generate all the sequences. Properties can include “X is always true” or “never true” or “eventually true”, or “a state is unreachable” or “a state is sometimes reachable”.

I noticed that “always” and “eventually” are temporal operators in TLA+ (and other temporal logics) and they can be evaluated on a single behavior of a program. “Eventually” is a liveness property that in theory can only be evaluated on an infinitely-long behavior. But Antithesis lets you configure a test like, “once all fault-injection ceases, the system achieves state X before some timeout T”.

“Sometimes” and “unreachable” are hyperproperties that must be evaluated on the set of all possible behaviors. They’re not supported in TLA+. It’s cool that Antithesis allows them.

Ankush Desai

Gain confidence in system correctness using formal and semi-formal methods. Ankush talked about P, his formal modeling language. Unlike TLA+, P has a very Pythonic syntax and an intuitive structure: communicating processes.

I think Ankush said that procedure bodies in P can actually be coded in Python, they don’t just look like Python? Unclear.

P has two conformance-assurance features that interest me, but they seem shady. 1) The docs say P can be compiled to C/C#/Java, but Ankush didn’t mention this in his talk so maybe it’s not fully implemented. 2) P supports trace-checking: you can add logging statements to your implementation and feed the logs to a trace-checking utility called PObserve. Ankush mentioned this utility in his 2022 HPTS talk (slides, p. 20), and it gets a paragraph in Ankush and Marc Brooker’s Systems Correctness Practices at AWS: Leveraging Formal and Semi-formal Methods, and he mentioned it again at BugBash. But PObserve doesn’t seem open source or described in detail anywhere.

Lightning talks

Felicitas Pojtinger: Commoditize your CI compute. She started with a nice idea from Spolsky: “Smart companies try to commoditize their products’ complements.” She showed a technology from startup Loophole that makes cloud providers’ spot instances an interchangeable commodity, by live-migrating onto and off of spot instances without losing state. You can have a long-running job jump from one spot instance to the next, across cloud providers, while maintaining TCP connections etc.

William Schultz: Interactive formal specifications. Will (MongoDB research) has renamed his tla-web tool to Spectacle, it’s a partial implementation of TLA+ in JavaScript. You can interactively run a spec’s actions forward and backward, choosing the next state at each transition and inspecting variables and visualizations, and share traces with your colleagues using a URL.

Jayaprabhakar Kadarkarai: Modeling distributed systems. “JP” has a new modeling language FizzBee. His language focuses on ease-of-use for engineers, which I agree is lacking in most other languages. He’s made impressive progress developing solo for the last 2-3 years. FizzBee has a Pythonic syntax and a ton of features, like several builtin visualizations and interactivity, and a prototype of probabilistic performance modeling.

Marco Primi: The traveling anteater. Entertaining talk about using Antithesis to “solve” the traveling salesman problem, just demonstrates that Antithesis is a heuristic tree-searcher.

State of Reliability panel discussion

Five men with war stories, jokes, and conflicting opinions. One interesting idea: a goal of quality and testing is to reduce nondeterminism, but AI and ML are shifting us to a world where good software is legit nondeterministic. We’ll have to develop a new testing philosophy.

Evaluation

The conference organizers had an all-male speaker lineup when they first announced BugBash. They corrected this, but they still had no women on their panel. I complained about this, and I hope they do better next year.

I knew BugBash would be a relevant conference for me and the MongoDB Distributed Systems Research Group, but it was far better and more relevant than I expected. Lots of people I know from distributed systems research and formal methods were there. I made useful connections with people, I learned valuable things, and I had some moments of inspiration that could become research projects.

Images by the sublime Boris Artzybasheff.

Are We Serious About Using TLA+ For Statistical Properties?

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 10 May 2025 10:59:24 +0000

At this year’s TLA⁺ community event, I tentatively proposed adding features to make the language useful for performance modeling. Here’s the video, and a written version is below.

Half Our Problems

In 2022, I saw Marc Brooker give a talk called “Formal Methods Only Solve Half My Problems” at HPTS. He published a blog post about this too. He said that TLA⁺ can check correctness (safety and liveness), but not performance characteristics. “What I want is tools that do both: tools that allow development of formal models … and then allow us to ask those models questions about design performance.” He acknowledges that correctness is important. But it’s not enough to say nothing bad ever happens, and something good eventually happens—we want to know how quickly something good happens!

How would we actually create such a tool? Marc said that queueing theory was the kind of math that was obviously useful for this.

Learn Queueing Theory?

So I decided to learn queueing theory. I asked Marc what book I should read, and he recommended Performance Modeling and Design of Computer Systems, Queuing Theory in Action. The title sounds like it’s the perfect book for our purposes, so I recruited Andrew Helwer and Murat Demirbas to read it with me. We spent 8 months reading most of the book and doing a lot of the problem sets. It was definitely interesting, but mostly irrelevant to the actual problem of, you know, performance modeling and design of computer systems.

As an example of a queueing theory problem, here’s a diagram from Chapter 2 of the book.

There are three servers, each with a queue where tasks arrive. A server processes tasks from its queue, and then the task either leaves the network, or it gets sent to another server’s queue. There are probability distributions for all these events: the delays between task arrivals, the time it takes for a server to process a task, the chance a task leaves the network or goes to another server.

What kinds of questions does queueing theory answer about this system?

In the book, one of the exercises asks a classic queueing theory question, which is also a practical question: what’s the maximum arrival rate at Server 1’s queue that the system can handle?

I did my homework with paper and pencil. I solved some equations using the techniques in the book and came up with an answer: the max arrival rate is 5.2.

I don’t think we should try to answer performance questions by writing equations and solving for a variable! It’s hard to get right, easy to get wrong, and you have to recalculate every time the inputs change. I think we should just use simulation—run an experiment thousands of times and take the average. Call it “Monte Carlo” simulation if that makes you feel better.

Java Modelling Tools

There’s an open source toolkit that will do this for you, an academic project called the Java Modelling Tools. They’re a bunch of tools for modeling queue networks and then answering questions about them, through some combination of equation-solving and simulation. I started from the exercise in the book, and I used JMT to draw this very ugly version of the same queue network:

I configured a bunch of things and ran a simulation where I varied the arrival rate at Server 1:

JMT lets me sweep the value of the arrival rate from 0 to 10 by increments of 0.1—that’s the independent variable—and I can simulate the system and measure the server utilization that results—that’s the dependent variable. You can see that when the arrival rate reaches 5.2, server utilization reaches 1.0, meaning it’s completely utilized and the system has reached its capacity. That’s the answer I got with my pencil, too. This kind of simulation works even for complex queue networks where the math is hard or impossible.

What I’ve learned from the book and the tool is that queueing theory has super-useful concepts: arrival rate, service rate, utilization, ergodicity, Little’s Law, service discipline, open vs. closed loop, and many more. But the math is heinous. There are some simple equations to start you off, like Little’s Law, and then all hell breaks loose. So, don’t try to learn the math. No matter how good you are at math, queueing theory math won’t get you what you want. Specifically, you’ll never be able to estimate system performance by solving equations. You can only solve the equations for extremely simple systems. If you want to model a real-world system, the math will be an open research question, or literally impossible.

Just run simulations.

If we use simulated queue networks to model performance, have we solved both Marc’s problems? No, Marc wants one tool that both checks an algorithm’s correctness and models its performance, with a single specification.

TLA+ for Performance Modeling

The same year Marc talked about this, Jack Vanlightly and Markus Kuppe presented “Obtaining Statistical Properties via TLC Simulation.” They described how to write a spec and use TLC to measure statistics. (Video, slides, Jack’s article, Murat’s review.) I’ll show some code examples from Jack’s spec for a gossip protocol.

\* Increment the updates counter by the number of incoming peer states.
TLCSet(updates_ctr_id, TLCGet(updates_ctr_id)
                       + Cardinality(DOMAIN incoming_peer_states))

The spec uses TLCSet to record each statistic. Here it’s counting the number of times nodes get new information. TLCSet and TLCGet access a global scratchpad, so you can record things outside of the state space.

CSVWrite(
  "%1$s,%2$s,%3$s,%4$s,%5$s,%6$s,%7$s,%8$s,%9$s,"
  \o "%10$s,%11$s,%12$s,%13$s,%14$s,%15$s,%16$s,"
  \o "%17$s,%18$s,%19$s,%20$s,%21$s,%22$s,%23$s,"
  \o "%24$s,%25$s",
  <<behaviour_id, 
    r, RoundMessageLoad(r), DirectProbeDeadMessageLoad(r),
    IndirectProbeDeadMessageLoad(r), TLCGet(updates_pr_ctr(r)),
    TLCGet(eff_updates_pr_ctr(r)),
    alive_count, suspect_count, dead_count, alive_states_count,
    suspect_states_count, dead_states_count,
    infective_states_count, infectivity, cfg_num_members,
    cfg_dead_members, cfg_new_members, SuspectTimeout,
    DisseminationLimit, cfg_max_updates, cfg_lose_nth,
    cfg_peer_group_size, cfg_initial_contacts, MaxRound>>,
  RoundStatsCSV)

This code appends a line to the CSV file, recording the results of the execution. Jack had to hand-code this in TLA⁺, and hand-code the logic for when to run this during TLC’s execution, and so on. It’s impressive that it works at all, but it’s kind of a hack.

So my Complaint Number 1 is the syntax. What we have now in TLA⁺ is hard to write, it attracts bugs, and it clutters up the specification.

\*  'probabilistic' is a random chance of losing the message
\*  'exhaustive' is for model checking where both options are explored
GetDeliveredCount() ==
  CASE MessageLossMode = "probabilistic" ->
      IF RandomElement(1..cfg_lose_nth) = cfg_lose_nth
      THEN {0}
      ELSE {1}
    [] MessageLossMode = "exhaustive" -> {0,1}

SendMessage(msg) ==
  \E delivered_count \in GetDeliveredCount() :
    \* ... send the message if delivered_count is 1 ...

Here’s some code to determine whether a message is delivered or lost. Jack has a config option called MessageLossMode, he has to set it to “probabilistic” to make statistics work, or set it to “exhaustive” to make model-checking work. My Complaint Number 2 is that randomization is incompatible with model-checking and you have to code workarounds. (Correction: Markus says this is fixed now!)

Jack has another a constant called cfg_lose_nth, let’s say it’s 4, then in probabilistic mode there’s a one-in-four chance that a message is lost. You could extend this to any rational number, like two-out-of-three, but you can’t work with irrational numbers. TLC doesn’t have floating-point numbers at all. So my Complaint Number 3 is that randomization is very limited. It’s just Dungeons & Dragons dice throws.

If you want to measure performance, you need some sort of cost function. E.g., sending a message to Tokyo might cost 2.5 times as much as sending it to London. Or its cost might be exponentially distributed. Here’s some imaginary TLA⁺ for accumulating the cost of an algorithm:

\* In your dreams
TLCSet(cost, TLCGet(cost) + 1.0)
TLCSet(cost, TLCGet(cost) + 2.5)
TLCSet(cost, TLCGet(cost) + Exponential(3))

All the examples I’ve seen of statistical properties in TLA⁺ are counting the number of times some discrete event occurs, like counting the number of messages. But if we want to measure the performance of a system as a whole, we need to measure a variety of events, and they have to have relative costs! Complaint Number 4 is we have no floats and no probability distributions besides “uniform”.

Statistical Properties in TLA⁺, So Far

I think it’s just a proof of concept. Jack and Markus’s technique works and it’s useful. What they accomplished is impressive and it started the conversation. But it’s insufficient if we’re serious about using TLA⁺ for statistics. TLCSet and the CSV module provide a manual, low-level API to save statistics. Compared to safety and liveness checking, which are first-class features of the language and tools, statistical modeling really clutters up the specification.

As we saw with Jack’s gossip protocol, it’s not only a large amount of code, but it’s error-prone, you have to write it into each spec, and you don’t get the benefit of a tested and documented library. People need a concise and easy API for saving statistics.

The Randomization module only gives us uniform distributions with integer parameters. Performance modeling requires lots of distributions, such as Poisson, exponential, and Zipf. Limiting probabilities to ratios of integers is an artifact of our unfinished tooling; we should give TLA⁺ authors the freedom to choose whatever probabilities they want.

If you want to model the performance of a whole system, you need to combine the costs of different actions. You can do that manually with TLCSet, but there’s no standard convenient way to do it.

And there are no floats. This makes sense for model-checking, but it’s a silly limitation for performance modeling. I’ll stop talking about that before I get too annoying.

Statistical Properties: Are We Serious?

So, what we have right now in TLA⁺ isn’t really solving both of Marc’s problems. Should we try? Should TLA⁺ be the tool that solves both, or should we stay focused on safety?

I don’t know. I haven’t implemented anything, I’m just offering ideas. And I want to admit I’m not offering any time or money to this project, so I can’t make any demands. I’m just brainstorming with you.

State of the Art

As a start, let’s look at some existing tools: JMT, PRISM, Runway, and FizzBee.

State of the Art #1 of 4: Java Modelling Tools

We saw JMT above; it’s the tool where you can draw queue networks. It’s made for statistical modeling and answering performance questions. It has a point-and-click interface, which is good for small models and bad for big ones. There’s also a Python API for generating large models. It comes with lots of probability distributions built-in, it can model cost functions, and you can use real-world data sets as inputs! E.g., if you have some data about the distribution of network latencies, you can import that into a JMT model.

State of the Art #2 of 4: PRISM

PRISM is built for exactly this problem. It’s a “probabilistic model checker”: you write a spec of an algorithm and ask statistical questions, like the probability of something happening, or the average cost.

Actions in PRISM look like this:

[my_action] x=0 -> 0.8:(x'=1) + 0.2:(x'=2);

You have an action name (“my_action”), a guard, which is some predicate (this action is enabled if x is zero), and some state updates with probabilities. There’s a 0.8 probability that it sets x to 1 and a 0.2 probability that it sets x to 2. These numbers are either probabilities that something happens, or the rates at which things happen, depending on whether you’re writing a discrete-time or continuous-time model.

PRISM has built-in support for cost functions. A cost is just anything you want to measure, any way of expressing performance. PRISM calls costs “rewards,” but if you’re thinking of reinforcement learning, stop. This is not related to reinforcement learning.

rewards
    x=0 : 100;
    x>0 & x<10 : 2*x;
endrewards

This code says that every state where x equals zero has a reward of 100, and if x is between 0 and 10 then the reward is twice x. You can imagine writing a reward function that measures revenue, or a reward function that’s actually a cost function that measures something bad, like latency.

PRISM has remarkably powerful property expressions. Here’s an example from their website, it’s “the probability that more than 5 errors occur within the first 100 time units is less than 0.1.” This is an assertion that PRISM checks.

P<0.1 [ F<=100 num_errors > 5 ]

Here’s another, “the probability that process 1 terminates before process 2 does.” This is a question—what is the probability?—and PRISM will give you the answer.

P=? [ !proc2_terminate U proc1_terminate ]

You could express safety as an assertion: the probability of something bad happening is zero. Liveness is an assertion that the long-run probability of something good happening is 1. But there are much more complex and powerful probabilistic properties you can express and check in PRISM, like saying that the 95th-percentile latency of an algorithm is less than some acceptable amount.

The PRISM website has a bunch of great examples. Here are statistics from a PRISM model of a gossip protocol. As nodes discover each other over time, they find shorter paths from node to node:

This kind of analysis and graphing is built into PRISM, it was born for this.

PRISM can only express models that are simple enough to analyze by solving equations. So the PRISM language has no arrays, no hashtables, no sets. It’s just scalar variables with float values, pretty much. That makes PRISM specs kind of awful.

Here’s the PRISM gossip protocol. This model has 4 nodes. Each node’s view of each other node is a separate variable, so that’s 16 variables:

// initial view of node 1 (can see 2 one hop away)
const int iv1_1_a = 2;
const int iv1_2_a = 0;
const int iv1_1_h = 1;
const int iv1_2_h = 4;

// initial view of node 2 (empty)
const int iv2_1_a = 0;
const int iv2_2_a = 0;
const int iv2_1_h = 4;
const int iv2_2_h = 4;

// initial view of node 3 (can see 2 one hop away)
const int iv3_1_a = 2;
const int iv3_2_a = 0;
const int iv3_1_h = 1;
const int iv3_2_h = 4;

// initial view of node 4 (can see 2 one hop away)
const int iv4_1_a = 2;
const int iv4_2_a = 0;
const int iv4_1_h = 1;
const int iv4_2_h = 4;

If you had 5 nodes, you’d need 25 variables. You can’t write any generic code. As the number of nodes grows, the code in this spec grows. This is Node 1’s code for talking to Nodes 2, 3, and 4:

// send to node 2
[push1_2_0] s1=3 & send1=id2 & i1=0 -> (i1'=i1+1);
[push1_2_1] s1=3 & send1=id2 & i1=1 & v1_1_h<4 -> (s1'=0) & (i1'=0) & (send1'=0);
[push1_2_end] s1=3 & send1=id2 & ((i1=1&v1_1_h=4) | (i1=2&v1_2_h=4)) -> (s1'=0) & (i1'=0) & (send1'=0);
// send to node 3
[push1_3_0] s1=3 & send1=id3 & i1=0 -> (i1'=i1+1);
[push1_3_1] s1=3 & send1=id3 & i1=1 & v1_1_h<4 -> (s1'=0) & (i1'=0) & (send1'=0);
[push1_3_end] s1=3 & send1=id3 & ((i1=1&v1_1_h=4) | (i1=2&v1_2_h=4)) -> (s1'=0) & (i1'=0) & (send1'=0);
// send to node 4
[push1_4_0] s1=3 & send1=id4 & i1=0 -> (i1'=i1+1);
[push1_4_1] s1=3 & send1=id4 & i1=1 & v1_1_h<4 -> (s1'=0) & (i1'=0) & (send1'=0);
[push1_4_end] s1=3 & send1=id4 & ((i1=1&v1_1_h=4) | (i1=2&v1_2_h=4)) -> (s1'=0) & (i1'=0) & (send1'=0);

Node 2 borrows Node 1’s code by horribly renaming all its functions:

push1_2_0=push2_1_0, push1_2_1=push2_1_1, push1_2_2=push2_1_2, push1_2_3=push2_1_3, push1_2_end=push2_1_end,
push1_3_0=push2_3_0, push1_3_1=push2_3_1, push1_3_2=push2_3_2, push1_3_3=push2_3_3, push1_3_end=push2_3_end,
push1_4_0=push2_4_0, push1_4_1=push2_4_1, push1_4_2=push2_4_2, push1_4_3=push2_4_3, push1_4_end=push2_4_end,
push2_1_0=push1_2_0, push2_1_1=push1_2_1, push2_1_2=push1_2_2, push2_1_3=push1_2_3, push2_1_end=push1_2_end,
push3_1_0=push3_2_0, push3_1_1=push3_2_1, push3_1_2=push3_2_2, push3_1_3=push3_2_3, push3_1_end=push3_2_end,
push4_1_0=push4_2_0, push4_1_1=push4_2_1, push4_1_2=push4_2_2, push4_1_3=push4_2_3, push4_1_end=push4_2_end

I’m only showing you a fraction of this code to preserve your sanity. Nodes 2 and 3 have the same kind of thing. This is an abomination. This code is banned by the Geneva Convention. The grad student who wrote this code was found lying on the floor of the computer science lab, drooling.

Actually it was probably generated by a program in another language. PRISM is like an assembly code for expressing Markov chains. This makes it tractable for PRISM’s equation solver to come up with exact solutions, but it’s obviously not a good language.

Start of the Art #3 of 4: Runway

In 2016, Diego Ongaro announced a formal modeling system called Runway (website, article, Andrew’s summary.) Ongaro had published Raft a couple years before, then he announced this very promising-looking thing but stopped working on it the same year. This is a snippet from a Runway specification of Raft.

function quorum(serverSet: Set<ServerId>[ServerId]) -> Boolean {
  return size(serverSet) * 2 > size(servers);
}

function sendMessage(message: Message) {
  message.sentAt = later(0);
  message.deliverAt = later(urandomRange(10000, 20000));
  push(network, message);
}

Runway’s syntax is familiar for programmers. And Runway has randomization built in. If Runway is in model-checking mode, it somehow tries all possibilities; I don’t know how. If Runway is in simulation mode, randomization works the way you’d expect, and Runway can gather statistics.

Here’s an elevator simulation written in Runway:

The visualization on the right needs lots of custom code, but the graph on the left is I think easy to create with Runway’s built-in tools. If we want to develop TLA⁺ tools in this direction, we should make this kind of graph convenient for spec authors to generate.

State of the Art #4 of 4: FizzBee

FizzBee is a new modeling language developed by a solo author, Jayaprabhakar “JP” Kadarkarai. Here’s an example FizzBee spec from the FizzBee docs:

cache.fizz

perf_model.yaml

atomic action Lookup:
  cached = LookupCache()
  if cached == "hit":
      return cached
  found = LookupDB()
  return found
 
func LookupCache():
  oneof:
    `hit` return "hit"
    `miss` return "miss"

configs:
  LookupCache.call:
    counters:
      latency_ms:
        numeric: 10
  LookupCache.hit:
    probability: 0.2
  LookupCache.miss:
    probability: 0.8

The spec describes a service that tries to look up data in a cache, then falls back to a database query. The algorithm is in cache.fizz on the left; Lookup and LookupCache are defined there. (I omitted LookupDB.) On the right is a config file that’s only for probabilistic modeling! It defines the cost of a call to the LookupCache function: 10 milliseconds of latency. PRISM avoids the valenced “cost” / “reward” terms and calls this value a “counter”, but it can be a float. Counters and probabilities are kept in this separate config file. They don’t clutter the spec, and they’re not needed for ordinary model-checking (i.e. checking safety properties).

The LookupCache function is nondeterministic, it either finds the data or doesn’t. The transitions are labeled as `hit` or `miss` in the spec, and their probabilities are defined as 0.2 and 0.8 in the config file. Again, the algorithm is kept separate from the probabilistic config.

FizzBee’s probabilistic modeler, a.k.a. the performance model checker, is a separate program from the safety model-checker. The former produces the state graph and saves it to disk; the latter reads the state graph and produces statistics. If you run the performance modeler on the files above, it prints:

Metrics(mean={'latency_ms': 84.4}, histogram=[(0.2, {'latency_ms': 10.0}), (1.0000000000000002, {'latency_ms': 103.0})])
   2: 0.20000000 state: {} / returns: {"Lookup":"\"hit\""}
   4: 0.72000000 state: {} / returns: {"Lookup":"\"found\""}
   5: 0.08000000 state: {} / returns: {"Lookup":"\"notfound\""}

The mean latency to look up one value is 84.4, and the return value of Lookup is “hit” 20% of the time, “found” 72% of the time, and “notfound” 8% of the time, as we’d expect from the defined probabilities.

Here’s the config file again, but with a probability distribution for the cost function.

perf_model.yaml

configs:
  LookupCache.call:
    counters:
      latency_ms:
        distribution: lognorm(s=0.3, loc=2)

FizzBee lets you choose any probability distribution supported by SciPy, which is basically all of them. Or you can bring your own histogram of costs: if you have experimental data, you can use that, the same as with the Java Modelling Tools. Marc Brooker said he wants a tool that uses “real-world data on network performance, packet loss, and user workloads,” and FizzBee allows this.

I think FizzBee’s design is smart. FizzBee has built in some important ideas: cost functions, probability distributions, and probabilities of state transitions. We don’t have to write a bunch of error-prone code for each spec, the batteries are included. What’s really nice is that the performance stuff is all in a separate file, so it doesn’t clutter the main spec. In TLA⁺ we try to keep model-checking details out of the main spec; FizzBee lets you do the same with performance modeling.

I don’t know how much of this is implemented. FizzBee is new and JP has been developing it solo. But that’s not important because we’re just gathering design ideas for what TLA⁺ might do.

A Menu for TLA⁺

I’ve grouped the features we’ve seen so far into three categories:

Some of the state-of-the-art tools can annotate state transitions with probabilities. Some support cost / reward functions, which FizzBee calls “counters”. PRISM has powerful statistical property expressions. In Runway and FizzBee, model-checking is compatible with performance modeling (and now TLA⁺ too), and FizzBee nicely separates performance modeling config. Some include convenient charting. They all support floating-point numbers. Some of them ship with common probability distributions for rates and cost functions. Some can use experimental data as a probability distribution (“bring your own histogram”). Some of them come with solvers that calculate answers precisely, but this limits the models to those that can be solved in closed form. Others use statistical sampling, which is imprecise but much more powerful.

Possible Syntax??

Here’s the moment I dread, when I suggest a syntax. Everywhere a spec has nondeterminism for model-checking, we need to replace that with some sort of probability distribution. E.g., this formula picks an element nondeterministically from a set:

\* MySpec.tla
SendMessage(m) ==
  \E messageIsDropped \in {FALSE, TRUE}:
    ...

If you want a distribution besides uniform, you could wrap the set (or any collection) in an operator like this:

\* MySpec.tla
SendMessage(m) ==
  \E messageIsDropped \in MessageLossProbability(FALSE, TRUE):
    ...

The operator does nothing in model-checking mode, but it’s like a label for a probability distribution.

\* MySpec.cfg
DISTRIBUTION
  MessageLossProbability = BooleanChoice(0.23)

In the config file, labeling this operator as a DISTRIBUTION means the model-checker treats it as either true or false, and it explores both possibilities nondeterministically. The performance-modeler knows that this is a particular probability, perhaps BooleanChoice is the name of a built-in utility to return its first argument with probability 0.23, otherwise its second argument. Like FizzBee, we use labels to tie the spec to the configuration.

We also need cost functions in TLA⁺. We could set the cost of each action to be a constant or a probability distribution. The distribution could come from real-world data, and it could be parameterized if the action is parameterized.

\* MySpec.cfg
COST
  SendMessage = Exponential(3.17)

This is the part of my talk I’m least sure about. If you hate this syntax, that’s fine, don’t get angry at the whole idea because you hate my syntax.

TLA⁺ with Probabilistic Solvers

If we implement any of this in TLA⁺, we’ll need some way to actually answer these probabilistic questions. In order of ambitiousness:

Just use TLC’s -generate option, generate thousands of behaviors, average the stats.
Use -generate, run until stats stabilize within some precision. Perhaps prune branches of the state graph as they stabilize. JMT does similar smart exploration of simulations.
Use PRISM’s solvers, by translating the state graph to PRISM code, or otherwise borrowing PRISM’s implementation.
Write a solver or solvers from scratch: translate the state graph to a Markov chain and find its steady-state probability distribution.

The latter two options will find closed-form solutions with precise answers, but severely limit our options for what models we can use for probabilistic questions. I think we should just use simulation, period.

TLA⁺ with Performance Modeling

If we actually had TLA⁺ with performance modeling, it could be really cool! One model could:

Express the algorithm.
Check correctness.
Evaluate performance.
Simulate “what-if” experiments using real-world inputs.

We could confidently explore optimizations: we’d know both whether a change to an algorithm is safe, and if it wins us anything.

Questions

I didn’t take questions during my talk, because I didn’t have any answers, so I asked the audience questions instead.

What syntax should TLA⁺ use for annotating state transitions with probabilities?
What syntax for cost functions?
How do we separate performance-modeling config from the spec and model-checking config?
Should TLC do the probabilistic checking, or another tool?
Could the TLA⁺ Foundation get new funding for this work?
Is any of this a good idea or should TLA⁺ stick to correctness?

Jesse's 2025 TLA+ Community Event Notes

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 08 May 2025 21:47:33 +0000

Here are my notes from the 2025 TLA⁺ Community Event. See also my colleague Murat Demirbas’s notes (he was one of the event’s organizers). The talks were all recorded, and the videos, slides, and abstracts are posted.

Some years there are TLA⁺ “conferences” co-located with industry conferences, and some years there are “community events” co-located with academic conferences. This year was a community event, the first TLA⁺ community event in North America. There were about 20 participants, evenly divided between academia and industry. I knew most of them already. If I recall correctly, the last TLA⁺ thing I attended was the 2021 conference; it had more participants, perhaps because it was co-located with the huge StrangeLoop industry conference.

ModelFuzz: Model guided fuzzing of distributed systems

Srinidhi Nagendra, Max Planck Institute for Software Systems.

Srini summarized his paper, Model-guided Fuzzing of Distributed Systems. Srini’s ModelFuzz is coverage-guided fuzzing of the implementation, where “coverage” is defined by the model. ModelFuzz executes an implementation test (which executes a workload and occasionally crashes or restarts a server), captures an execution trace, maps the trace back to a sequence of TLA⁺ model states, and guides the next round of tests to maximize coverage of the TLA⁺ model states. For distributed systems, maximizing this sort of coverage is a more efficient way to discover bugs than e.g. branch coverage like AFL uses.

Srini’s example is a Raft spec and several Raft implementations. He records all messages among Raft nodes, as well as events like node crashes, and replays this execution on the model. The spec had to be slightly restructured to make it easier to map implementation actions to model actions. He also needed to “abstract states” (something like a symmetry set in TLA⁺ config) so that states which differ in uninteresting ways are not considered distinct states that improve coverage. E.g., if you crash the leader 1000 times, you’ll get 1000 terms without truly exploring new states. Choosing the right model abstraction is an art. I don’t understand why he reinvented this idea instead of just using TLA⁺’s existing symmetry sets or views.

ModelFuzz generates implementation tests in a really surprising way. Srini created a central proxy through which all Raft messages pass. ModelFuzz counts the number of messages that are delivered to each server, and then “mutates” a test by buffering messages in the proxy and changing the order in which the proxy delivers messages, as well as perturbing the schedule of server crashes. There is a lot of nondeterminism in the system already and ModelFuzz is jiggling it even more. Srini shows that mutating tests strategically will moderately improve model coverage, compared to just rerunning the test many times and relying on inherent nondeterminism to improve model coverage, and compared to purely random mutation of tests. The charts he showed us seemed to show a rather small improvement, but he found a lot of new bugs!

It’s interesting that Srini was able to explore more of the state space with an intervention that’s reasonably easy to implement. But I think Antithesis is better. Model-guided fuzzing is still useful, but I predict that Antithesis or other deterministic hypervisors will soon be the overwhelmingly dominant means of actually doing the fuzzing.

Automating Trace Validation with PGo

Finn Hackett (presenting) and Ivan Beschastnikh, University of British Columbia.

Finn is my PhD fellow and will be my PhD intern starting in June. His PGo project translates a variant of PlusCal into Go. He wants to use trace-validation to check that the Go really matches the PlusCal; his trace-validator is called TraceLink. It uses log statements that are autogenerated along with the rest of the Go code, and it introspects the PlusCal to see what variables must be matched per trace line, and generates some TLA⁺ to validate a trace against the implementation. If validation fails, TraceLink provides some debug info, as TLA⁺ aliases that appear in TLC’s counterexample trace. This includes a list of enabled next-actions at the moment the trace diverged from the spec.

For each PlusCal label block (equivalent to a TLA⁺ action), PGo outputs critical section of Go code, which ends in commit/abort. If it commits, it updates the program’s state and logs the state transition for trace-checking’s sake. If it aborts it changes no state, but it still logs the state it observed, so even aborted critical sections can be trace-checked.

Finn’s system uses vector clocks to order log messages. This means there are many possible total orderings, and all of them should be valid behaviors according to the spec! (I keep forgetting and misunderstanding this.) A PGo-generated system should log all modeled variables on each state transition, and always increment the vector clock during every interaction between processes, so every one of the total orders is valid according to the spec. That gives TraceLink some options:

check every possible log ordering against the spec
check one arbitrarily-chosen ordering
check a set of orderings that satisfies some coverage metric

Finn has chosen a hybrid: TraceLink arbitrarily chooses one ordering of the log and checks it against the spec, but for each log message it asserts that each possible next state is valid according to the spec, before it advances to a single next state and repeats. He says this gives him a nice compromise between performance and completeness.

TraceLink generates from the log a TLA⁺ spec, which validates that the trace conforms with the original PlusCal spec of the program. This trace-checking spec could be huge, but TraceLink compacts it by factoring out common “shapes” of validation and storing the values in a binary file that TLC can read.

Finn gets diverse traces from the implementation by adding an exponentially-distributed random sleep after each critical section.

This work is specific to implementations generated by PGo, could be adapted with some manual effort (add PGo-style instrumentation to existing implementation) or by building a PlusCal-to-another-language compiler that automatically adds this instrumentation.

Translating C to PlusCal for Model Checking of Safety Properties on Source Code

Guillaume Di Fatta (presenting), Emmanuel Ohayon, and Amira Methni, Asterios Technologies.

Asterios is a defense contractor, they make a realtime microkernel for embedded systems. Guillaume and his colleagues wrote a tool that auto-translates the microkernel scheduler from C to PlusCal. The C is simple and self-contained, which helps. The translation progresses through a series of intermediate representations. He wrote a tiny stack-based VM in PlusCal to interpret the final IR. This is cool but outside my interests. An intriguing suggestion for future work: use model-checking to show the auto-translated PlusCal refines a more abstract spec.

TLA⁺ for All: Model Checking in a Python Notebook

Konstantin Läufer (presenting) and George K. Thiruvathukal, Loyola University Chicago.

Konstantin talked about teaching formal methods to undergrads at his school and globally. He described where FM fits in a curriculum for CS students, particularly among other classes about software testing, which increase in power as they work through the curriculum. (He mentions the useful ZOMBIES mnemonic for unittesting!) He wants to integrate FM into existing classes, e.g. a data structures class could use Alloy or LEAN. Finally, he shows that he’s created a browser-based zero-install environment for running TLC.

Formal models for monotonic pipeline architectures

J.-P. Bodeveix, A. Bonenfant, T. Carle, Mamoun Filali (presenting), C. Rochange.

They want to analyze the Worst-Case Execution Time (WCET) for pipelined hardware architectures. They’re concerned about “timing anomalies” e.g. where a cache hit makes execution slower. Mamoun didn’t discuss performance analysis in TLA⁺, though, he instead walked through a TLA⁺ spec and safety properties in detail.

TLA⁺ Modeling of MongoDB Transactions

Murat Demirbas (presenting) and Will Schultz MongoDB

MongoDB introduced single-document transactions in version 3.2 (2015), multi-document transactions in version 4.0, (2018), and sharded transactions in version 4.2 (2019). Murat described the layers of the transaction algorithm: storage, replica set, sharded cluster. The algorithm is hard to reason about and verify: it was developed incrementally, its use of ClusterTime and speculative majority adds complexity, and some of the hard bugs are in fault-tolerance and the interactions of transactions with DDL and chunk migration between shards. For now, Murat’s and Will’s TLA⁺ spec disables chunk migration and ignores DDL.

To check isolation, our spec maintains a history variable “ops” where it stores the history of transactions. We use Crooks’s client-centric isolation definitions, modeled in TLA⁺ by Soethout. The interactions between transactions and timestamps are key to isolation guarantees.

Interesting audience question: Crooks’s formalism doesn’t catch phantoms due to predicate reads, how do we ensure we’re preventing phantoms? Murat: that’s only a problem with serializable isolation, but the strongest we provide is snapshot isolation.

We use model-based test case generation to check conformance of the storage layer spec and implementation. Since the spec is modular, the storage spec is separate from the other aspects of the transactions spec, which helps.

Murat and Will analyzed the permissiveness of our transaction protocols compared to the behaviors allowed by Crooks’s formalism. We allow 70-80% of the behaviors that are theoretically allowed for snapshot isolation and read-committed—this suggests if we were more permissive we could allow more parallelism. I have a draft of the paper that I haven’t read yet, now I’m really curious how they measured this.

Are We Serious About Using TLA⁺ For Statistical Properties?

A. Jesse Jiryu Davis MongoDB

This was my presentation. Here’s the video and a detailed writeup. In summary: Marc Brooker in 2022 asked for tools that model performance and check safety, with a single spec. Markus Kuppe and Jack Vanlightly that year presented a way to do it in TLA⁺, but I think it’s an impractical prototype because it’s missing essential features. I asked, are we serious about making TLA⁺ do performance modeling? I reviewed some existing tools for ideas to inspire us: Java Modelling Tools, PRISM, Fizzbee, Runway. I proposed how we might do performance modeling in TLA⁺, and asked the audience whether we should proceed and how.

It’s never been easier to write TLA⁺ tooling!

Andrew Helwer

Andrew described the existing infrastructure on which you can build your own TLA⁺ tools. There are a bunch of TLA⁺ parsers, SANY is the only full-featured one. Andrew’s Tree-Sitter grammar is good for prototyping new TLA⁺ notation because it tells you how much ambiguity you’d introduce. For model-checking there’s TLC, Apalache, and now Will Schultz’s Spectacle can do some model-checking. There’s Snowcat (a type-checker), two auto-formatters (tlaplus-formatter and tlafmt), LSP (language server protocol), Andrew’s TLAUC (Unicode converter), and the TLA⁺ VS Code extension.

The main tools have a lot of legacy code, defined as “code without tests” or, according to Andrew, “code that no one understands.” SANY & TLC were started 25 years ago, overhauled 10 years ago for version 2, those authors are gone now. Markus knows more than anyone, but there are large sections of the code he doesn’t know / no one knows. The model-checking code is straightforward, but TLAPM (the proof system) is esoteric. TLAPM was started with a grant at INRIA but is not vigorously developed now. Apalache was spun off by Informal Systems but it hasn’t been adopted by the TLA⁺ Foundation; it’s quiescent.

Andrew wants a world where we’re unafraid to make large changes to core TLA⁺ tools. He’s written a lot of parser tests with his TLA⁺ Foundation funding, but we need more tests. We should onboard new core contributors: Andrew is writing onboarding docs and tutorials for contributors. He’s also written how to create a new TLA⁺ tool.

He presented some possible future directions, like simplifying and modernizing TLA⁺ syntax, and improving model-checking performance 1000x.

Announcements

Markus says the TLA⁺ issue tracker in GitHub now labels some issues as TLA⁺ Foundation funding candidates. He describes the grant proposal process.

Markus demonstrated, with a sped-up video, that GitHub Copilot can solve the Diehard problem with TLA⁺ in 4 minutes of reasoning, with some human intervention. We should explore TLA⁺ plus LLMs more. The Foundation and NVidia came up with the “TLAI” challenge, for novel combos of TLA⁺ and AI.

We’ll have a TLA⁺ conference / community event next year, but we don’t yet know where or when.

Extracurriculars

Andrew Helwer and I visited Niagara Falls the day after the conference. He’d never seen them. I saw the falls when I was a kid and hardly remember, aside from a memory of a helicopter ride I took over the falls with my mother. She’d said, “This is so expensive, promise you won’t blink for the whole ride.”

I give the falls 3 out of 5 stars: lots of water, but not very tall, and you can’t see much because of all the mist.

From the delightful “Subpar Parks”.

We had a great time in the vast and rusty old powerplant, a total steampunk playground, with a half-mile tunnel where you walk deep underground from the subterranean turbines to the base of the falls. There’s a hagiographic exhibit about Nikola Tesla, the wronged genius who single-handedly invented electricity, radios, and VTOL aircraft? The show turned out to be a propaganda operation by the Serbian government—Tesla was born in Serbia.

Andrew and I hiked around the Niagara Glen Nature Reserve, where we saw some lovely mossy boulders, and Canada goose families with floppy goslings.

The futuristic images are from trading cards published by the Echte Wagner margarine company in early 20th Century Germany.

Holy Week at Christ Episcopal Church

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 27 Apr 2025 08:04:34 +0000

Holy Week services (Maundy Thursday, Good Friday) at Christ Episcopal Church in Poughkeepsie. The church has operated since the 1760s; the current building was erected in 1889. My friend Emily is the church’s assistant rector, she was ordained there last year. She invited me to photograph the services this month.

Shot with a Nikon F3 and a Nikkor 50mm f/1.2 on Portra 800 35mm film.

Images © A. Jesse Jiryu Davis

Mohonk Preserve Rangers, part 2

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Fri, 11 Apr 2025 19:36:29 +0000

April 9, 2025. Mohonk Preserve rangers training for rescue operations. I spent a day with them earlier, in March.

Images © A. Jesse Jiryu Davis

How Long Must I Test?

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 07 Apr 2025 22:36:05 +0000

When you’re hunting for bugs with a nondeterministic test, how many times should you run the test? The program might be deterministic: the same input produces the same behavior. Presuming you’re confident this is the case, you could fuzz the input for a while, hoping to catch bugs. But if your program is nondeterministic, you must not only give it random inputs, you must re-test it on the same input many times, hoping to achieve high branch coverage and catch rare bugs.

I was at the first annual Antithesis conference this week. Antithesis has built a hypervisor that makes nondeterministic tests into deterministic ones, and they built a coverage-guided fuzzer to search for deep bugs. (I evaluated an early version of their product in 2020 for MongoDB, we’ve been a customer ever since.) So I’ve been wondering about Antithesis and other testing methods: how efficient are they, and how long do we have to run them to be confident we’ve found most of the bugs?

Deterministic Programs

You’re in luck, inputs and program behaviors are 1:1. But some bugs are triggered by only a subset of inputs, and there are too many possible inputs for you to test them all. How many should you test?

My colleague Max Hirschhorn told me about a 2020 paper, Fuzzing: On the Exponential Cost of Vulnerability Discovery. The paper itself is confusing and contradictory, so I’ll try to summarize: If you choose inputs at random (with replacement), the number of inputs required to find each additional bug rises exponentially. This applies to stupid totally-random fuzzers and also, depressingly, to smart coverage-guided fuzzers. I assume the coverage-guided fuzzers have a much smaller exponent, though.

The first half of the paper shows charts from experiments with fuzzers (AFL and LibFuzzer) and hundreds of programs. The programs must all be deterministic, because the authors don’t consider that one input could produce multiple outcomes. They discover an “empirical law” that the marginal number of inputs to discover a bug increases exponentially in nearly all cases.

The authors go on to mathematically model a dumb blackbox fuzzer that chooses inputs uniformly randomly from a domain D, which can be subdivided into S subdomains called species. One species is one kind of outcome recognized by the fuzzer, e.g. a crash at a particular instruction, or a particular path through the program’s branches. Some of the species are the bugs we’re hunting. Each input has one species, and the probability of a random input belonging to a particular species i is p_i, which is the proportion of inputs in D that lead to i. The fuzzer chooses a subset of n inputs from D, and the expected number of species it discovers is:

This is intuitive: the probability of discovering a single species is the inverse chance of failing to discover it n times, and the expected number of discoveries is the sum of those probabilities over all the species.

So far this is straightforward, but why does it lead to an exponential rise in the marginal number of inputs to discover another species, i.e. an exponential decline in the rate of species discovery over the course of a fuzzing campaign? Alas the authors give two contradictory explanations. First, perhaps real programs’ p_i probabilities are power-law distributed, i.e. the most probable species is much more likely than the next most probable and so on, flattening out towards the tail where rare species are about equally improbable. This theory matches the authors’ experiments in the first half of the paper, but it’s not an explanation for why species should be distributed this way.

Immediately after explaining this theory, the authors blithely contradict themselves with a different theory:

When collecting baseball cards, the first couple of cards are always easy to find, but adding a new card to your collection will get progressively more difficult—even if all baseball cards were equally likely. This is related to the coupon collector’s problem … covering one more branch or discovering one more bug will get progressively more difficult—so difficult, in fact, that each new branch covered and each new vulnerability exposed comes at an exponential cost.

The coupon collector’s problem applies when species are all equally likely. As n grows, the number of trials to discover n species (“collect n coupons”) grows like n log(n), which is much slower than exponential. But if there is a fixed number of species n and you’ve discovered k of them, what’s the marginal cost to discover one more? How does it compare to exponential growth?

It’s a poor fit. The exponentially increasing difficulty of finding a new species is not well-explained by the coupon collector’s problem, the power-law distribution must be the real reason. But that just poses a new question: why are the probabilities of species power-law distributed?

Here’s another question: why aren’t coverage-guided fuzzers asymptotically better than this? Indeed, at the Antithesis conference, Zac Hatfield-Dodds suggested that they are. He gave a talk about his property-based testing library Hypothesis, and threw out a thought-provoking aside. If I heard him correctly, he said “coverage-guided fuzzing reduces the time to find a bug from exponential to polynomial.” This paper seems to contradict him, but perhaps I don’t understand his meaning. I need to think or investigate more.

Nondeterministic Programs

Now things get really hard. Multithreaded programs interleave instructions nondeterministically, and distributed systems suffer unpredictable network delays, message loss, and clock skew. When you test these programs, not only must you try lots of inputs, but you must try each input many times.

If you run the same test of a nondeterministic program repeatedly, you’ll probably get different outcomes, but not efficiently: mostly the program will just follow the same path over and over without discovering new bugs. You can explicitly increase randomness by introducing random network delays or various faults, as Jepsen does.

You can find concurrency bugs in multithreaded programs much more efficiently if you control thread scheduling. Finn Hackett, a MongoDB PhD fellow and intern, showed me the paper A Randomized Scheduler with Probabilistic Guarantees of Finding Bugs, in ASPLOS 2010. The authors point out that even a simple multithreaded program has an overwhelming number of possible schedules: n threads that collectively execute k instructions have on the order of n^k schedules. But only some of the scheduling choices matter for reproducing any of the program’s concurrency bugs. This paper defines a nice term, bug depth, “the minimum number of scheduling constraints that are sufficient to find a bug.”

For example, in the paper’s Figure 1(a), a bug is reproduced if Thread 2 accesses a variable before Thread 1 initializes it. This bug has one scheduling constraint, i.e. depth 1. In Figure 1(b), the bug requires Thread 1 to run an assignment after Thread 2 runs an if-statement and before it runs a method-call, so it has depth 2. The authors describe a thread-scheduling algorithm that finds any single bug of depth d with probability 1/nk^d-1, much better than the theoretical lower bound of 1/n^k, i.e. the probability of picking one schedule out of all possible schedules. You can expect to find a depth-1 bug in an n-threaded program in n tries and a depth-2 bug in nk tries.

Their algorithm is called Probabilistic Concurrency Testing (PCT). You provide PCT with a program that executes up to k instructions on n threads. You provide a desired bug depth d, and an input that could trigger a bug of depth d. PCT’s job is to find a schedule that does trigger it. PCT assigns random initial priorities d, d+1, …, d+n to the n threads. PCT chooses d-1 random “change points” in the program. It allows the program to execute one instruction at a time on one thread: the highest-priority runnable thread, i.e. the thread with the largest priority number that isn’t waiting for anything. When a thread stumbles upon the i^th change point, PCT changes the thread’s priority to i and continues. PCT uses higher numbers for threads’ initial priorities (≥d) than it uses for change points (<d), so that a thread tends to be unscheduled when it hits its first change point, perturbing the system and increasing the chance of bugs.

Here’s Figure 6(b) from the paper. It shows the same depth-2 bug as before. PCT reproduces it by assigning priority 3 to Thread 2, and priority 2 to Thread 1 (white circles), and inserting a change point (black circle) that changes Thread 2’s priority to 1. The probability of PCT assigning these initial priorities is 1/n, and its probability of choosing this change point is 1/k. Thus there’s a 1/nk chance of finding this bug in one try, which matches the authors’ promise of finding a depth 2 bug in nk tries on average.

In practice PCT is much more efficient than its worst-case bound, since there are usually many ways to reproduce a given bug.

The authors implemented PCT for real:

For fine-grained priority control, we implemented PCT as a user-mode scheduler. PCT works on unmodified x86 binaries. It employs binary instrumentation to insert calls to the scheduler after every instruction that accesses shared memory or makes a system call. The scheduler gains control of a thread the first time the thread calls into the scheduler. From there on, the scheduler ensures that the thread makes progress only when all threads with higher priorities are blocked.

I think by “shared memory” they mean the heap, which is shared among threads. In the abstract description above we called the total number of instructions k, but the authors’ implementation of PCT only places change points before synchronization points, like pthread calls or atomics, and k is the number of these synchronization points. Surprisingly, PCT can run huge programs like Mozilla at one-third normal speed. The paper’s evaluation section describes the bugs they found and how quickly, and the various optimizations and heuristics they added to the algorithm. This part interested me:

Final Wait: Some concurrency bugs might manifest much later than when they occur. We found that PCT missed some of the manifestations as the main thread exits prematurely at the end of the program. Thus, we artificially insert a priority change point for the main thread before it exits.

There must be hordes of rarely-manifesting use-after-free bugs where the main thread deletes global variables as it exits, and background threads sometimes wake up just before the program terminates.

Deterministic Simulation Testing

At MongoDB we run certain concurrency tests, which we know are highly nondeterministic, many times each night. There are rare bugs that pop up once every few months, or few years. This is bad, because we don’t know which change introduced the bug, and once we diagnose it, we can’t be confident we fixed it.

Some companies like FoundationDB built deterministic simulation testing into their code from the start. When FoundationDB was acquired by Apple, some of the FoundationDB team left to create Antithesis and bring deterministic testing to projects that didn’t build it in. Running MongoDB’s nondeterministic tests in Antithesis has been helpful, and I expect we’ll use them more.

Inconclusion

Distributed systems engineers (myself included) typically aren’t knowledgeable enough about randomized testing and we don’t deploy it thoughtfully. We should ask rigorously, exactly how much confidence do I gain with each trial, and how can I test nondeterministic programs efficiently? And the most vexing question of all: when can I stop? I don’t know the answers yet.

Images: Fabre’s Book of Insects (1921)

Zen priest Kenzen in New York City

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Fri, 04 Apr 2025 17:39:42 +0000

April 1, 2025. Zen priest Kenzen, formerly named Gessho, who was recently ordained at the Village Zendo in New York.

Images © A. Jesse Jiryu Davis

Mohonk Preserve Rangers

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 26 Mar 2025 11:43:51 +0000

March 25, 2025. Mohonk Preserve rangers training for rescue operations. They practiced building anchors, ascending and descending ropes, rescuing an unconscious climber hanging on a rope, and securing a patient in a litter. (More photos here.)

Images © A. Jesse Jiryu Davis

Gravity Vault Climbing Team

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 19 Mar 2025 17:27:54 +0000

February 18, 2025. The local youth climbing team training at the gym.

Images © A. Jesse Jiryu Davis

Red Rocks

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 19 Mar 2025 10:58:13 +0000

I climbed with five friends in Red Rocks, outside Las Vegas. We had a medium-disappointing week. The sandstone there is fragile for a day or two after it rains, and it rained twice, so we were relegated to mediocre limestone climbing instead of world-class sandstone for most of the trip. Even on the limestone days our time was cut short by cold winds and occasional hail. My major objectives like The Fox and Cactus Massacre will have to wait. I never took out my Canon, but I had some fun with my phone camera.

Images © A. Jesse Jiryu Davis

Review: SwiftPaxos: Fast Geo-Replicated State Machines

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 06 Feb 2025 21:32:59 +0000

SwiftPaxos: Fast Geo-Replicated State Machines, in NSDI 2024, proposes a Paxos variant for networks with high latency, and different latencies between different pairs of nodes. Here’s a video of my presentation to the DistSys Reading Group, and a written review of the paper below.

Previous Paxi

Paxos, as you know, is Leslie Lamport’s solution to the problem of consensus among unreliable nodes. The original Paxos achieved consensus on a single value (or “decree”), and it takes 5 one-way network delays in the happy path, counting client-server delays as well as inter-server. The standard analysis of Paxos’s performance assumes that each one-way network delay is the same.

Multi-Paxos

Multi-Paxos is more practical. It handles the real-world scenario where a group of nodes must agree on a sequence of commands that modify a shared state machine. There’s a long lived leader, and the client sends its request to the leader. The leader broadcasts the client’s command to all the nodes. It commits the command once it hears replies from a majority of nodes, at which point it can execute the command and send the result back to the client. In the happy case, Multi-Paxos takes 4 one-way network delays per value (again counting client-server and inter-server chatter).

Diagram from Wenbing Zhao, “FastPaxos Made Easy”, 2015

FastPaxos

FastPaxos reduces network delays in the happy case. The client sends its request directly to all nodes at the same time. When the leader hears from three quarters of the nodes, who all agree to put the same command in the same slot, the leader can commit and execute the command and reply to the client. This saves a single, one way network latency, for a total of 3 per value.

Diagram from Wenbing Zhao, “FastPaxos Made Easy”, 2015, plus my scribbles

Why is the “fast quorum” three quarters of the nodes? When I reviewed the Nezha paper a few months ago, I tried hard to understand this, and I think I succeeded. Then I forgot. While I read SwiftPaxos, I tried to remember. I failed. So let’s assume that three quarters is the size of a fast quorum, and that avoids certain problems that a mere majority would cause.

In FastPaxos, different clients are all broadcasting simultaneously to all of the nodes, so there can be conflicts: different nodes can receive different commands, or in different orders. Some of the nodes could tell the leader, “I want command c₁ in slot i”, others could say, “I want command c₂ in slot i.” The leader resolves a conflict by starting a new round using classic Multi-Paxos. Thus conflicts makes “Fast” Paxos slower than Multi-Paxos.

If you use a geo-distributed deployment with clients in many regions, conflicts are more likely, because their broadcasts reach different nodes with different delays. And in a geo-distributed deployment, restarting consensus after a conflict is very expensive!

SwiftPaxos

This paper introduces SwiftPaxos. It claims to improve on FastPaxos in the happy case, with only 2 one-way message delays—just from the client to the leader and back! Even in the slow path, there are only 3 one-way delays. I’ll compare all these paxi and describe SwiftPaxos in more detail, and we’ll see if we agree.

Conflict Avoidance

(image source)

Sequels to FastPaxos attempt to avoid conflicts with various strategies:

Egalitarian Paxos (EPaxos) is leaderless. Clients broadcast all commands to all nodes. Each node tracks each command’s dependencies and waits until a strongly-connected component (SCC) of the dependency graph appears. Then all nodes execute commands in the SCC in the same deterministic order, even when some nodes think they observe dependency cycles. The size of the SCCs each node must track, and the delay before a command is executed, can be large (theoretically unbounded).
Nezha is similar, except it introduces deadline ordered multicast (DOM), which relies on reasonably synchronized clocks. DOM schedules each command to run sometime in the near future, and hopes that on three quarters of the nodes, the same set of commands will arrive before their deadline, so that will reduce conflicts. Nezha also falls back to a slow path when DOM makes a mistake. But Nezha recognizes that different orders are okay among non-conflicting commands (“commuting” commands) because their order doesn’t matter relative to each other.
SwiftPaxos. Once again, nodes track dependencies among commands. SwiftPaxos never has dependency cycles (stay tuned). Like Nezha, SwiftPaxos ensures all nodes run conflicting commands in the same order, but they can run commuting commands in any order. SwiftPaxos has a low-key leader who’s mostly needed for accelerating the slow path.

What Are Nodes Agreeing To?

To understand all these systems, I found it helpful to ask what value nodes are agreeing to.

Paxos, Multi-Paxos: nodes agree that command c is in slot i, so they agree about the total order of commands.
EPaxos, Nezha, SwiftPaxos: nodes agree that command c has dependencies s.

In EPaxos there’s no consensus on ordering! Only after consensus do the nodes look at the dependency graph that they agreed to, and then resolve cycles and run the commands in the same partial order, where conflicting commands are ordered the same on all nodes. So order is part of the consensus protocol for Paxos and Multi-Paxos, but not for EPaxos. Nezha and SwiftPaxos are similar to EPaxos, but their consensus protocols prevent cycles.

When Can a Node Run a Command?

Paxos, Multi-Paxos: as soon as it’s committed.
EPaxos: after its dependencies (special rules for cycles).
Nezha, SwiftPaxos: after its dependencies (never any cycles).

In Paxos and Multi-Paxos, as soon as command c is committed, all the commands that c can depend on are already committed, because c is already placed into a total order. In EPaxos, a command can run after the strongly-connected component to which it belongs has been identified, and its dependencies have run. If there’s a cycle, EPaxos breaks it by running the command with the lowest id number first. In EPaxos a command might wait for an unbounded amount of time before it can run: an adversarial workload can blow up the size of these strongly connected graph components. Since Nezha and SwiftPaxos prevent cycles, there’s a bounded delay between nodes agreeing about a command and executing it.

Semi-Strong Leader

The role of the leader is where SwiftPaxos sharply differs from its predecessors. Every SwiftPaxos quorum must include the leader, but clients can propose commands to any node. The SwiftPaxos leader is sort of first among equals: it’s not as strong as in Multi-Paxos, but stronger than in Egalitarian Paxos (which of course has no leader).

In both Multi-Paxos and SwiftPaxos, there are N nodes with ids 0 through N - 1, the leader of ballot b is the node with id = b mod N, and this leader keeps its job until failure.

As in FastPaxos, SwiftPaxos has slow quorums and fast quorums. In SwiftPaxos, a slow quorum is a majority of replicas, including the leader. This is a departure from other protocols! A majority is not enough to commit a command, the majority must include the leader.

Each fast quorum is either:

More than ¾ of nodes, including the leader, or
A specific set of nodes which is a majority including the leader.

You can configure the system to use one kind of fast quorum or the other. The first is more fault-tolerant, the second is faster. Read Section 5.5 where the authors explore the tradeoffs; it’s subtle.

The SwiftPaxos Algorithm

Let’s talk about the algorithm already. How does it actually work?

First, a client sends command c to all the nodes, the same as in FastPaxos. Each node N does the following:

Compute c’s dependency set: prior uncommitted conflicting commands. For example, c reads some key x, and node N already knows of an uncommitted command that modifies x, so that uncommitted command is in the dependency set of x.
Broadcast c’s dependency set to the other nodes.
Wait for messages agreeing about c’s dependencies from a fast quorum.
If a fast quorum agrees about c’s dependencies, follow the fast path:
- Wait until all of c’s dependencies are committed.
- Commit c.
- Wait until all of c’s dependencies have executed on N.
- Execute c.
Otherwise, if no fast quorum agrees about c’s dependencies, follow the slow path:
- The leader proposes its value for c’s dependencies.
- Do normal Multi-Paxos agreement.

In this figure from the paper, there are three clients. The circled nodes p1-3 are a fast quorum. In this case we’re using SwiftPaxos’s quirky configuration, where the fast quorum is a specific set of nodes that forms a majority and includes the leader.

Figure 1 from the paper.

The three clients submit commands y, x, then z, in that order. But due to geo-distribution, the clients have different latencies to different nodes, and each node gets these commands in a different order:

Figure 1 plus my scribbles.

Some of these commands commute and some of them don’t.

All nodes agree about y’s dependencies. Node p1 decided that y does not conflict with x, and so y has an empty set as its dependencies on p1. Nodes p2 and p3 didn’t have to think at all: they had no prior uncommitted commands when y arrived, so they also decided that y’s dependency set is empty.

There is some disagreement about command x. It arrived before the other commands on p1, so its dependency set is empty there. On p2, x arrived after y, but p2 decided it doesn’t conflict, so its dependency set is empty there too. But on p3, x arrived after z, and these commands do conflict, so x depends on z. Thus p3 has a different opinion about x’s dependency set than the other nodes. Since we’ve configured SwiftPaxos with a special fast quorum consisting of these three nodes, any disagreement among them aborts the fast path and forces SwiftPaxos to fall back to the slow path.

Command z has the same outcome: it arrived before conflicting commands on some nodes and after conflicting commands on other nodes, so z has to fall back to the slow path too, i.e. normal Multi-Paxos consensus. A unique feature of SwiftPaxos’s fallback is that a node can vote twice in the same ballot: once for a fast-path proposal and once for a slow-path proposal. Since the leader must be a member of both the fast and the slow quorum, it can ensure only one proposal wins.

Apart from the consensus protocol, there are also optimizations in the client-server protocol. The leader optimistically executes any command immediately, before it’s committed, and sends the result to the client. If the leader and client learn the command was committed they can trust this result. Otherwise, they discard it. This saves some network latency compared to the normal pessimistic execution. On that fast path, a client submits its command to all nodes, receives agreeing replies directly from a fast quorum of nodes, and learns that its command was committed in just 2 one-way delays. The slow path takes only 3.

Why Is This Simpler Than EPaxos?

SwiftPaxos is simpler than EPaxos, believe it or not. Why? I think it’s because EPaxos commits commands in any order: a command c’s dependencies must all be known, but not necessarily committed yet, before c is committed. That’s how EPaxos ends up with cycles, and why it must build up a certain kind of subgraph (a strongly connected component) before it can start executing commands. SwiftPaxos waits to commit a command until after all of its dependencies are committed. This is why there are no dependency cycles, and why SwiftPaxos doesn’t suffer unbounded delays like EPaxos. (I didn’t understand this until I read the proof in the appendix.)

After a leader change, there’s a recovery protocol in SwiftPaxos which is more complicated than what we’ve seen so far. The recovery protocol does have to deal with cycles. I wonder: if SwiftPaxos used Raft elections, which choose a member from the majority with the longest log, might that prevent cycles during recovery?

Their Evaluation

The authors have a nifty approach to evaluation. They have sites scattered around the earth. Eight sites run clients, 3 sites run a server each, and 2 sites run both clients and a server. At the best client site, in the middle row of the chart, clients are near the nodes, and at the worst site at the bottom clients are far from them, so we can see the effect of getting a fast quorum agreement more or less often. The top row shows the average across all client sites. They compare how various protocols perform given different geo-locality and different rates of conflicts among commands.

SwiftPaxos permits two ways of defining a fast quorum, and for this experiment they choose the second option: a specific set of nodes, a majority including the leader, constitute the only fast quorum.

Figure 7a from the paper.

The y axis is each protocol’s latency speedup relative to Paxos, and the x axis is how often commands conflict. SwiftPaxos is the gold line. SwiftPaxos almost always beats the other protocols, of course, because this is the SwiftPaxos paper. But notice how some of the other protocols, particularly CURP+, are a little better at low conflict rates at the worst site. I don’t know the CURP+ protocol, so I don’t know why it’s better in this scenario.

Figure 7b from the paper.

Here’s the same average, best, and worst sites, evaluated by the cumulative distribution function of latency. The conflict rate is fixed at 2%. Again, SwiftPaxos beats all the others at the best site and the average site. But a couple of the other protocols are actually better, at this 2% conflict rate, at the worst site where the client is very far from a fast quorum.

My Evaluation

The paper is well-written, but the protocol is complex and hard for me to understand, probably because I don’t know EPaxos well. I had to read the appendix to understand why it avoids dependency cycles, unlike EPaxos.

I’ve heard people say EPaxos is impractical, mainly because of the unbounded execution delay and accumulation of state at the nodes. EPaxos is interesting research and often cited, but not actually used. Perhaps SwiftPaxos is the practical sequel to EPaxos? It’s a new paper, we’ll have to see how the community responds and builds upon it. The savings in one-way network delays seems significant.

Audubon, the American swift.

Mixed-Media Climbing Art

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 05 Feb 2025 17:22:58 +0000

Some of my photographer friends hand-make books, sculptures, etc. from their photos. They inspire me to be more inventive.

Dustin holding his own mixed-media climbing art, framed with old climbing slings.
Dustin training on his hangboard, framed in climbing chalk.
Chris climbing Bonticou Crag, framed with local rocks.

Images © A. Jesse Jiryu Davis

New Paltz in the Fall

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 28 Jan 2025 19:23:43 +0000

October 22, 2023. There was stupidly dramatic light and clouds over New Paltz in the late afternoon that day. I forgot to publish this until now.

Images © A. Jesse Jiryu Davis

Dustin Portzline

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 28 Jan 2025 08:28:37 +0000

January 27, 2025. My climbing mentor, rock guide and artist Dustin Portzline, at home.

Images © A. Jesse Jiryu Davis

Which Beings Are Sentient?

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 25 Jan 2025 16:40:05 +0000

Which animals are sentient, and how can we liberate them from suffering? This is my January 23, 2025 dharma talk at the Village Zendo. I reviewed the philosopher Jonathan Birch’s book “The Edge of Sentience” and shared good news about humanity’s moral progress regarding animal welfare. Watch the video above, read the transcript below, or subscribe to my podcast.

TRANSCRIPT

I would like to introduce you to somebody. This is Sojourner Truth Armstrong Davis, she is a Chinese dwarf hamster, and she is a little over a year old. We got her at a pet store across the river in Poughkeepsie, so her life is now more than half completed. She was in a cage with some sisters when we got her and we brought her home, and she’s been alone ever since. I mean, we’re her friends, but it’s unclear if she knows that we exist.

Actually, it’s unclear if she knows she exists.

This “problem of other minds” is a classic problem in Western philosophy. This question—what do we know about other beings’ internal experience, based on what we can observe about their behavior or their anatomy or by measuring their brains with EEGs or whatever? What do we know about other minds, and specifically, do we know whether they’re conscious or not? It seems obvious that I’m conscious—I know that I exist. I experience something, as Descartes said, “I think therefore I am.” Buddha said that this is kind of overstated, that based on our experiences, we tend to conclude that our consciousness is a lasting, separate thing that we need to protect and promote, whereas, in fact, my consciousness is more like a candle flame or a wave on an ocean or a bubble on a stream, as it says in the Diamond Sutra, I think. Temporary and not separate. But it doesn’t mean that I don’t exist! My consciousness is happening now, and other humans mostly behave like they’re conscious too. So it seems reasonable to guess that they are, but I can’t prove it. And with nonhuman animals, it’s much harder.

In what ways is Sojourner’s mind like or unlike mine? I’m pretty sure she has no language, although she makes a lot of noise. Does she think? She has some behaviors that sort of seem like planning. We feed her a variety of seeds and she doesn’t eat them all at once. She caches them around her cage. Well, if we give her fresh food, she eats right away. But if she gets food that seems like it’s going to last, she hides it. So this all seems pretty sophisticated. Mostly she saves her food in her little house, inside her cage, which is her nest, but then she also stores it in a few other places. Is she calculating that she wants backups, in case the main cache of food in her nest is lost somehow? I don’t know. She often shells a seed before she hides it. Is that to save space? I don’t know. When it’s cold in our house, she’ll stuff her little house with fluff for insulation. And one time, when we’d been gone for about a week and the house was quite cold, we set the thermostat down to 50 degrees, she had stuffed fluff around the outside of the house and piled it on top of her house to increase the insulation, which we hadn’t seen before.

Is that because she understands heat transfer and insulation in the way that a human does, or is it just an instinct? Her ancestors, who did that kind of thing at random, survived and passed on that instinct to her. I don’t know. Instinct seems more likely.

My own cognition isn’t as clear cut either! When it’s cold in the house, I get an extra blanket from the closet. Is that because I understand heat transfer and insulation, or is that an instinct? Or is that operant conditioning: I tried it in the past, and it felt warmer, so now I’ve trained myself to do that. Or is it imitation: I’m mimicking a behavior I saw my mother do when I was a child. As both Buddha and Freud have observed, we’re not very good at observing the operations of our own minds.

The world of her senses is very different from mine. Her hearing is about as sensitive as mine. Her ears are much cuter, but they’re not necessarily better. I can hear low tones better than she can, whereas she can hear much, much higher tones. So the world would sound different to her than to me. I couldn’t find any studies on dwarf hamster olfaction, but if she’s anything like a mouse, then she has three or four times as many olfactory genes as I do. So she can smell a much wider variety of scents, and she’s much more sensitive to subtle smells than I am.

Her vision is not very good. It’s blurry. She can’t focus on things unless they’re right in front of her, in the middle of her vision. She can barely see green and blue. She can’t see red at all. But her night vision is fairly good, and she has a very wide peripheral angle. So she seems to be optimized for detecting an owl swooping down on her in the dark, and for detecting a dark corner that she can run to to escape it. She also has very long whiskers. They’re half the length of her body, and she uses her whiskers and her smell to navigate. I think that the sensations of her whiskers must be very vivid to her.

I have no idea what any of this is like. What is it like to have her mind and body? The philosopher Thomas Nagel, in 1974 wrote an essay, “What Is It Like To Be A Bat?”, where he talked about a bat’s echolocation. He argued that with the bat’s echolocation, even if you understood everything about the physics of its ear and its neurons and its nerves, even if you could model this all with equations, you still wouldn’t understand what it’s like to be a bat. And with Soji as well, her sensorium and her consciousness are just irreducibly different from ours.

Is Soji sentient? She seems quite emotional. When I pick her up, she usually clicks her teeth together. This is called bruxing, and people agree that rodents make this sound when they’re happy. And normally, when I pick her up, she might also chatter, and I have no idea what that means. That might be happy too, because she makes this noise when I put her in her plastic ball, and she seems to enjoy running around the house, banging her plastic ball into things, so maybe that means she’s happy. But then she also makes that chattering sound when I put her back in her house. And she also tends to make it just when she’s sitting and there’s absolutely nothing going on, that’s actually when she’s the loudest. So maybe she’s announcing that this is her territory, wherever she happens to be. Maybe she’s horny. Maybe she’s expressing something that’s beyond human comprehension. I don’t know.

I’m in the middle of this fascinating book called The Edge of Sentience, which was published last year. It’s by the philosopher Jonathan Birch (who has nothing to do with the John Birch Society). It was published last year. It’s a free download from edgeofsentience.com or you can buy it. Birch is proposing an ethical framework for deciding which beings might be sentient or not, because if they are sentient, then we have an ethical duty not to cause them undue suffering. Birch starts out by talking about consciousness. He distinguishes some aspects of the mind that we might call consciousness. And he breaks it down into three meanings.

The first is selfhood, what we were talking about, this experience of being some sort of lasting single thing. And then there’s also sapience, which is intelligence: learning, problem solving, analyzing. And then finally, there is what he calls the “broad” sense of sentience, which is phenomenal consciousness, or “raw feels”, just bits of experience in time. And he says that if you combine this with valence, which is having experiences that feel good or feel bad, then these two things together, having consciousness and feeling good or feeling bad, this is the “narrow” sense of sentience, which is having valenced conscious experience. Any being, any system of neurons or software or whatever, that can have valenced conscious experiences, this is a sentient being, and we have a duty not to make it suffer.

So he’s trying to build this ethical framework for dealing with creatures where we’re not sure if they’re sentient or not. So he says,

A sentient being (in the sense relevant to the present framework) is a system with the capacity to have valenced experiences, such as experiences of pain and pleasure.

And if a being might be sentient, then he says, we have a duty to avoid causing gratuitous suffering to this being:

Framework Principle 1. A duty to avoid causing gratuitous suffering. We ought, at minimum, to avoid causing gratuitous suffering to sentient beings either intentionally or through recklessness/negligence. Suffering is not gratuitous if it occurs in the course of a defensible activity despite proportionate attempts to prevent it. Suffering is gratuitous if the activity is indefensible or the precautions taken fall short of what is proportionate.

So he’s not proposing to outlaw all meat or all animal research, just causing gratuitous suffering. This seems reasonable to me, and what I think is really interesting about Birch’s book, and what has been the most newsworthy aspect of it, is that he is trying to minimize ethical risk, moral risk, and this is the risk that we could be causing suffering to a sentient being without knowing it.

In a way we’re already familiar with moral risk. If I drop a thumbtack on the ground in a public place where people might walk barefoot, I clearly have a moral duty to pick it up, even though I might never know that somebody steps on it after I leave. They might not be permanently damaged, but it would be painful, and I have a duty to avoid the risk of causing suffering to that sentient being in the future.

Birch is talking about a different sort of moral risk. Think about dropping a live lobster into boiling water. There’s no question about what’s going to happen to the lobster, but there is a question about whether it’s a sentient being who feels pain, experiences it, in some way that we are morally obliged not to cause. If so, boiling it alive is immoral. So there’s some doubt whether it’s a moral or not, and it hinges on the lobster’s sentience. We may never know for sure, but it’s easy enough to avoid boiling lobsters alive. So in Birch’s opinion, we should do that to avoid the risk of causing suffering without knowing. Birch cause calls this the “precautionary principle”, and I think it’s a really interesting way to think about ethics.

So Birch divides all systems into three categories, and system is a deliberately vague and maybe strange word for a being, but it encompasses animals, plants, maybe it encompasses ant colonies as a whole, and it encompasses artificial intelligence, if such a thing already exists. So bring in large language models like GPT, just because it’s such a weird outlier, so that it helps us think about the whole range of possibilities. Here are Birch’s categories:

Number one, sentience candidate. This is the highest level. We’re not sure it’s sentient, but we’re pretty sure, and sentience candidates are anything where there’s evidence that these beings could experience pain and pleasure. We have some idea of what might cause pain and pleasure to them. So obviously, we can guess that all mammals feel pain, they react the same way to the same painful stimuli, and they have the same brain structures for experiencing the pain as we do, so mammals almost certainly sentient. Birch thinks that all adult vertebrates—mammals, fish, reptiles, amphibians—are also sentience candidates, but not necessarily vertebrate embryos. So, fish eggs probably are not sentient. Probably don’t have to worry about causing suffering to them.

When it comes to invertebrates like insects, insects don’t seem to feel pain if their exoskeleton is damaged, interestingly, maybe because there’s no point evolving that, since their exoskeletons can’t heal. But it’s possible that they do feel pain if they’re poisoned or heated. Birch analyzes their brain structures and behavior and says that insects are sentience candidates too, and he says that we should take proportionate steps to avoid gratuitous suffering to them. We don’t need to be as careful about causing pain to bugs as to hamsters, but we should take proportionate steps.

The second category for Birch is an investigation priority, and Birch puts some snails and worms here. Interestingly, he thinks spiders are less likely to be sentient than bugs, and there’s a whole chapter of brain structure analysis to explain why. Since the risk that these are sentient beings is lower, we don’t have to be as cautious about causing them suffering, but we should hurry to figure out whether they should be upgraded or not. So these are priorities for us to investigate further.

Finally, the third category is not sentient. Birch doesn’t think that plants, single celled organisms like bacteria, rocks, oceans, the earth, are sentient. These beings or things, can have very complex behaviors, but they don’t have neurons or anything like them, and so Birch doesn’t think that they experience anything.

What about a large language model like GPT? We don’t know what it might find pleasurable or painful. It’s possible that asking it for a recipe is the equivalent of giving it an orgasm, and that asking it to write a poem is the equivalent of boiling it alive. But there doesn’t seem to be any way to know. There’s no behavior it evinces that resembles pleasure or pain, and it doesn’t have any structures that are anything like those with which animals experience pleasure and pain. So GPT is, for the moment, considered not sentient. My personal opinion here is that large language models are going to top out at high intelligence, but not sentience. They can think, but they can’t feel. Artificial intelligence will eventually be sentient, but it will be some other architecture, some other method of simulating intelligence, that will also feel. Birch thinks that we need to be very careful not to accidentally create a sentient artificial intelligence that suffers without our knowing it, because that would be a moral risk.

There’s this very interesting example called OpenWorm. It’s a research project to simulate in software every neuron of the tiny worm called C. elegans. So C. elegans, the real worm, has exactly 302 neurons. Every adult C. elegans has the same neurons, connected in the same way. Researchers have just mapped them out and simulated them in software. And this project is called OpenWorm, and it has many contributors.

Recently, a researcher connected up the worm neuron simulator to a physical robot made out of Legos, and it sort of maybe a little bit behaves like a worm; it navigates the space using some of the same methods that a worm would use.

So this is an interesting example, because Birch thinks that a C. elegans, a real one, is an investigation priority. What about a simulation of a C. elegans worm? I think that’s an investigation priority too. And if the worm were upgraded to a sentience candidate, something that we really have to avoid causing suffering to, what about the simulation of it? I see no difference. I don’t think that the hardware matters at all. I think it’s just software. I think that a simulation of a sentience being is sentient, and we must avoid making such a thing suffer.

Okay, so this is a dharma talk. What are Buddhists supposed to think about all of this? I’m reading this excellent book, An Introduction to Buddhist Ethics by Peter Harvey, published in 2000. It’s got a lot of early Buddhist teachings, from the Pali canon, which I didn’t know until I read it, because I’ve only practiced Zen. In Zen, when someone asks a question, the teacher answers with riddles and quotes ancient Chinese poetry and says, you know, “The butterfly carries the precious jewel over the mountain.” So whenever I encounter early Buddhism and Shakyamuni Buddha, I really enjoy them. The old man was an intellectual. He thought everything through, and when somebody asked him a question, he would typically answer it in detail and tell you what he thought. And it’s clear that Buddha was very concerned about nonhuman animals. Indian religions in general are very concerned with animal welfare, in a way that might be unique among world religion families. Maybe because they say that humans are reincarnated as animals and vice versa. Or maybe the causation goes the other way: maybe they respect animals so much that they see no reason why humans couldn’t be reincarnated as animals.

Both Hinduism and Buddhism teach that each of us has been, in some past life, a worm, a ghost, a God, a hamster, and will be again. Any being we ever meet at any time has at some point been our parent or our child in some past life that we both had, and we will have this relationship again in the future. According to the Pali canon, Buddha said,

This samsara is without discoverable beginning. A first point is not discerned of beings roaming and wandering on, hindered by ignorance and fettered by craving.
Whenever you see anyone in misfortune, in misery, you can conclude: “We too have experienced the same thing in this long course.”

I love Buddha’s sort of scientific worldview—he doesn’t know. He says samsara’s without discoverable beginning.

Or more concisely, in another sutra he says,

The bones you’ve left behind in transmigration are greater than a mountain.

All these bodies you’ve died as, are greater than a mountain. But you don’t have to believe in reincarnation in this way, to see that we all experience misery. We all yearn to be liberated from suffering, and so it’s natural to feel sympathy for anybody who’s suffering.

Early Buddhist ethics teaches that harming a big, complex animal is worse than harming a small or simple one, and this is similar to Jonathan Birch’s framework. But on the other hand, Buddha didn’t seem to spend actually all that much time distinguishing between sentient and non sentient beings, like asking, is this or is this not? In Pali, he just talked about sattas, which is equivalent to the Sanskrit sattvas. In general, when we vow, as part of the Four Bodhisattva Vows, “to save all sentient beings”, in English, this, I think, comes from the Avatamsaka Sutra, the Bodhisattva vows of Samantabadhra. Or at least that was maybe an influence on it, via Chinese and so on. And Samantabadhra just says “saving sattvas” in Sanskrit, “saving beings”, maybe with the implication that sattvas are sentient. Certainly, if they’re worth saving, they must be. But it’s not a division between sentient and insentient that Buddhism really emphasizes.

Buddha, his goal was to free beings from suffering, by teaching them about the causes of suffering and the methods of being liberated from suffering. As we learn about suffering and we see that we’re trapped in it, and so is everybody else, it’s natural for us to be sympathetic for each other’s misery. Everybody similarly hates pain, yearns for peace.

Buddha taught that we shouldn’t cause suffering for any sentient beings, nor kill them. In the Dhammapada, which is the oldest Buddhist scripture, and maybe the most plausibly related to what the old guy actually said, he’s quoted as saying,

All tremble at punishment,
Life is dear to all.
Comparing others with oneself,
One should neither kill nor cause others to kill.

The early sutras say that we should help other beings and not cause them suffering, because that’s essentially good, but also acting this way is good for me. It accumulates good karma for me, and that good karma kind of pushes me along, gives me momentum to achieve escape velocity from suffering and achieve nirvana. But doing good because it’s going to be good for me, is only sort of good. Doing good for goodness’ sake is best, and your intention when you act is really key. Buddha said that if you harm unintentionally, it won’t cause you to suffer in the future. And if you cause good by accident, it won’t help you in the future either. This is in contrast to Jainism, another Indian religion that was arising around the same time in India. In Jainism, unintentionally harming a bug is quite bad, not as bad as intentionally, but still something that you work very hard to avoid. Even today, you’ll see Jain monks often wearing a mask to avoid accidentally inhaling a bug, sweeping the ground to avoid accidentally stepping on one. They even are so concerned about killing plants that they won’t eat root vegetables, like digging up a carrot kills the carrot, whereas plucking a fruit leaves the tree intact, and so they don’t eat root vegetables. Buddhists are a lot more relaxed about this stuff. In Theravada, they’re vegetarian, and they certainly try to avoid killing bugs, but they don’t go quite so far, because as long as your intentions are good, that’s probably good enough.

But Buddha said to think carefully about the consequences of your actions. So if you’re just like a well meaning idiot, obliviously causing carnage everywhere, that’s not the path to nirvana either. So I think Jonathan Birch’s framework is largely generally compatible with Buddha’s teaching here too. If we suspect that insects experience pain, for example, we shouldn’t ignore the possibility of that inconvenient truth because it would, it would be inconvenient for us. We should be curious. We should investigate whether they’re sentient. We should be willing to change our behavior if it turns out that they might be.

A difference between Jonathan birch and Shakyamuni Buddha is that Birch is not very concerned about killing animals. He’s worried about how farm animals, for example, would experience being raised and slaughtered, but he’s not against eating meat if it could be raised and slaughtered ethically. If it can’t be ethically farmed, he’s against it. For example, he thinks that octopuses cannot be raised ethically as livestock. So he’s for a ban of that. But he’s not against eating wild caught octopus, I think, if that could be done without pain. And honestly, this is a relief to me, because I do eat meat, I do kill mosquitos. Not causing undue suffering when I do that is something that I can aspire to. This is obviously incompatible with the First Precept. I took jukai in 2006 and I promised not to kill beings. I recognize the contradiction here, and I don’t have an answer to you for that. But avoiding causing suffering to sentient beings is something that I could aspire to.

A couple of years ago, due to Brexit, the United Kingdom was revising its animal welfare laws. It had withdrawn from the EU. And the EU had a treaty that said that animals are sentient beings, and member states must pay full regard to the welfare of animals. But the EU didn’t specify which animals are sentient or not. So I mean, technically, a brine shrimp is sentient. Do we have to worry about whether it experiences suffering or not? Unclear. When the UK replaced the European treaty with its own law, the first draft specified all invertebrates, so mammals, fish, reptiles, amphibians. Some activists argued for the inclusion of two specific groups of invertebrates. These are cephalopods like octopuses, squids and cuttlefish, or decapods, like lobsters and crabs. So in comes our hero, Jonathan Birch. The UK hires him to lead a commission to study the possibility that cephalopods and decapods are sentient beings. And I don’t know how long it took them, years? They wrote a hundred-page document with 300 citations to research studies. And the upshot is that the big, smart cephalopods, like octopuses, are likely enough to be sentient that British legislature has to keep their welfare in mind, and the same for lobsters and crabs. Of course, the news reports are overblown. They say, UK declares lobsters are sentient beings, but it doesn’t mean that you’re going to be tried for murder if you eat a lobster in London. It just means that the UK legislature must consider their welfare when it passes future laws. That’s it.

But still, even though the impact of the sentience act of 2022 might be small, it’s an example of the widening moral circle, widening it to include more beings. The philosopher Peter Singer wrote about this. He said that the Expanding Circle is an example of moral progress, of society’s moral beliefs improving over time. He said,

The circle of altruism has broadened from the family and tribe to the nation and race, and we are beginning to recognize that our obligations extend to all human beings. The only justifiable stopping place for the expansion of altruism is the point at which all whose welfare can be affected by our actions are included within the circle of altruism.

Progress. Progress is a very modern concept, maybe the modern concept. Only once our technology and social change was happening rapidly enough that you could actually observe that change in one lifetime, did we start to think that progress, continuous improvement, was a trend that we could observe and expect to continue. And moral progress is this astonishingly modern idea, that there is some better morality. We don’t know yet what it is. We can only guess. There might be principles that are better, but we don’t know what they are, but we or our descendants in the future, will believe them and behave accordingly.

The idea makes me very uncomfortable, because it seems to indicate moral realism, or objective morality, that there is some moral truth that’s out there outside of our current beliefs, and that society can either progress toward it or move away from it. Otherwise how is it possible to define whether a change in morality is moral progress or just moral change? I’m pretty sure that there is no moral truth, that societies just choose their axioms and try to reason logically from them, and that every philosopher who does this either comes up with axioms that logically lead to repugnant conclusions that offend our instincts, or that they lead to logical contradictions, generally both, and then we just try again and again, and it’s doomed.

Buddha taught, I think, that there is moral truth. That behaving in accord with the truth that we’re all interdependent, and in accord with the truth that all sentient beings want to be liberated from suffering, behaving in this way is essentially good. I mean, now that I say that, it sort of sounds reasonable. That as long as there are sentient beings in the universe, and it’s true that we’re not separate, that it makes sense for us to try to help each other all achieve what we want. And what we want naturally is not to suffer, kind of by the definition of suffering. So maybe, if there is a moral truth, he was right. Nagarjuna wrote that the precepts are empty, that they have no essential nature, they’re skillful means, and that they’re useful so long as they match our circumstances. That also sounds right to me. But then how do you define useful or not useful? Skillful or not skillful? This is all very ambiguous to me.

But if moral progress is real and it involves less suffering for sentient beings, then moral progress is happening now and and it’s happening extremely rapidly. Over 1000 companies have pledged not to use meat or eggs from caged chickens in the US. Forty percent of hens are now cage free, and that was only 6% a decade ago. Starting last year, McDonald’s, all the chicken meat and eggs that it uses come from cage free hens, and that’s two years sooner than McDonald’s had promised.

Male chicks are generally useless to agriculture, and so in the past, they were generally handed sexed and then ground up alive. Now it’s technologically possible to sex them in ovo and destroy them before they’re hatched. So in my opinion, this is less suffering. This is morally superior, and this affects billions of chicks every year.

The UK’s largest grocery chains will stop selling live lobsters and crabs in their stores, so you don’t see those miserable lobsters in a tank there anymore. France’s largest producer of trout will now stun them unconscious before killing them, before slaughtering them. Germany’s largest grocery stores have cut the price of fake plant-based meat so that it’s equal to the price of animal meat. And this price cut led to a 30% increase in the sales of fake meat.

Billions of sentient beings every year are affected by these changes, and there’s just a ton of momentum happening right now to accelerate moral progress when it comes to animal welfare in agriculture. So as bodhisattvas, we should rejoice that all of these beings are suffering less. One of the vows of the Bodhisattva Samantabadhra in the Avatamsaka Sutra, was to “rejoice in the merit and virtues of others.” So let’s rejoice that people are caring about animal welfare and doing this research and making these promises and making these changes, instead of feeling guilty that we’re not doing enough, or discouraged because there’s still so much suffering in the world. A bodhisattva’s attitude is joy that we’re all moving in the right direction.

Of course, we can also find ways to do our individual part and to keep pushing in the right direction. We can be selective in what animal products we buy some of the grocery store. Labels really make a difference. Free range fowl, chicken etcetera, meat and eggs really is a meaningful label. It’s regulated by the USDA. Other animals, not so much, but it makes a difference for chickens. And then there are non government organizations that have more rigorous standards, and you can look for their certifications on animal products at the grocery store too. Obviously, eating less meat is better. The Farm Animal Welfare Program at Open Philanthropy is an effective activist and research group.

Mostly, I just want you to know that even though there’s a lot of grim news and a lot of ways in which humanity is going in the wrong direction, This is something where we’re seeing a lot of progress right now, and it’s something to keep an eye on and to rejoice in.

Image sources:

Winter 2024 Meditation Retreat

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 13 Jan 2025 13:11:35 +0000

December 2024. The Village Zendo’s winter meditation retreat in Garrison, NY, led by head student Ryugin.

Images © A. Jesse Jiryu Davis

Is an Enlightened Person Afraid of Illness and Death?

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 21 Dec 2024 18:53:54 +0000

This is a dharma talk I gave at the Village Zendo December 12, 2024, about my mother’s brain surgery and end-of-life decisions. Here’s the video and a transcript.

Two months ago, my mom had emergency brain surgery. She had an artery in her brain that was basically completely blocked. This is the artery that feeds Broca’s area and the temporal lobes. These are the language centers, and language is my mother’s whole life. She’s a scholar of philosophy and law and religion. Reading and writing are her whole career. So this was especially scary for her. She had surgery on a Monday.

Keishin and I went into Mount Sinai and it was supposed to be, I don’t know, six to eight hours. It took 10 hours. We were just waiting and waiting and waiting in the big waiting area that they have there. Finally, we found out that she was done and in the next building over. We went and sat with her until she woke up. When she woke up, she couldn’t speak at all, which nobody had warned us about. When she had gone into surgery, she was fine. She’d had a few episodes of difficulty speaking, but she was totally fine. And then after the surgery, she woke up, and as she was coming out of the anesthesia, she looked over at me. We made eye contact. It seemed like she recognized me. She smiled at me, and then she said “dub dub dub dub dah,” as if she expected me to understand.

And then maybe she seemed scared. There were some tears coming out of her eyes, and she was—she was in the bed spasming. Grabbing the edges of the bed and shaking it, and her legs were kicking. I had no idea what was going on with her, and I was so scared. I said, “Can you point to some part of you that’s not comfortable,” and she couldn’t seem to understand what I was saying. She would look at me and she would nod, but then she wouldn’t comprehend. Or she couldn’t express anything.

She had this huge bandage around her head, and she kept reaching up to scratch behind the bandage, and pull at the bandage. And I kept grabbing her hand and putting it down and saying, “It’s okay, just be patient. Leave that bandage alone. I don’t know why you can’t talk. I think it’s probably just the anesthesia. You’ll probably be able to talk soon. Just relax. Just be patient. Just keep waiting, Mom. I think you’re going to be okay, leave the bandage alone.” We did this for hours. At some point, she finally evaded me and pulled her bandage off, and it was heavy with blood. It was just soaked with blood, and her pillow was soaked with blood.

Whenever this experience got too much for me, I would still be sitting there holding her hand, but I would close my eyes, breathe into my stomach and just be there. I wasn’t reassuring myself because I didn’t know anything. I was just breathing. I was aware of Keishin sitting in the room across from me and watching me, watching my mother. Knowing that was reassuring.

Every once in a while, a doctor would come in and get in Mom’s face and loudly ask, “Do you know where you are?” She’d say “buh.” The doctor would say, “What year is it? Is it 2022? Is it 2024?” She would just look at them. One guy strolled out of the room and whispered to a nurse, “Word salad.”

I asked, “So it’s normal? How long is this going to last? She didn’t have aphasia before the surgery.” And the doctors said either it’s normal, or we don’t know, or wait until you can see some other doctor in the morning.

And finally, it was nine o’clock and we had to go home.

Over the next few days, I did find out, okay, maybe this is at the bad end of a normal spectrum. It doesn’t mean that she’ll never speak again. We still don’t know.

She’d had surgery on a Monday, and Thursday was really one of the worst days of my life. I got a call around nine-thirty in the morning from the attending physician, saying, “Your mother has taken a turn for the worse. She may have had another stroke.” So jumped on my bike and I rode over. I was staying in New York. I rode to the hospital again. Jennifer was upstate. I got her on the phone, and we talked to the doctor.

What happened? “Not sure, but what little language ability and responsiveness she had is gone.”

Will she get better? “Don’t know.”

Before my mother’s surgery, she had said to me, “Look, if something goes wrong and I can no longer write anything worth publishing—two tickets to Switzerland.” Meaning we were going to go to Switzerland together for assisted suicide. This was on my mind the whole time. When I saw her, I would say, thinking she might understand: “Mom, I think you’re going to get better, but if not, I remember your wishes. You can trust me. I remember what I promised.”

Thursday morning was the time to put this into effect.

I said, is she going to get better? “We don’t know.”

Okay. Is she on life-supporting treatment? “Not right now.”

Is there a way that we can end her life? “We can stop feeding her.”

Okay, let’s give it a few more days.

And then she started talking, that Sunday morning, seven days after her surgery. I was home in the city, and while I was in the shower, mom called. When I got out, I picked up her voicemail, and she had said, very carefully, “Hi, Jesse, it’s Dena. Could you bring some coffee when you come?”

That was the first full sentence I’d heard from her in almost a week.

So she spent five weeks in the hospital, total. She finally came home. She’s been home for, I guess, about two months now, little less. And continues to improve. She’s walking with a cane, she’s speaking, she’s reading, she’s writing. Definitely still affected by the surgery, but also definitely not the worst that we had feared.

I’m telling you this story, because this is about Zen training. This is about, why are we meditating? How can we use this training? How can we get it in our bodies such that when the shit hits the fan, we can put it into practice without thinking about it?

Sometimes the old Zen texts seem to think that there’s no point to meditation. They seem to make fun of people who think that there is a purpose to meditation. Every once in a while, there will be a story about some monk who thinks that meditation makes your mind like a perfect mirror that reflects reality with no judgment, with no opinion. And this seems great. It both seems like a very effective state of mind to be in for responding to events, and it seems very peaceful. You don’t see a mirror freak out about what it reflects. It just reflects.

Sounds lovely, right? But then you can tell really the Zen attitude is making fun of this desire to become a mirror. So there’s the contest—no actually, first I’m going to talk about this koan from the Eighth Century.

Nangaku went to his student Baso’s hut and asked, “What are you doing these days?”

Baso the student says, “These days I just sit.”

Nangaku the teacher: “What is the purpose of sitting in zazen?”

“The purpose of sitting in zazen is to become a Buddha.”

Nangaku the teacher, he gets a terracotta tile and rubs it on a rock.

“What are you doing, master?”

Nangaku: “Polishing a tile.”

“What is the purpose of polishing a tile?”

“I’m polishing it into a mirror.”

“How can polishing a tile make it into a mirror?”

“How can sitting in zazen turn you into a Buddha?”

And there’s an earlier story about a poetry contest between two students of Hongren. They’re competing to become his Dharma successor. There is the learned aristocratic student Shenxiu, and there’s the working class stiff Huineng.

So the learned aristocrat, he writes:

The body is the tree of enlightenment,
The mind is like a bright mirror’s stand;
Time after time polish it diligently,
So that no dust can collect.

The other student, the working class stiff Huineng writes,

Enlightenment is not a tree,
The bright mirror has no stand;
Originally there is not one thing—
What place could there be for dust?

And he, of course, is considered the winner and our ancestor, right? The one who says there’s no mirror to polish, there’s no dust to remove.

So what’s the point of meditation? Become a flawless mirror that perfectly reflects reality with no smudgy delusion? Or is that idea actually stupid and there is no point to meditation whatsoever. And if so, what were we doing for that last half an hour? And why?

And even taking another step back from this, some texts really seem to criticize asking questions like this, like, “What is the purpose?”

There’s the koan where Jizo asks Hogan, “What’s the point of going on a pilgrimage?” Hogan says, “I don’t know.” Jizo says, “Not knowing is most intimate.”

So should we be asking this at all? Should we be trying to know what is the purpose of meditation, or not?

Now, look, I’m Jewish. I can’t help asking questions like this, analyzing texts, getting myself involved in the arguments. There’s a two-thousand-year-old Jewish joke about this: Somebody who is considering converting to Judaism goes to the Rabbi Shammai and asks: “Explain to me the whole Torah while I stand on one foot.” Shammai is offended and chases the person out.

The same person goes to Rabbi Hillel and asks the same question: “Explain all of Torah to me while I stand on one foot.” Hillel says, “What you would hate if it were done to you, don’t do it to other people. The rest is commentary. Go study it.”

It’s a famous story because it gets both sides: Wisdom is very simple, and it’s very complex. And the complexity is worth studying.

So let’s do this. Let’s boldly analyze what is the point of meditation.

Here’s one answer: We are likely to experience old age and sickness, and we are definitely going to die. And most of us experience some anxiety about this. So maybe the purpose of meditation is to resolve this anxiety. In the old stories about Buddha, yes, for him, resolving this anxiety was the thing that motivated him to meditate. After studying meditation for six or seven years, he had learned yoga and magical powers, and he had almost starved himself to death, he sat under a tree and made the vow that he would not get up again until he had resolved this anxiety about sickness and old age and death.

And his insight after eight days of sitting was that suffering is caused by thirst, by trishna, by the desire for things to be different from how they are. For less of what we don’t want. For more of what we do want. And that is on a foundation of delusion, that the self is this separate and lasting thing that I can guard the borders of. That I can defend from the things that I don’t want. The borders behind which I can accumulate more of the things that I do want. And by seeing that this is all an illusion, we can resolve this anxiety about the end of this.

The monk Dogen had a similar story. He’s the founder of our lineage in Japan. He was born around 1200. When he was two, his father died, and when he was seven, his mother died, leaving him an orphan. At his mother’s funeral, he watched the incense rise up into the air and disappear. And he was devastated. Does this happen to everybody? Does this happen to everything? Will this happen to me? How do I understand this? How do I live with this knowledge? So that was what initially drove him to the monastery.

Now, in the 1700 years between the lifetime of Buddha and of Dogen, a lot had been written down. Dogen read a lot of this stuff—he was a big scholar—and he read all of these paradoxical texts like, “There’s no mirror for dust to settle on.” Or, “Meditating to become like a Buddha is like polishing a tile to make it a mirror. It’s hopeless.” And he thought, okay, there’s something they’re not telling me: Why are we doing this?

He traveled from Japan to China when he was 23, and traveled around China for four years. He found a teacher, Rujing, who seemed to him to have the real-deal original Chinese Buddhism. He studied with Rujing, and had a great insight. And his insight was: practice and enlightenment are the same thing. He wrote:

To suppose that practice and realization are not one is a view of those outside the way; in Buddha Dharma they are one and the same. Because practice within realization occurs at the moment of practice, the practice of beginner’s mind is itself the entire original realization.

When I first heard this slogan, I found it completely uninspiring. “Practice is enlightenment” pissed me off.

At 22 I was miserable. I was depressed. I was smoking pot every day. I was failing at my job. I hated myself. I turned to meditation for help, something that would fix me. I wanted to get enlightened, because I heard that enlightenment was liberation from suffering, and I was suffering. So I checked myself into a monastery for a year. And we chanted Dogen. We read Dogen together. I kept getting this message, “practice is enlightenment.” And I thought, this is a scam. Practice isn’t enlightenment. It’s a grind.

So I ignored this slogan, that practice is enlightenment, for years.

But after about 15 years of meditation, it started to make a little bit of sense. This is just my view of the teaching, but it’s been helpful to me, maybe it’ll be helpful to you: I am no longer trying to become enlightened. I’m trying to be enlightened more of the time.

Like right now, I’m paying attention to you. I’m aware that I’m in this room. I am trying to speak the truth. I’m being enlightened. Then in a little bit I’ll get on the subway. I’ll shove somebody aside, I’ll get lost in thought, I’ll be deluded. And then I’ll come to, and then I’ll be enlightened again. And for sure, after spending so many years of practice, both sitting cross legged and staring at the floor and everyday practice, I am being enlightened more often than I was before.

Now, I’ve read stories about people who become enlightened, and it’s a permanent change in their life. There are thousands of these stories. And I can’t say that I know for sure that this is just a folk tale. But as I get older, as I have more experience—I become more skeptical of these stories. Most of them are hundreds of years old. We don’t know these people, we don’t have a lot of independent witnesses of their behavior. And it’s a little bit suspicious that of all of the modern people who have claimed to be enlightened, almost inevitably they turned out to be alcoholics who sexually abused their students. I think we need to think very carefully about what path we are on. If that is one of the terminal subway stations, we need to figure out which station we transfer at. And I think we also just need to question whether this mindset of achieving some lasting change, as opposed to practicing being enlightened now, what effects those two things have in the long run on who we are and how we treat other people.

So I’m just trying to be enlightened more, and that’s what “practice is enlightenment” means to me. When I’m practicing, I’m being enlightened.

It’s true that being enlightened means not suffering. When I was sitting with my mom, when she came out of surgery, or when I was talking to the doctor about maybe ending her life—those were some of the most challenging moments in my life. But I wasn’t suffering. I wasn’t wishing it was different. I wasn’t thinking, “poor me.” I wasn’t thirsting for something else, because I was in the moment. I was very aware that I was being called to respond and that I needed to bring my whole self to it. And therefore I wasn’t suffering at all, because I wasn’t thinking about, “Do I prefer this or not?” I was just responding.

So Mom’s recovering slowly, as I said. This Thanksgiving, she actually was able to come up to New Paltz and have Thanksgiving dinner with Keishin and me—which, you know, was dicey. I wasn’t sure, but it worked out fine.

After Thanksgiving dinner, I saw her standing at my kitchen counter, and she was picking up one foot and putting it down, picking up one foot and putting it down. She said, “I bet you can do this for a whole minute, right?”

I said, do what?

She said, “Stand on one foot for a minute.”

I said yep.

She said, “It’s a physical therapy thing. I can only do it for a few seconds.”

And I said, well, I better summarize the Torah real quick then.

She laughed! She got it immediately. She said, “Very clever. Very well done.”

And I was aware, and I am aware, that my mother is just about the only person I know who would get that joke, who I can make that joke to. I didn’t know if she was gone or not, and she’s back. I have her, for now. Not forever, but for now. And that’s what being enlightened means for me, is to appreciate that.

Images: Early Illustrations of the Nervous System by Camillo Golgi and Santiago Ramón y Cajal.

Peter's Kill in the Snow

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 05 Dec 2024 21:33:41 +0000

December 5, 2024. Peter’s Kill in the Gunks, during a snow shower. I ran out to get some photos of the snow falling in the morning light. While I was on the clifftop the storm returned and absolutely nuked me. It was impossible to keep my lens clean enough to take more photos, and the sunlight was blocked by clouds, so I came home.

Supper's Ready

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 24 Nov 2024 22:27:12 +0000

November 23 and 24, 2024. Gan attempting Supper’s Ready, a 5.12a climb in the Gunks, with her climbing partner Kenny.

P.S.: On November 25, she sent.

Predictive Scaling in MongoDB Atlas, an Experiment

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 31 Oct 2024 16:52:03 +0000

At MongoDB, we experimented to see if we could predict each DBaaS customer’s demand fluctuations, and auto-scale them using this foreknowledge. Senior Data Scientist Matthieu Humeau and I spoke about this experiment at Data Council and NYC Systems. Here’s the video from NYC Systems, and a written version is below.

Replica Sets

MongoDB is generally deployed as a group of servers, where one is the primary and at least two are secondaries. The client sends all writes to the primary, and the secondaries replicate those writes, usually within milliseconds. The client can read from the primary or the secondaries. So secondaries can take some of the query load, and they’re hot standbys. If the primary fails, a secondary automatically becomes primary within a few seconds, with zero data loss.

MongoDB replica set.

Atlas

MongoDB is free and open source, you can download it and deploy a replica set yourself, and lots of people do. But these days people mostly use our cloud service, MongoDB Atlas. Atlas started out as a database-as-a-service. Now we call it a Developer Data Platform because it offers a lot more than a database; we have triggers and events and streaming analysis and edge computing and vector search. But this experiment focuses on the DBaaS.

The DBaaS is multi-region—customers can spread their data around the world or locate it close to their users—and it’s multi-cloud, it runs on AWS, GCP, and Azure. A customer can even deploy a replica set that includes servers in multiple clouds at once.

MongoDB’s cloud is actually Amazon’s, Microsoft’s, or Google’s cloud.

MongoDB’s secret business model (not drawn to scale).

Atlas customers decide how many MongoDB servers to deploy in Atlas, what clouds to deploy them in, and what size of server: how many CPUs, how much RAM, and so on. Each server in a replica set must use the same tier. (With exceptions.) We charge customers according to their choices: how many servers, what size, and how many hours they’re running. Of course, most of the money we charge our customers, we then pay to the underlying cloud providers. Those providers charge us according to the number and size of servers and how long they’re running. If we could save money by anticipating each customer’s needs and perfectly scaling their server sizes up and down, according to their changing demands, that would save our customers money and reduce our carbon emissions.

Tier	Storage	RAM	vCPUs	Base Price
M10	10 GB	2 GB	2 vCPUs	$0.08/hr
M20	20 GB	4 GB	2 vCPUs	$0.20/hr
M30	40 GB	8 GB	2 vCPUs	$0.54/hr
M40	80 GB	16 GB	4 vCPUs	$1.04/hr
M50	160 GB	32 GB	8 vCPUs	$2.00/hr
M60	320 GB	64 GB	16 vCPUs	$3.95/hr
M80	750 GB	128 GB	32 vCPUs	$7.30/hr
M140	1000 GB	192 GB	48 vCPUs	$10.99/hr
M200	1500 GB	256 GB	64 vCPUs	$14.59/hr
M300	2000 GB	384 GB	96 vCPUs	$21.85/hr
M400	3000 GB	488 GB	64 vCPUs	$22.40/hr
M700	4000 GB	768 GB	96 vCPUs	$33.26/hr

MongoDB Atlas-on-AWS pricing

We sell MongoDB server sizes as a set of “tiers”, those are named like M10, M20, and so on, on the left. Those map to specific instance sizes in the cloud provider, so an M10 is a certain size of AWS server, and we charge a certain price on AWS. If the customer chooses to deploy their M10 on Azure or GCP then the size and price will be slightly different.

Manually-Triggered Scaling

A customer can change their server size with zero downtime. Here’s the process:

The customer clicks a button or executes an API call to resize their servers to a chosen tier.
Atlas chooses a secondary and takes it offline,
detaches its network storage,
restarts it with a different server size,
reattaches the storage,
and waits for it to catch up to the primary, by replaying all the writes it missed while it was down.
Atlas scales the other secondary likewise.
Atlas tells the primary to become a secondary and hand off its responsibilities to another server.
Atlas scales the former primary.

Gaston Rebuffat.

The whole process takes about 15 minutes, and the customer can read and write their data normally throughout. Usually the customer’s application doesn’t even notice the scaling operation, except that once scaling is complete, performance is faster or slower, and the price is different.

Atlas Autoscaling Today

Today’s state of the art.

Atlas customers can opt in to autoscaling, but today’s autoscaling is infrequent and reactive. The rules are:

scale up by one tier after 1 hour of overload,
scale down by one tier after 4 hours of underload.

Overload is defined as over 75% CPU or RAM utilization, and underload is less than 50% of either. (Details here.) Atlas only scales between adjacent tiers, e.g. if an M60 replica set is underloaded, Atlas will scale it down to M50, but not directly to any tier smaller than that. If the customer’s demand changes dramatically, it takes several scaling operations to reach the optimum server size. This means servers can be overloaded or underloaded for long periods! An underloaded server is a waste of money. An overloaded server is bad for performance, and if it’s really slammed it could interfere with the scaling operation itself. So Matthieu and I envisioned…

The Ideal Future

The prototype.

In the ideal future, we would forecast each replica set’s resource needs. We could scale a replica set up just before it’s overloaded, and scale it down as soon as it’s underloaded. We would scale it directly to the right server size, skipping intermediate tiers. We’d always use the cheapest size that could meet the customer’s demand.

Predictive Scaling Experiment

Matthieu and I performed an experiment over the last year to see if predictive scaling was possible in the MongoDB Atlas DBaaS.

The experiment was possible because Atlas keeps servers’ performance metrics in a data warehouse. We have a couple of years of data about all servers’ CPU and memory utilization, the numbers of queries per second, inserts per second, etc., all at one-minute intervals. Atlas has about 170,000 replica sets now, each with at least three servers, so it’s a stout data set. We chose 10,000 replica sets where customers had opted in to the existing reactive auto-scaler, and we analyzed their 2023 history. We split the history into a training period and a testing period, as usual with machine learning, and trained models to forecast the clusters’ demand and CPU utilization. (CPU is the simplest and most important metric; eventually we’ll forecast RAM, disk I/O, and so on.) Once we’d prototyped this predictive scaler, we estimated how it would’ve performed during the testing period, compared to the reactive scaler that was running at that time.

The prototype had three components:

Forecaster: tries to predict each cluster’s future workload.
Estimator: estimates CPU% for any workload, any instance size.
Planner: chooses cheapest instance that satisfies forecasted demand.

For each replica set, its history is an input to the Short-Term and Long-Term Forecasters. (I’ll explain why we have two Forecasters soon.) The Forecasters must be retrained every few minutes, as new samples arrive.

From the same data warehouse we sampled 25 million points in time from any replica set in Atlas. Each of these samples includes a count of operations per second (queries, inserts, updates, etc.), an instance size, and the CPU utilization at that moment. We used this to train the Estimator, which can predict the CPU utilization for any amount of customer demand and any instance size. This is a hard problem, since we can’t see our customers’ queries or their data, but we did our best. The Estimator must be retrained rarely, when there’s new hardware available, or a more efficient version of the MongoDB software. (Eventually we plan to train an Estimator for each MongoDB version.)

The Forecasters and Estimator cooperate to predict each replica set’s future CPU on any instance size available. E.g., they might predict that 20 minutes in the future, some replica set will use 90% CPU if it’s on M40 servers, and 60% CPU if it’s on more powerful M50 servers.

Predictive Scaling: Planner

Let’s look at the Planner in more detail. Here’s a forecasted workload, it’s forecasted to rise and then fall.

So the Planner’s plan is to use M40 servers until it would be overloaded, then switch to M50 during the peak, then switch back. Notice the replica set should start scaling up 15 minutes before the overload arrives, so the scale-up is complete in time to avoid overload. It starts scaling down as soon as the risk of overload has passed.

Predictive Scaling: Long-Term Forecaster

Our goal is to forecast a customer’s CPU utilization, but we can’t just train a model based on recent fluctuations of CPU, because that would create a circular dependency: if we predict a CPU spike and scale accordingly, we eliminate the spike, invalidating the forecast. Instead we forecast metrics unaffected by scaling, which we call “customer-driven metrics”, e.g. queries per second, number of client connections, and scanned-objects rate. We assume these are independent of instance size or scaling actions. (Sometimes this is false; a saturated server exerts backpressure on the customer’s queries. But customer-driven metrics are normally exogenous.)

Our forecasting model, MSTL (multi-seasonal trend decomposition using LOESS), extracts components from the time series for each customer-driven metric for an individual replica set. It separates long-term trends (e.g., this replica set’s query load is steadily growing) and “seasonal” components (daily and weekly) while isolating residuals. We handle these residuals with a simple autoregressive model from the ARIMA family.

MSTL (multi-seasonal trend decomposition using LOESS)

By combining these components, we forecast each metric separately, creating a “Long-Term Forecaster” for each. Despite the name, the Long-Term Forecaster doesn’t project far into the future; it’s trained on several weeks of data to capture patterns, then predicts a few hours ahead.

How often is demand seasonal?

Most Atlas replica sets have daily seasonality. About 25% have weekly seasonality. Generally if a replica set has weekly seasonality it also has daily seasonality. Hourly seasonality is rare, and anyway it isn’t helpful for planning a scaling operation that takes a quarter-hour. Replica sets with sufficient daily/weekly seasonality are predictable by the Long-Term Forecaster.

Example “long-term” forecast.

But only some replica sets have seasonality! For non-seasonal clusters, the Long-Term Forecaster’s prediction of customer-driven metrics is unusable.

	Seasonal Clusters	Non-seasonal Clusters
Connections	3%	50%
Query Rate	19%	71%
Scanned objects Rate	27%	186%
		↑ unusable

So we added a “self-censoring” mechanism to our prototype: the Long-Term Forecaster scores its own confidence based on its recent accuracy, and only trusts its prediction if its recent error has been small.

Predictive Scaling: Short-Term Forecaster

What can we do when the Long-Term Forecaster isn’t trustworthy? We didn’t want to fall back to purely-reactive scaling; we can still do better than that. We prototyped a “Short-Term Forecaster”: this model uses only the last hour or two of data and does trend interpolation. We compared this to a naïve baseline Forecaster, which assumes the future will look like the last observation, and found that trend interpolation beats the baseline 68% of the time (29% reduction in error).

Approximation of local trends for near future forecast.

Predictive Scaling: Estimator

The Forecasters predict customer demand, but we still need to know whether CPU utilization will be within the target range (50-75%). That’s the Estimator’s job. The Estimator takes the forecasted demand and an instance size (defined by CPU and memory), and outputs projected CPU. Using a regression model based on boosted decision trees trained on millions of samples, we’ve achieved fairly accurate results. For around 45% of clusters, our error rate is under 7%, allowing us to make precise scaling decisions. For another 42%, the model is somewhat less accurate but useful in extreme cases. We exclude the remaining 13% of clusters with higher error rates from predictive scaling.

Example of input and output of Estimator.

Predictive Scaling: Putting It All Together

With both forecasts and CPU estimates, the Planner can choose the cheapest instance size that we guess can handle the next 15 minutes of customer demand without exceeding 75% CPU. Our experiment showed that this predictive scaler, compared to the reactive scaler in use during the test period, would’ve stayed closer to the CPU target and reduced over- and under-utilization. For the average replica set it saved 9 cents an hour. That translates to millions of dollars a year if the predictive scaler were enabled for all Atlas users.

	Predictive auto-scaler	Reactive auto-scaler
Average distance from 75% CPU target	18.6%	32.1%
Average under-utilization	18.3%	28.3%
Average over-utilization	0.4%	3.8%

What’s next? Matthieu and other MongoDB people are improving the Estimator’s accuracy by adding more customer-driven metrics, and estimating more hardware metrics: not just CPU, but also memory, and perhaps co-modeling CPU and memory to capture interactions between them. We want to investigate the minority of customers with bad estimates and ensure the Estimator works for them too. We’ll try building specialized Estimators for each cloud provider and each MongoDB version. MongoDB can’t see our customers’ data or queries, but we can gather statistics on query shapes—maybe we could use this to improve estimation.

I can’t tell you a release date. Par for a public blog post, but still disappointing, I know. In this case we honestly need more experiments before we can plan a release. A private beta for a few customers will come soon. Before we can unleash a complex algorithm on our customers’ replica sets we need a lot more confidence in its accuracy, and a lot of safeguards. We’ll always need the reactive auto-scaler to handle unexpected changes in demand. But I’m excited at the prospect of eventually saving a ton of money and electricity with precise and proactive auto-scaling.

Predictive and reactive auto-scalers, cooperating.

Burial Vessel

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Fri, 25 Oct 2024 20:01:42 +0000

Elizabeth Gross weaving a vessel for green burial.

Fossymandias

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 21 Oct 2024 09:27:10 +0000

I met a programmer of an antique tongue
Who said: A vast, maintainerless pile of code
Lies in GitHub. All around it flung
Half-finished branches where development slowed,
And READMEs full of bombast, heralding features
Tell that its authors once with passion toiled
Without release, their hearts chained to their object,
Until their hearts gave out, their ardor spoiled:
And in the manual these words appear:
“My name is Fossymandias, the Greatest Project:
Look on my works, ye Mighty, and despair!”
No further boasts were heard from the engineers
Of that colossal wreck. Boundless and bare
The empty commit log stretches countless years.

Image: The Younger Memnon, © The Trustees of the British Museum. Shared under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) licence.

Climbing at Bonticou Crag

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 19 Oct 2024 22:24:45 +0000

October 19, 2024. Climbing undocumented routes in the Gunks.

Join Me At NYC Systems October 17

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 10 Oct 2024 17:36:40 +0000

New York’s hottest club is NYC Systems. I’m speaking there October 17 with my colleague Matthieu Humeau. We’ll describe our experiment with predictive auto-scaling in MongoDB Atlas. The other presenter that night will be Andrew Werner, talking about instrumentation of compiled code. Unfortunately for you, registrations filled up a half hour after they opened. Join the waitlist! Or watch our presentation on the same topic at the Data Council conference this spring.

NYC Systems is a tech talk series recently begun by Phil Eaton. Phil also created a giant online book club that reads books about software internals. He’s talented at organizing communities, I’ve been observing from the sidelines and I’m impressed.

Antlion Crag

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 22 Sep 2024 18:11:44 +0000

September 21, 2024. Chris on Refrigerator Crack 5.10a and Cloud Nine 5.9 at the new Ant Lion Crag in the Gunks.

Are You Prepared To Save A Life?

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 09 Sep 2024 22:55:07 +0000

You can find yourself in a critical situation without warning. Is Zen practice preparing you?

Here’s a dharma talk I gave to the Village Zendo on September 5, 2024, about the classic koan, “Nansen kills the cat.” The first couple seconds of the video were cut off, sorry. There’s a written version below.

At the monastery on Mount Nansen, the monks of the eastern and western halls were arguing about a cat. This was late morning sometime around the year 830, in southern China. It was a cold winter, the monks didn’t have much to do. They were gathered in the kitchen, the warmest place in the monastery, and their boredom and isolation had boiled over and they started some argument about this cat. The abbot, Master Nansen, heard them. He stormed in and grabbed the cat. He picked up a big cleaver from the kitchen counter and said, “If you can say a word of Zen, I won’t kill it.” All the monks froze. He looked around. There wasn’t a word. He pinned the cat to the counter and THUNK cut it in half. Its hind legs, disconnected, twitched. Its pink guts fell on the floor in a tangle, jiggling. The monks stared. One of them ran out the door and threw up in the snow.

What were these stupid monks arguing about? Who should feed the cat? Whether a cat belongs in a Zen temple?

In a commentary, Tenkei said they were arguing over whether the cat has Buddha-nature or not. If so, Master Nansen’s teaching cut through that argument. No Buddha-nature, no not-Buddha-nature. No cat. Or as Dogen said, Nansen didn’t cut through the cat, he cut through “Buddha”. I hope the stupid monks got the message. The poor cat didn’t decide to be an example in a philosophy lesson. It had sneaked into the monastery one night, following the smell of the mice who lived in its walls in the winter, eating the barley from the temple storehouse. The cat had been skittish at first but it came to trust the monks. And Nansen murdered it.

Here’s a 15th-Century Japanese painting of Nansen and the cat. He has a nice sword here, but why does a monk have a sword? I think he used a kitchen knife.

Did Nansen actually do this? Or did he maybe pantomime it? I’ve read essays on this koan that say this ancestor we revere wouldn’t actually murder a cat to teach his students. Wansong wrote this 250 years after the incident:

Eminent Xin of the Liao Dynasty criticized Nansen’s group for killing a living being, committing wrongdoing. Head Monk Wen, on the other hand, helped Nansen out, saying, “An ancient text has it that he just made the gesture of cutting—how could he have simply cut it in two with one stroke, sending fresh blood gushing?” In these two critiques of the ancestor, Mr. Wen’s fault is the graver, whereas Mr. Xin’s fault is the lesser.

In other words, according to Wansong, go ahead and say it was wrong to the kill cat, but don’t say he didn’t really do it!

Gerry Shishin Wick agrees, he said, “Contrary to the opinions of some modern Zen teachers, that knife did cut the fur, the skin, the muscle, and the viscera.”

Why are these people insisting Nansen really killed the cat? This is about the stakes of our practice. We spend thousands of hours staring at the floor. Are we or aren’t we training ourselves to respond in life or death situations?

I’ve been rock climbing for a few years and I do it in a style called “trad” or “traditional” lead climbing, which is fairly dangerous. Trad climbing involves constant risk assessment and managing your fear.

(Note to readers: this section is best watched on video. I used props.) When I’m leading a climbing route, I climb up a more-or-less vertical cliff. I wear a harness and I tie the rope to it, which will catch me comfortably if I fall. My partner has the rope going through a belay device. The belayer can slowly give out slack, or when I fall my belayer uses the device to catch me. The device is attached to the belayer’s harness so their weight counters my weight and stops my fall.

But in the beginning, the belayer can’t do anything, because I’m climbing from next to the belayer to above them. The first 10 or 15 feet of a trad climb is always totally unprotected, the rope and the belayer are useless, you just must not fall. As soon as I can, I find a crack in the rock and I place some piece of protection there. Usually it’s a cam. I carry a dozen of these with me as I climb, in different sizes. They work like harpoons: you retract the lobes, stick the cam into the crack in the rock, and then it’s stuck. Later when the climb is finished, you remove the cam by retracting the lobes again and pull it out.

As I climb up the cliff, I place protection as I go, and clip the rope through it. So if I fall, the belayer locks their belay device, and I fall down past my last piece of protection and eventually the rope stops me. Hopefully, I stop before I hit a ledge or the ground and get hurt. As I climb above the last piece I placed, I always risk a fall at least twice that distance. If I climbed 5 feet above my last piece, I’ll fall 10 feet. Actually, it’s always much more than twice as far, because the rope is elastic and there’s always slack in the rope, and I might pull my belayer off the ground some distance before I stop.

The worst risk is when I climb through a section of rock where there’s no good cracks for placing protection, or the cracks are the wrong shape. I might place a cam into a funny-shaped crack and think it’s the right size cam and it will hold me, but it’s not and it pops out and I fall much farther. This has never happened to me, but it happens to climbers all the time, and it’s my main fear when I’m climbing.

Here’s a video of me lead climbing a route that was easy and chill for me. When the video starts I already have one cam in the rock. I make a move, find the next good crack, place another cam, and keep climbing:

View this post on Instagram

A post shared by A. Jesse Jiryu Davis (@jessejiryudavis)

Being afraid of falling and injury when I’m climbing is a fascinating state of mind! In regular life I’m not physically afraid, hardly at all—driving a car or biking in the city are the most dangerous things I do besides climbing, but I’m so used to those I’m not afraid. Besides, climbing is special: we like to say it’s “the slowest dangerous sport.” You can be clinging to the wall in some precarious place, trying to decide what to do next, feeling afraid, for quite a while. So you can watch your scared mind at work, see what it does, and learn to manage it. When I’m afraid, it’s hard to focus on problem-solving. It’s hard to focus on how to minimize risk and complete this climb. I get distracted by worrying about falling, wishing I hadn’t started the climb, feeling hopeless.

That’s the Zen of climbing for me: observing how fear works, and learning how to redirect my attention to the actual climbing. There’s a handful of techniques: focus on the breathing, breathe loud and slow. Focus on the senses, the feel of my fingertips and toes on the rock. Look around, look down, acknowledge the height I’m at. “Don’t look down” is terrible advice. There are mantras, like “I’m safe right now,” or my favorite, “Let the body climb.” These days all the climbing coaches are saying you should meditate! And of course I bring my 20 years of attentional training to climbing, but I also use tools that are specific to climbing, and specific to fear.

Last month I was trying one of the most challenging trad lead climbs I’d ever done, and my mentor Dustin was belaying me and coaching me from the ground. Two things happened that rattled me. First, when I was starting the climb, before I’d placed any protection, when I was just free-soloing the first 10 feet of the climb, my hand slipped. I didn’t fall, but it totally rattled me, and I climbed down to the ground and breathed for a couple minutes. I climbed up again, past that point, placed a few pieces of protection, and got past an overhanging roof. I was sure that was the crux, the hardest part of the climb, so I stood on a little foothold for a while, shaking out my forearm muscles, catching my breath, calming myself down. I thought I could just cruise up the rest of the climb and I was feeling good.

Then came something that really threw me: I started to climb up again, I placed two more pieces, and arrived at the true crux. It turned out the hardest part was higher up. I could see what I needed to do, I needed to step on a tiny foothold, then lunge leftward and catch a big handhold. It was a fairly easy move, but committing: I was practically jumping for the hold. If I didn’t catch the handhold I would definitely fall. And because I was now 5 feet above my last cam, I’d fall 10 feet minimum, more likely 20 feet.

I felt like I had two minds. The thinker said, it’s a longish fall but safe, my last cam is totally solidly placed in the crack, I want to try this move. C’mon, try it!! And my body said, “No.” At the point of no return, I froze.

I repeatedly climbed up to the crux, hesitated, and climbed back down to a ledge below my last cam. I couldn’t calm myself down and I started to give up. I asked Dustin to pull the rope tight so I could hang from my cam. If he had, I would have ruined my chance to claim a clean ascent. Dustin said, “I can do that, but are you sure? Do you want to try one more time?” Just saying a word, and hearing him be calm and encouraging, gave me the oomph I needed. I took a couple deep breaths and tried one more time. As I approached the crux and hesitated again, Dustin yelled up, “Increase the tempo!” I lunged for the next hold and latched it, and finished the climb.

Here’s a time lapse video Dustin made:

View this post on Instagram

A post shared by Dustin Portzline (@dustinportzline)

What I love about climbing is how it makes my mind visible. I knew what had rattled me: I slipped near the start, and I was surprised by the second crux. I knew what I had to do, and I knew whether I was hesitating or committing to do the move. In the rest of my life there’s so much … cloudiness. What must I do or say that I’m so afraid of, I don’t even let myself see it? How am I hesitating right now?

What about you, is there something you should do but you’re hesitating? Are you admitting to yourself that you must do this thing, or are you trying to ignore it?

Sometimes the crux comes at us visibly, like Nansen with a cleaver. It says, “Say a word!” Sometimes it’s easier to ignore. Either way, if we stand frozen and mute like Nansen’s monks, the cat dies. A mentor or a teacher is crucial in that moment. Because I trust Dustin’s wisdom, when he shouted up “increase the tempo!” it broke through my hesitation and I did the move with my whole being. Zen teachers in other areas of my life have given me the courage to unfreeze myself. What’s your crux right now? Who can help you unfreeze yourself?

Joshu

When Nansen killed the cat, his wisest student Joshu was away, traveling. He returned that afternoon. He shook the snow from his wool cape and hung it up at the monastery entrance. He kicked off his heavy shoes and put on his straw sandals, and went in to say hello to Nansen. Nansen told him what had happened that morning. Joshu took off his sandals, put them on his head, and walked out. Nansen called after him, “If you’d been there you could’ve saved the cat!”

What’s it mean, sandals on the head? It was 1200 years ago and no one’s sure anymore. Maybe sandals on the head symbolizes death: people were buried barefoot and maybe their sandals were placed above their heads in the grave? Or maybe people put their sandals on their heads at funerals, or maybe that interpretation is just a garbled version of the first interpretation? Either way, Nansen tells Joshu a story about death, and Joshu’s gesture is just an acknowledgment. “Yes. Death. Impermanence. Nothing lasts.”

We don’t have to know what Joshu’s gesture meant exactly. The point is, he acted. He didn’t hesitate, he responded. If he’d been in the kitchen that morning, when all the other monks froze, Joshu’s gesture would have been enough to save the cat.

Monastery in Kalimpong, West Bengal

The worst climbing accident I’ve seen was a couple months ago. I had started leading an easy climb at a popular area, there were a few other climbers on routes nearby. I was about 25 feet up, when I heard the obvious sound of a climber falling near me: the jangling of his gear and the thud of his body hitting the rock. Someone screamed, and someone else shouted, “Call 911!”

I was not out of danger myself, so I deliberately did not look to the side, I didn’t want to see the accident until I was safe. I looked down, I saw that I could safely climb the 25 feet down to the base, so I did that, pulling my protection out of the cracks as I went until I was standing next to my belayer on the ground.

Then I looked over, about 50 feet to my right there was a climber lying on a ledge, a few body-lengths above the ground. I learned later that he’d been leading a hard, overhanging climb, he fell, and two or three cams that he placed popped out of the rock so he fell almost all the way back to the ground. He tumbled, hit his head, his helmet broke off, he hit his head again, and came to a stop on this ledge where he was lying unconscious. By the time I looked there was already a climber who had reached him and was checking his pulse and trying to talk to him.

I coiled my rope and moved my pack out of the way and waited for rangers to arrive. I wanted to be helpful. I was curious, too. How badly was the climber hurt? How was the rescue going to go? What more could I learn about the accident?

Just minutes after the accident a few rangers showed up. One of them, a young slim guy, took charge. He was brisk and friendly, almost cheerful. He was enjoying moving fast and putting his skills to work, enjoying his mastery. He looked up at the climber, asked the other people at the cliff base: what’s his name? Morgan. What happened? He fell and hit his head. All these ropes hanging from the wall right now, what’s this attached to? What’s that attached to? This ranger, this Joshu, he figured out what he needed to do, what resources were available, and made a plan. By this time a dozen climbers had gathered at the base, so he gave us all jobs. After a couple minutes he had three belayers managing three ropes, he’d tied himself to one of the ropes and climbed up to join Morgan. Morgan was somewhat conscious now, and Joshu introduced himself. He didn’t say, “I’m so sorry this happened to you,” he didn’t say, “Oh my god look at all this blood.” Two other rangers arrived with a stretcher. Joshu and another climber lowered Morgan from the ledge down into the stretcher, with the three belayers all working in sync, with Joshu instructing them all.

Once Morgan was in the stretcher on the ground, we needed to carry him down to the road where the ranger’s pickup truck was waiting. Joshu told us not to just pick up the stretcher and walk down to the road with it, because the ground was rocky and slippery with leaves, we could stumble and drop Morgan. Joshu said, “We’re going to do the caterpillar thing.” As if everyone was familiar with this technique and everyone knows it’s called the caterpillar. We lined up in two rows and picked up the stretcher, then passed it to the next set of people downhill. Then the uphill people ran around behind the backs of the people holding the stretcher, so they became the new downhill people, and the uphill people passed the stretcher down to them. It was fun, like a bucket brigade or school game. Whenever I was holding the stretcher I looked down at Morgan’s face. He was bloody and it seemed like his nose was broken, and he was going in and out of consciousness, but he was alive and talking.

The caterpillar reached the road and we loaded Morgan into the ranger truck, and off he went. I think the truck brought him to a helicopter, which brought him to a hospital. I’ve heard he was still conscious when he reached the hospital, so I assume he’s ok.

I noticed about myself how unfazed I was. Sure, it’s the worst accident I’ve seen, but I wasn’t surprised about anything. I know that this is exactly the kind of accident trad climbers have. It was a reminder how important it is to place pieces of protection correctly, so they’ll hold me when I fall, but I already knew that. I wasn’t upset, I was just curious.

There’s a coldness to me that I’ve long disliked about myself. I’m not very sympathetic to strangers, I don’t usually cry at movies. But after this experience I’ve decided it’s ok—I still act helpfully. In fact, my coldness can be an asset, it seems to make me good in emergencies.

In contrast to Joshu and the other climbers on the scene, there was one young woman who had completely lost it. I think she and Morgan were both part of a group of climbers who’d come up from the city for the day. She was walking around the base of the cliff sobbing. She was crying so hard that two of her friends had to help her walk to a flat spot, out of the way, where she could sit down and cry while they held her. The whole time we were rescuing Morgan, 20 or 30 minutes, she was off to the side crying with her two friends.

I don’t know her, I don’t know why the accident shocked her in a way it didn’t shock me. Maybe she was very close to Morgan, maybe it was her first time climbing outside and she didn’t know how dangerous it is, wasn’t prepared for someone to get hurt. I don’t blame her. But she couldn’t save the cat.

I’m going to keep climbing for many years and I’ll see more accidents. I’m taking some classes soon, like Wilderness First Aid, so I can be more helpful when that happens. If someone I know gets hurt, I won’t be so calm, but I think I can hold it together. I’m more confident about that now.

When you read about Zen it’s easy to misunderstand the message. We can think, Zen practice has no goal. The Heart Sutra says, “no path, no wisdom, no gain”. Huineng said, “there’s no mirror to polish and no dust to fall on it.” Dogen says, “we practice because we’re already enlightened.” But practice has an effect. You know that—it has an effect on you right now, and over the decades of your practice it will transform you. It’s crucial to ask yourself, how are you training yourself? How are you changing?

When Nansen comes with his cleaver, you’ll only have a moment to respond. Will you freeze, or will you save the cat?

Review: Amazon MemoryDB: A Fast and Durable Memory-First Cloud Database

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 26 Aug 2024 08:25:42 +0000

Amazon MemoryDB: A Fast and Durable Memory-First Cloud Database, by six AWS engineers, in SIGMOD 2024. AWS hacked up Redis (I mean this respectfully) to produce a Redis-compatible database-as-a-service called MemoryDB, with better durability and consistency. Here’s a video of my presentation to the DistSys Reading Group, and a written review of the paper below.

The Problem

It’s me, hi, I’m the problem, it’s me.

Redis is popular because it’s fast and supports fairly powerful data structures, which makes some kinds of applications much easier to build. But Redis has basically no durability or consistency guarantees. So Amazon wants to sell a better Redis.

Digression: there’s been some license drama. Redis’s owner, Redis Labs, changed the license from open source to source-available. My company made a similar move and I think it’s justified. There’s a fork of open source Redis called Valkey now, and it has a new logo:

I have this thing where I get older but just never wiser.

Valkey sounds like a Norse warrior woman to me. I think their logo should look like this:

I’m a monster on the hill.

However, AWS marketing says MemoryDB is “OSS Redis-compatible”, and they don’t mention Valkey, I don’t know how this will play out long term. Will Amazon contribute to Valkey? Or will proprietary Redis, Valkey, and AWS’s version of Redis drift apart forever?

One day I’ll watch as you’re leaving.

Anyway. Amazon wants to sell a better Redis, with stronger durability and consistency. How are they going to do it?

The Solution

Guaranteeing durability and consistency in a distributed database is always complex. In this paper, the authors’ solution is to make a black box, call it “the transaction log”, put all the complexity inside, and close the box.

I should not be left to my own devices.

The authors don’t describe the transaction log internals; we don’t know how it provides the guarantees on which MemoryDB relies. Presumably that’ll be a future paper, or maybe we can just infer it’s running Raft or Paxos. This is frustrating for me because I’m here for the distributed systems, but this isn’t a distributed systems paper—it’s mostly a software engineering paper. This paper is about how Amazon decomposed Redis into two parts: 1) the transaction log, and 2) everything else. They replaced the log with something better, put the parts back together, and created MemoryDB.

Too big to hang out, slowly lurching toward your favorite city.

MemoryDB Architecture

Figure 1 from the paper.

Here’s the MemoryDB architecture. Clients send writes to the primary. In vanilla Redis, the primary streams operations asynchronously to secondaries (aka “replicas”). The MemoryDB primary saves writes to the transaction log synchronously. How did the authors make the change from async to sync without rewriting Redis internals? There’s a doohickey called a Tracker which knows when writes become durable, delays acks until then, and blocks reads on dirty data. This provides linearizability on the primary. My impression is it lives outside the Redis core code, and it intercepts requests and replies.

Secondaries receive entries from the transaction log only after they are multi-AZ-durable. A client can opt in to read from one replica and get sequential consistency, or multiple replicas and get eventual consistency. In both cases, MemoryDB secondaries have better consistency than Redis secondaries, since the former don’t observe writes that can be rolled back.

Off-Box Snapshotting

Figure 2 from the paper.

Snapshots are useful for initializing new followers or disaster recovery. Normal Redis does snapshotting by forking the main process. One child keeps processing transactions (taking advantage of copy-on-write) while the other child creates the snapshot from its read-only copy of the data. These two children are competing for RAM and CPU, though, so the machine has to be overprovisioned to make headroom for occasional snapshotting.

With MemoryDB, when Amazon wants to take a snapshot it does it “off-box”. They start up a new Redis machine, restore it from the last snapshot in S3, replay the subsequent transactions, save a new snapshot to S3, and terminate the Redis machine. The customer workload proceeds without interference.

Advantages of a Disaggregated Log

MemoryDB is much like Aurora: it keeps the open source execution layer at the top, ensuring compatibility and avoiding reimplementation, but replaces the transaction log at the bottom with a proprietary service that’s more scalable and durable. The authors claim their mysterious transaction log service guarantees 11 9s of durability. That’s more 9s than you can shake a stick at!

Tale old as time.

Once the log is separated from the execution layer, you can scale durability separately from availability. For example, a single-primary-only deployment is low-availability but high-durability. If the primary dies, you may wait a while for a new primary to be initialized from the last snapshot, but you won’t lose data.

Some other advantages seem Redis-specific. Since Redis is single-threaded, it’s crucial to offload as much work from the primary as possible. A vanilla Redis primary must work to fan-out log entries to its secondaries, but a MemoryDB primary writes each entry only to the transaction log, which handles fan-out. Furthermore, Redis elections provide few guarantees: a newly elected primary might be missing recent writes, and there can be multiple primaries. MemoryDB uses the log’s strong consistency as part of its election protocol to guarantee a single primary with all the previous primary’s writes.

Scaling

MemoryDB, like Redis, is both sharded and replicated. Amazon can scale MemoryDB nodes up or down, in or out, in three dimensions:

The number of replicas per shard.
Vertical scaling (using more or less powerful instances).
Horizontal scaling (adding or removing shards).

Number of replicas: More replicas permit read scaling (with weak consistency) and higher availability (there are more hot standbys). To add a replica, AWS restores one from the last S3 snapshot, then replays the transaction log.

Vertical scaling: AWS replaces each secondary with a new one using a different instance size, then hands over leadership and deletes the old primary. The paper says MemoryDB has “a collaborative leadership transfer, where the old instance actively hands over leadership, which minimizes downtime”. There are no details, but MongoDB and some other systems have similar ideas. I can imagine that the old primary ensures the new one is nearly caught-up in the transaction log before it starts the handover, then it stops accepting writes, relinquishes its lease, and tells the new primary to run for election immediately instead of waiting for a timeout.

Horizontal scaling: Redis supports sharding using something like consistent hashing. There are 16,384 slots. Each slot has a subset of the keys, assigned according to a hash of the key. Each shard is a replica set, and each shard owns a subset of the slots. Perhaps open source Redis can change the number of shards and reassign slots, the paper doesn’t cover this. In MemoryDB, slots are reassigned in the way you’d expect: the old owner streams data and data-changes to the new owner until they’re almost in sync, then the old owner stops accepting writes, streams the final updates to the new owner, and they both commit a change to the sharding metadata to transfer ownership.

Presumably the transaction log service can also scale, but the log service is a black box so we don’t know.

Upgrading Versions

This part of the paper was clever and novel to me. MemoryDB upgrades the software version of a replica set thus:

For any v1, v2: v2 can read v1’s log entries, but maybe not vice-versa.
Each log entry is marked with the version of the primary that created it.
Secondaries upgrade to v2 first.
If there’s a failover during upgrade and a v2 node becomes a primary, v1 nodes don’t replicate its entries.
Snapshots are built with v1 until all nodes are v2, so snapshots are legible if the upgrade is aborted.

This is simpler than MongoDB’s solution. It also doesn’t solve all the problems MongoDB solves, but still—I admire it.

Their Evaluation

For a read-only workload, MemoryDB is about as fast as Redis on small instances. There’s a step change when the instance size is upgraded to 2xl, then no more benefit. The authors say this is because of “Enhanced IO Multiplexing”. I think this is because Redis is single-threaded, but Enhanced IO Multiplexing lets some IO work be offloaded to other threads. Apparently that removes a bottleneck when they upgrade to 2xl, but then they hit some other bottleneck and bigger instances don’t help.

For a write-only workload Redis writes faster than MemoryDB, but there’s no durability guarantee. It’s interesting that, even though Redis is single-threaded, it can still get a performance boost from a bigger instance. Bigger instances have the same CPUs, just more of them, which I think is useless for Redis. But bigger instances also have more network bandwidth, maybe that helps.

My Evaluation

This is a solid industrial-track paper about pragmatic software engineering for a distributed DB. It’s frustrating for distributed algorithms buffs, but we’re not its audience. For us, I wish Amazon had published a paper about the log first, or included some log details in this paper. Nevertheless I like it. The upgrade protocol is wisely designed.

What's Next for Women in Tech?

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 24 Aug 2024 16:53:51 +0000

The Washington Post says “the movement to diversify Silicon Valley is crumbling”, and the story is particularly dire for gender-diversity nonprofits: Girls In Tech has dissolved, so has Women Who Code, and software companies have sharply cut their grants to others. But were they helping? The article links to this Department of Labor chart which shows women’s share of all STEM employment has grown from 25% in 2000 to only 26% in 2022. For some reason the Post didn’t cite a worse statistic on the same chart: women’s share of “computer occupations” has fallen from 30% to 24%.

I don’t know if organizations like Women Who Code fundamentally made a difference or not. I’ve partnered with a similar group called PyLadies on mentoring new conference speakers, and it seems like they make the tech scene more welcoming for women who are trying to break in. They also offer a space for experienced women to coordinate and support each other, perhaps this keeps mid-career women from leaving the industry. PyLadies is still here and stronger than ever.

Like most male programmers I was aware of the gender imbalance since I went to college, and awoke to the urgency of the problem when Tracy Chou, Jesse Jackson, and others demanded that the big software companies publish their diversity stats. Ellen Pao’s activism made an impression on me too. In 2016 some women engineers at MongoDB, where I work, called a big meeting to discuss improving diversity and inclusion there. As I recall, the meeting was stressful and exciting—it was a big-bang for diversity efforts at MongoDB.

I teamed up with a couple of colleagues in the engineering department and started shopping for diversity workshops. We tried an Ally Skills workshop (which I’d recommend to anyone) and researched some other classes, but our then-CTO Eliot Horowitz wasn’t convinced we should bring any of these trainings to the company. He demanded evidence that these things were effective. I was frustrated at the time, and grateful in retrospect. I think Eliot was right. That year I read Iris Bohnet’s book What Works: Gender Equality by Design, which was very critical of diversity trainings. They seem to be counterproductive as often as not. Bohnet reviewed the evidence and argued instead for redesigning corporate processes and structures to be more inclusive.

My activism within MongoDB declined after 2016. Executives and HR took over and there seemed to be less appetite for white male engineers like me to lead the charge. This might be for the better: the managers had the power and perhaps the expertise I lacked. A 2018 survey matches my experience. It found that individual tech employees’ participation in diversity and inclusion fell by half year-over-year. Meanwhile, MongoDB and other companies have changed for the better, but somehow those Department of Labor numbers haven’t improved.

What makes a difference for women in the software industry? My female colleagues tell me it’s most helpful to have an affinity group within the company where they can be candid with each other, and to have sympathetic managers who are open to hearing about problems with the company culture, and to have women in leadership.

And what can men do? I’m certain that individual male programmers can choose to make a difference by mentoring junior women engineers. I’ve believed this for a long time, Iris Bohnet said so in What Works, and the Harvard Business Review said so back in 2016. Mentorship boosts early-career women in tech, and it mitigates the loss of mid-career women. I asked one of my female colleagues yesterday what has been most helpful for her, and she said “individual mentorship” first, before I revealed that that’s what I think, too.

My analytics tell me that you, dear reader, are probably a male programmer. If you’re an experienced engineer, you should mentor women. If you’re worried about appearances, just don’t be creepy and you’ll be fine. Mentorship includes advice, a sympathetic ear, advocacy, and cheerleading. Here’s 10 ways to support women in tech, from the National Center for Women and Information Technology.

Coaching Will Be The Last Human Job

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 17 Aug 2024 14:46:40 +0000

Sam Altman and Arianna Huffington plan to make an AI health coach, which will address chronic illness by encouraging people to eat better, take their medications, and get more sleep and exercise.

Consider what it’s like to be a busy professional with diabetes. You might be struggling to manage your blood-sugar levels, often missing meals and exercise due to a hectic schedule. A personalized AI health coach, trained on your medical data and daily routines, could provide timely reminders to take your medication, suggest quick and healthy meal options, and encourage you to take short breaks for exercise.

A busy professional might benefit from an AI coach, but as Ben Krauss points out, it’s the working class who suffer most from chronic disease, and have the highest obstacles to a healthy lifestyle.

I want to criticize a different aspect of this pitch:

Yes, behavior change is hard. But through hyper-personalization, it’s also something that AI is uniquely positioned to solve.

Behavior change is something that a human coach is uniquely positioned to solve. No matter how smart, a mechanical coach’s impact is limited by my ability to ignore it. I’ve set myself all sorts of digital reminders and nudges and I routinely tell them to piss off. My employer nags me with automated messages about various policies, I delete them reflexively. So long as I know my nagger isn’t human, I feel no obligation to respect it.

I have at various times hired a personal trainer, a couples therapist, and a climbing mentor. In all cases, I granted my coach the authority to nudge me to do something. I had some long-term goal (fitness, an honest relationship, courageous climbing), but I was hampered by my short-term laziness or fear. I used my coach’s authority or charisma or whatever as extra oomph to overcome my reluctance. AI could one day match any coach’s expertise and personalization, but I doubt it will replace a human coach’s ability to exert peer pressure. Only a human can do that, because only a human is my peer. AI will either be my servant or, if it isn’t, then AI will no longer care about sharing recipes with me, it will be too busy converting me to paperclips.

Here’s a recent example of peer pressure. My climbing mentor Dustin Portzline was belaying and coaching me while I was scared on a climb. I repeatedly approached the crux, hesitated, and climbed back down to a ledge. I couldn’t calm myself and I started to give up. I asked Dustin to pull the rope tight so I could hang from it—if he had, I would have ruined my chance to claim a clean ascent. Dustin said, “I can do that, but are you sure? Do you want to try one more time?” His calmness and encouragement helped me collect myself. I respect him and I wanted to do my best for him. I tried once more. As I approached the crux and hesitated again, Dustin yelled up, “Increase the tempo!” …that’s what I remember. He says he yelled, “Move faster!” I was kind of blacked-out with fear, half-animal, but I stand by my recollection. Anyway, his instruction was the extra motivation I needed. I lunged for the next hold and latched it, and finished the climb.

Here’s Dustin’s version of events:

View this post on Instagram

A post shared by Dustin Portzline (@dustinportzline)

I experience from humans an “aura” which shines from no mechanical reproduction of humanity. Being face-to-face with another human has the most powerful effect on us, but even a text message from a human demands our consideration in the way an automated text doesn’t. If I respect someone, then I care what they think of me and I don’t want to disappoint them; I’ll try harder. My personal trainer made me stronger partly by prescribing the right exercises, and partly by watching me and yelling, “C’mon! One more!”

In fact the content of another person’s advice is less important than their attention. Therapists sometimes offer insight, but they mostly listen as their patients discover themselves. Factory workers are more productive when a researcher dims the lights, or makes them brighter—the brightness doesn’t matter, the workers are more motivated because they know a researcher is paying attention to them.

Of course, human coaches are very expensive, and Baumol’s cost disease makes them more so. If Altman and Huffington’s AI coach is cheaper than a human, then it’s worth a try, even though it won’t be as effective. But in the long term I expect Baumol’s cost disease to reverse: robots will do most of the work, and there will be a surplus of human labor looking for jobs. They will collect in the professions where the human aura is important or essential: therapists, stage actors, judges, nannies, masseuses, butlers, prostitutes, coaches.

There may be AI-native generations in the future that respect machines as much as they respect people. The special aura of humanity will be gone for them. A machine’s attention, although divided among millions of users, will be as satisfying to them as the undivided attention of a human. Confessing their shame to a machine will be just as great a relief. A robot yelling “one more!” will be just as motivating. An AI judge’s decision will carry the same weight. Good luck to them, I can’t imagine their lives. For a while, though, I think that coaching will be among the last essentially human jobs.

Photo: my eye.

Summer Meditation Retreat

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 15 Aug 2024 22:15:35 +0000

August 2024. The Village Zendo’s summer retreat. Walking meditation, 5am meditation by candlelight, and Zen student Seiryu’s first dharma talk.

Lights Out Climbing

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Fri, 28 Jun 2024 07:55:01 +0000

June 26, 2023. “Pride Climb & Glow Night” at the Gravity Vault climbing gym in Poughkeepsie.

New Abbot Ceremony

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 19 Jun 2024 10:20:55 +0000

June 16, 2024. The “Descending and Ascending the Mountain Ceremony”, or Shisanshiki, at the Village Zendo in NYC. Our abbot Enkyo O’Hara Roshi retired and was succeeded by Shinryu Sinclair Roshi.

Climbing in Lost City

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 15 Jun 2024 21:38:01 +0000

June 15, 2024. Climbing in Lost City, an undocumented area of the Gunks. The local tradition prohibits publishing information about climbing routes here. The first image is of Dustin Portzline on “Keelhauled”, an ungraded roof-crack boulder problem. The others are on a nameless 5.9 route.

Mohonk Trail Builders

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 11 Jun 2024 15:15:14 +0000

June 11, 2024. The Mohonk Trail Builders lifting big rocks to build a trail in the Mohonk Preserve.

Absurdland in the Gunks

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 09 Jun 2024 17:04:01 +0000

June 7, 2024. My friends climbing Absurdland in the Gunks as a storm approaches.

Bonticou Crag

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 05 Jun 2024 14:37:06 +0000

June 3, 2024. A Gunks cliff at sunset. I did a quick hike to the cliff in the evening and scrambled up, then I alternated taking photos and eating a sandwich as the light changed. A couple of these are HDRs: I steadied my digital camera on some rocks and took three exposure-bracketed frames, then merged them in Lightroom. The blossoms are mountain laurel.

Review: Detock: High Performance Multi-region Transactions at Scale

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 29 May 2024 09:18:59 +0000

Detock: High Performance Multi-region Transactions at Scale, SIGMOD 2023. This paper is about strict serializable transactions in a geo-distributed database. It’s named “Detock” for deterministic deadlock avoidance. My presentation to the Distributed Systems Reading Group is above, and my written summary is below.

Calvin

Detock’s goals:

Strict-serializable, multi-region transactions.
Low latency and high throughput under high contention.
Handle contention between multi- and single-region transactions.

Detock descends from a lineage of “deterministic” databases invented by Daniel Abadi and others, starting with Calvin: Fast Distributed Transactions for Partitioned Database Systems in 2012. Calvin decides in advance how a sequence of transactions will execute, before the transactions fan out to the partitions. (It took me years to realize it was named for John Calvin, who taught that souls were predestined for heaven or hell.) Then there was SLOG: Serializable, Low-latency, Geo-replicated Transactions in 2019, which applies deterministic transactions to geo-replication. Detock has the same architecture and mostly the same code as SLOG, but it resolves deadlocks differently, as you’ll see.

Detock’s Architecture

Detock is partitioned and distributed across geographic regions. A region can have multiple partitions. Each item in the datastore has one home region, which holds the authoritative fresh copy of that item. An item can be asynchronously replicated from its home region to other regions; the other regions can have a read-only cache of the item.

A transaction can arrive at any partition; the partition becomes the transaction’s coordinator. Transactions are one-shot, which is a requirement of deterministic databases. We can’t do SQL-style conversational transactions. The read set and write set are either declared by the client, or can be determined with static analysis of the transaction’s code if it’s a procedure, or else the coordinator does a reconnaissance transaction to determine the read and write set. If the coordinator uses a reconnaissance transaction, and then the data changes such that the read and write set become invalid, that’s detected somehow and the transaction is retried.

The coordinator uses a “home directory” to map data items to their home regions, annotates the read/write sets with their respective regions, and forwards the transaction to all participant regions.

Single-Home Transactions

For a single-home transaction, the transaction is added to the log in its home region. These logs are stored in batches and written to disk every five milliseconds (batched to minimize disk I/O, I think).

Single-home transactions are arranged in a graph according to their dependencies, and there won’t be any cycles. The paper explains why single-home transactions can’t have dependency cycles with each other, I think this is because they’re one-shot transactions which have already been totally ordered by the sequencer. Since they can’t have cycles, they don’t have deadlocks, so the scheduler dispatches them to the workers in a straightforward way. Whenever all a transaction’s dependencies have finished, the transaction can run.

There can be single-home transactions that are in one region but multiple partitions, the Detock paper isn’t interested in these. It just says, “Transactions accessing multiple partitions in the same replica follow a deterministic execution protocol similar to Calvin and thus do not require two-phase commit.” I guess I need to re-read the Calvin paper.

Each region’s local log is asynchronously replicated to other regions. All regions replicate each other’s transactions at different times, there’s no coordination. When a region replicates a remote log it can play the transactions there to update its local copy of the remote data. I think this makes all the copies eventually consistent, but there’s no stronger guarantee.

So let’s talk about multi-home transactions, this is the interesting part.

Multi-Home Transactions

Let’s say a transaction comes to some region, and its list of items spans several regions.

Here’s my understanding of the algorithm. A client sends a transaction to an arbitrary region. The forwarder checks the read and write sets and annotates the keys with their home regions. Since this transaction involves keys in Region A and Region B the forwarder forwards it to Region B.

In both regions, the forwarder notices that this is a multi-home transaction, so it creates something called a “Graph Placement Transaction”, I’ve drawn these as T1a and T1b.

A “Graph Placement Transaction” is the part of the transaction that only uses keys in one region. So T1a is the part of the transaction on keys in A, same for T1b. Graph Placement Transactions are like single-home transactions, and they’re added to a batch by the sequencer, like single-home transactions. But scheduling them is much more complex than scheduling single-home transactions.

Let’s say another transaction T2 arrives around the same time in Region B. And let’s say that T2’s Graph Placement Transactions are sequenced so that T2 is first in Region B and second in Region A.

The two regions’ log managers communicate the sequences they chose. As these Graph Placement Transactions arrive they’re processed by the scheduler. They arrive in different orders at different regions’ schedulers. So maybe in Region A, T1 arrives before T2, and vice versa in Region B. So the scheduler can’t just execute each transaction as soon as possible once all conflicting transactions are finished, that would lead to different outcomes in different regions. We need some way for these Graph Placement Transactions to be scheduled in the same order everywhere, despite the asynchronous replication. The authors write,

GraphPlacementTxns establish an order between multi-home and single-home transactions at the region that generated the GraphPlacementTxn. However, they do not globally order multi-home transactions, since two different regions may generate GraphPlacementTxns for a set of multi-home transactions in different orders. There is thus a concern that the generated graph may contain cycles, which would lead to deadlock during processing.

This deterministic scheduling is where Detock diverges substantially from SLOG. It’s the major contribution, and the hardest for me to understand.

Deterministic Deadlock Avoidance

To explain Detock’s deadlock avoidance, first imagine you know all the transactions that will ever occur, and you know the dependencies among them. You can condense the dependency graph into strongly connected components (SCC).

A strongly connected component (SCC) is a subgraph where all the transactions are reachable from all the others. Therefore it contains at least one cycle. Within an SCC, Detock schedules transactions according to their unique transaction ids, which are assigned by their coordinators. Among SCCs there are no cycles: the blue arrows can’t form cycles, since you’ve isolated the cycles within the orange SCCs. Therefore you can just schedule the SCCs by topologically sorting them.

This works if you know all transactions, but in reality transactions are arriving continuously. When is a transaction ready to be scheduled?

For every vertex corresponding a multi-home transaction T in the dependency graph, let GPTotal (T ) be the total number of GraphPlacementTxns generated for T, a counter GP (T ) is associated with T to keep track of the number of GraphPlacementTxns of T that have been added to the graph so far. We define two types of vertices:

A complete vertex T is either a single-home transaction or a multi-home transaction with GP (T ) equal to GPTotal (T ).

A stable vertex T is a complete vertex and there does not exist a path going from an incomplete vertex to T .

So a transaction is stable in a region if its Graph Placement Transactions have all arrived, as have those for all transactions it depends on. I guess GPTotal (T ) is calculated at the beginning, when the forwarder creates the Graph Placement Transactions. Once the transaction is stable it can be scheduled; transactions are scheduled in the same order in all regions.

I think that the ordering algorithm is the same as Egalitarian Paxos (“EPaxos”), and Detock introduces the mechanism for waiting until a transaction is “stable”, but that’s just my guess.

In pathological cases, conflicting transactions continuously arrive at different regions in different orders, and the set of unstable transactions grows forever. Detock gets livelocked: each transaction’s dependencies are never resolved and Detock can never start executing it. The more often transactions arrive in regions in the same order, the lower the risk of livelock. Detock improves its chances thus: the coordinators assign each transaction a timestamp in the future; a transaction is scheduled once its timestamp has passed, by which time most lower-timestamped transactions have probably already arrived. (This is very similar to Deadline-Ordered Multicast in Nezha and several earlier papers.) The authors call this “opportunistic reordering”.

Their Evaluation

Like a lot of papers lately, Detock has a massive evaluation section. The authors compare Detock to:

Calvin, which globally orders all transactions and has no optimizations for geo-distributed transactions,
SLOG, which globally orders multi-home transactions using one ordering service in one region,
SLOG (slow), which is Calvin plus global consensus for the ordering of multi-home transactions,
Janus, an EPaxos variant optimized for geo-distributed transactions,
CockroachDB, which uses Spanner-style nondeterministic concurrency control based on synchronized clocks.

For a fair fight, they reimplemented the first four protocols within the same codebase as Detock. They vary workload skewness, network latency, the ratio of multi-home to single-home transactions, the ratio of multi-partition to single-partition transactions within a region, etc. etc. for multiple benchmarks. I was most interested in this sub-chart of Figure 6 (I’ve edited its layout):

This is a YCSB experiment with high contention (a few very hot keys) and no data partitioning within regions. It’s a rare example of a system not beating all rivals in all circumstances in its evaluation section. Detock’s throughput falls as the percent of multi-home transactions rises, since multi-home transactions require more work at more regions than single-home. When most transactions are multi-home, in fact, Calvin’s naïve algorithm actually beats Detock. The paper claims (and I agree) that this is an unlikely scenario, though.

The graph shows how critical opportunistic reordering is for Detock: the blue line at the bottom is pathetic.

In other experiments, the authors show that Detock’s distributed processing of multi-home transactions beats SLOG, where the centralized ordering service is a bottleneck. Plus, SLOG suffers more from contention, since its multi-home transactions hold locks for longer. (See the SLOG and Detock papers for details.)

My Evaluation

This paper is well-written but its content is ineluctably complex. It’s not a single clever algorithm like timestamp as a service or leader leases. Instead, it’s a new combination of existing parts, many of them already intricate. If you don’t know Calvin and EPaxos and maybe SLOG, this paper is hard to read. But its complexity is realistic: real distributed databases are horribly complex. We make them by combining all the state-of-the-art components, trying to stake out an unclaimed position on the Pareto frontier. If you think Detock is complex, try a real-world protocol like Cassandra’s.

The authors can’t claim “our protocol is absolutely the best”; as always, they have to claim “our protocol makes better tradeoffs in realistic scenarios”. Detock looks like a solid improvement over previous similar systems, and I appreciate the giant effort the authors made to benchmark Detock and report their results honestly.

Review: Timestamp as a Service, not an Oracle

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 23 May 2024 15:18:28 +0000

Timestamp as a Service, not an Oracle, by authors from Alibaba Cloud, in Proceedings of VLDB this year. Watch my presentation to the Distributed Systems Reading Group above, or read my summary below.

Timestamp Oracles

Priestess of Delphi (1891) by John Collier.

An oracle is someone who speaks for a god and reveals divine knowledge. In computer science we’ve used “oracle” to refer to theoretical machines that could do something impossible, like solve the halting problem or produce truly random numbers. Also, for some reason, real actual machines that produce monotonic timestamps are called “timestamp oracles”.

A timestamp oracle is used by a distributed database to get monotonically increasing IDs, for ordering events. It’s a single server in your data center which provides a larger number every time you ask it. The timestamp might or might not be related to the wall clock.

Why not Lamport clocks or vector clocks? They require you to pass these clock values between clients and servers, through all the layers of your multi-tier architecture. I know personally that database users find that burdensome.
Why not synchronized clocks? Syncing clocks is hard, and no matter how precise the clock is, you need to add some latency to wait out the uncertainty. Well-synced clocks are becoming widely available, though; see Huygens and AWS Time Sync.

The paper mentions that timestamp oracles are used by various distributed systems: PolarDB-X, OceanBase, CORFU, TiDB placement driver, Percolator, Postgres-XL. I’ve only heard of half of these. The TiDB placement driver (“TiDB-PD”) includes a timestamp oracle in its implementation, and it’s the main example that this paper’s authors use as a comparison for evaluating their alternative.

Timestamp oracles only work for one data center. The timestamp consumers should be on a very low-latency link to the timestamp oracle server; cross-DC links are too slow for practical use. This paper doesn’t try to solve that problem: this paper’s timestamp-as-a-service is also intended for one data center.

A fault-tolerant timestamp oracle is a consensus group: each new timestamp is majority-committed. If the leader fails, the next leader must know the highest timestamp that the previous leader produced. The paper mentions that this can be optimized: the leader could reserve a range of timestamps. It majority-commits the range, and it gives out timestamps until the range is exhausted. A new leader reserves a higher range of timestamps than any previous one.

A consensus group is fault-tolerant, but nevertheless, losing the leader causes some brief unavailability. Especially since this consensus group must use timed leader leases, for speed and consistency. Therefore the new leader has to wait for the previous lease to expire. The paper shows that TiDB-PD is unavailable for 10 seconds after the leader dies. The black throughput line drops to zero each time the leader is killed:

Figure 9 from the paper, lower half

Besides being a single point of failure, the leader is a bottleneck—you can’t get timestamps from followers, so a system could saturate the timestamp oracle leader.

A Service, Not An Oracle

The authors propose a timestamp service, rather than an oracle. They say an oracle is one server, therefore a single point of failure, even if it’s the leader of a consensus group. A service, however, is not a single point of failure. I don’t know if everyone agrees with these definitions of oracle and service, but it’s a useful distinction for this paper. The paper says a Timestamp-as-a-Service is “a distributed algorithm that computes logical timestamps from a consensusless cluster of clocks.” That means no unavailability from loss of the leader, and no bottleneck on the leader. Consensuslessness is not only fun to say, it’s the funnest part of this paper!

The Algorithm, 1.0

The paper presents the Timestamp-as-a-Service (TaaS) algorithm in two stages, starting with a simplified version that assumes no server failures or message loss. There can be any number of clients, and N servers.

A client starts a session S by sending the bottom timestamp ⊥ to all servers. The paper generally uses σ (sigma) for a session but I’m going to use S because I don’t read Greek. That symbol for “bottom timestamp”, you could think of it as negative infinity. Whenever you see that symbol you know you’re probably in the world of lattices and order theory and abstract algebra.

A “session” in this paper is not a sequence of database commands. It’s just the commands required to get one timestamp, then the session is over. The client gets its timestamp and then forgets everything, no state persists into the next session. So the client always starts its session by sending ⊥, even if it got a timestamp in a previous session.

Each server has a persistent timestamp, which is somehow initialized to some value when the server is born. When the server receives the client request, it increments its timestamp and sends its new current timestamp back to the client. Thus if a server replies with “5”, the client knows the server’s current timestamp is 5, and it knows the server had a timestamp less than 5 before the client talked to it. This will be important.

The session is complete once all servers have replied. The client uses the M^th‑smallest timestamp from the replies, for some M ≤ N. All clients must use the same M.

M can be anything! M should be the smallest majority, so if N is 5 then M should be 3. The paper starts abstractly, by showing TaaS is correct for any M from 1 to N. Eventually it admits that for maximum fault tolerance, M should be the smallest majority, same as a quorum in Paxos or any consensus algorithm. But for now, M is any number 1 through N.

Let’s look at an example of TaaS 1.0 in action.

Figure 2 from the paper.

There are Clients V and Client W, and Servers X, Y, and Z. Session Alpha starts concurrently with Session Beta. Session Gamma starts after Session Alpha. Let’s say M = 2, so at the end of each session the client chooses the second‑smallest timestamp from all the server replies.

Session Alpha gets timestamp 1, Beta gets timestamp 2, and Gamma gets timestamp 3. Beta’s allowed to have any timestamp since it’s concurrent with the others. So you can see these timestamps uphold the linearizability guarantee. The important constraint is, Session Alpha’s timestamp must be less than Session Gamma’s, and it is: 1 is less than 3.

The client could send its messages to all servers in parallel, or any order within a session, and the latencies could be of any length, TaaS still works.

Theorem 1

“The timestamp for session T is guaranteed larger than the timestamp for any session S that ended before T began.”

This sounds like a linearizability guarantee, and I believe you could call the timestamp a linearizable data structure. Proof:

The M^th‑smallest response in S ≤ the M^th‑smallest server state at the end of S. (Servers’ timestamps increase monotonically, so their timestamps at the end of S must be ≥ their responses in S.)
The M^th‑smallest server state at the end of S ≤ the M^th‑smallest server state at the start of T. (Monotonicity, and S ends before T starts.)
The M^th‑smallest server state at the start of T < the M^th‑smallest response in T. (The client makes all the servers increment their timestamps during T.)

So by transitivity, the left side of #1 < the right side of #3. Q.E.D.

The Algorithm, 2.0

In the second and final version of TaaS, servers can crash-fail indefinitely, and they can come back online.

A restarted server still guarantees monotonicity, i.e. it remembers the last timestamp it produced before it crashed. The only way to guarantee this is some sort of replication.The paper suggests using RAID, or cloud storage like S3 which has its own replication, or make each timestamp server a consensus group with fault-tolerance. You might ask, if we make each timestamp server a consensus group, haven’t we come full-circle to the “timestamp oracle” that the paper says is bad? Not quite: now we have a separate group for each timestamp server, so if one group loses its leader, TaaS can proceed with the other groups without waiting for a new leader.

The TaaS client’s goal is to find a timestamp t such that:

t > M^th‑smallest of all servers’ timestamps when the session started
t ≤ M^th‑smallest of all servers’ timestamps when the session ends

These two properties are the two facts that Theorem 1 depends on.

Here’s an example where Server X is partitioned from the client. Let’s say M=2; we want the 2^nd‑smallest timestamp.

From Figure 3.

The client remembers that it got timestamp 5 in some past session from Server X. This memory is a new feature of the fault-tolerant version of TaaS.

In session δ, the client gets a 4 and a 5 from the servers it can reach. It picks 5 as the timestamp for this session, because the client knows, without talking to Server X:

5 > 2^nd-smallest timestamp at the start of the session: The client heard Servers Y and Z respond with timestamps 4 and 5, so their previous timestamps were less than 5.
5 ≤ 2^nd-smallest timestamp at the end of the session: The client knows Servers X and Z now have timestamps at least 5.

The two facts that Theorem 1 relies on are both true, so the client can pick 5 without talking to Server X.

From Figure 3.

In session ε the client gets 5 and 6 from the servers it can reach. Now it doesn’t know the second‑smallest timestamp. If Server X is talking to some other client, it might have advanced to 7; then the second‑smallest would be 6. Or Server X might be at timestamp 5.5—timestamps don’t have to be integers! (I wish the paper had mentioned this earlier.) If Server X has 5.5, then 5.5 would be the second‑smallest. We don’t know. What’s the solution?

The client does something new now. Instead of sending the bottom timestamp to all the servers, like it did before, it continues this session. It thinks 6 might be the second‑smallest timestamp. So it sends 6 to any servers that might have less than 6: Server Y. Then Server Y updates its own timestamp to at least 6, increments it by one, and returns it.

Now the client has 6 from Z, 7 from Y. X is still down, but the client doesn’t care, it can choose 6:

6 > 2^nd-smallest timestamp at the start of the session: The client heard Servers Y and Z respond with 5 and 6, so they started with less than 6.
6 ≤ 2^nd-smallest timestamp at the end of the session: The client eventually heard Servers Z and Y respond with 6 and 7, they ended with at least 6.

So the client achieved certainty by advancing one of the server’s timestamps. That’s a specific example, here’s the general algorithm.

Algorithm 2.0 pseudocode

The paper uses a pseudocode that’s hard for me to read, so of course I’ll make my own pseudocode that seems easier to me but may be worse for you. Sorry.

// global map from server to timestamp, initially bottom timestamp for
// all servers. the client persistently tracks this. 
cache = {server: ⟘ for server in servers}

// client code to acquire one timestamp
def do_session():
  // "session" is like "cache" but reset with top timestamp each session
  session = {server: ⟙ for server in servers}

  // like TaaS 1.0, initially send bottom timestamp to all servers
  send timestamp ⟘ to all servers
  while true:
    reply = await next reply

    // update global cache with max per server, local with min
    cache[reply.server] = max(cache[reply.server], reply.timestamp)
    session[reply.server] = min(session[reply.server], reply.timestamp)

    if we received at least M replies:
      candidate = Mth-smallest value in session
      if candidate ≤ Mth-smallest value in cache:
        return candidate
      else if no more quickly-available replies:
        // promote the candidate
        for server in servers:
          if cache[server] < candidate:
            send candidate timestamp to server

As the client gets replies, it updates the global and local maps. The purpose of these two maps is to determine when we know the two facts we need for Theorem 1.

Each server’s timestamp is guaranteed to increase monotonically, but a client could get replies out of order. So it uses max when updating the global cache, to ignore delayed messages. The client uses min when updating the per-session map called session. I don’t understand why; it seems that if the client got a 6 from Server X, then a 5, it should keep the 6, because it knows that 5 is a delayed message. The algorithm is still correct, it just has unnecessary retries.

How does the client know when it’s acquired a correct timestamp, at the return timestamp line? Remember, the goal is to find a timestamp t such that:

t > M^th‑smallest of all servers’ timestamps when the session started: at least M servers started < candidate, according to session-local map
t ≤ M^th‑smallest of all servers’ timestamps when the session ends: at least M servers end > candidate, according to global map

The client knows candidate satisfies the first criterion, because according to the session-local map, there are M servers that had smaller timestamps when the session started, before the client talked to them. All the values in the session-local map are values that servers incremented during the session, so the starting values were smaller.

The client knows candidate meets the second criterion because it explicitly checks it in the global cache. The client hasn’t talked to all the servers, but it’s talked to enough of them to know that this fact is true. It wants to ensure that future sessions are guaranteed to get a larger timestamp than this one, and it’s learned enough to guarantee that.

If the client has no candidate, it tries to make the second criterion true by advancing the timestamps on servers that might have less than candidate. The client tries all lagging servers, including those that were unavailable recently, in case they came back online.

At the line, else if no more quickly-available replies, there’s some timeout while the client waits to see if the remaining servers are going to reply soon or not. I guess that’s a tunable parameter.

What Should M Be?

It’s finally time to descend from lofty abstraction and decide a value for M. The authors write, “The system allows at most min(N − M, N − 1) downs while continuing its service.” So for optimal fault-tolerance M should be the median of 1 .. N:

I suppose it’s theoretically interesting that M could be any number, but practically, it should be a majority, which makes this protocol resemble consensus a lot. However, the authors insist this isn’t consensus, because servers never talk with each other. TaaS is a “bipartite architecture”, where clients and servers talk to each other, but servers don’t talk to servers and clients don’t talk to clients. There’s a nice little discussion in Section 6.2 of the definition of consensus and consensusless consistency.

Unique Timestamps

Concurrent sessions may get the same timestamp. Solution: each server appends its server ID to the returned timestamp.

Their Evaluation

Comparing TaaS to a consensus-based timestamp oracle (TiDB-PD), the authors claim:

In the happy case, TaaS is higher-latency than TiDB-PD, especially with more servers.
TiDB-PD stalls for 10 seconds after leader failure.
TaaS latency doesn’t increase much when servers are slow or dead, if a majority is healthy.

Thus TaaS is closer to constant work, which makes it more stable as part of a whole system and avoids metastable failures. Here’s Figure 9 again, now with TaaS included:

At the 1- and 2-minute marks, the authors killed a server. TiDB-PD suffered a complete stall in throughput, TaaS merely increased latency a bit.

My Evaluation

The algorithm is fairly simple, but the authors explain it with needlessly weird notation. These are three pieces of notation actually used in this paper:

This looks less like math and more like an alien language. The one on the bottom means the “second‑smallest timestamp received by session delta.” I guess that’s ok, but they never explain why delta gets an upside-down hat? Is that a rule in this alien culture?

More seriously, I’m curious about several questions:

TaaS doesn’t assume timestamps are integers. If it did, i.e. if there were a minimum increment amount, could the fault-tolerant algorithm be more efficient?
What happens when a client restarts and loses its long-lived global cache? Does that weaken fault-tolerance?
What about reconfiguration?: how are timestamps servers initialized, added, or removed?
Has Alibaba deployed this? It seems like they haven’t. Why?—did they use synced clocks instead?

I enjoyed this paper, and spent a long time understanding it (as you can see). It describes a new protocol in the classic distributed systems style. It provides rigorous explanations and proofs, and informative experiments. If you need monotonic ids within one data center and you can’t use synchronized clocks, TaaS is a simple solution with stable performance during failures.

Update: One of the authors has answered some of my questions about the protocol.

How To Use MongoDB Causal Consistency

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 14 May 2024 15:53:22 +0000

MongoDB implemented a consistency level called causal consistency in 2017 (version 3.6). It’s quite a handy consistency level, strong enough for most applications and still performant. I helped design the API for causal consistency, but when we released it I dropped the ball and didn’t publicize or document it well. MongoDB’s causal consistency didn’t get the fame it deserved in 2017; I’ll try to rectify that now.

MongoDB implements a Raft-like consensus protocol. Most people deploy MongoDB as a three-server replica set, with one primary and two secondaries. Only the primary executes writes, which are replicated by secondaries with a small delay. You can read from the primary or the secondaries. Your application talks to the servers via an instance of the MongoClient class.

The Problem With Secondary Reads

Liam

Let’s start with the classic example: you’ve implemented a social media site, with MongoDB as your database. Some influencer named Liam posts something witty to your site, then clicks on his profile to admire his wittiness. But Liam doesn’t see his latest post! He desperately refreshes, and phew: soon his new post appears. What happened?

The problem is that you have tried to decrease load on the primary by reading posts from secondaries. For example, in Python:

client = MongoClient("mongodb://srv1,srv2,srv3/?replicaSet=rs")
# Configure this collection object to read from secondaries.
posts_collection = client.db.get_collection(
    "posts",
    read_preference=ReadPreference.SECONDARY)

# The insert message goes to the primary
# and awaits majority replication.
posts_collection.insert_one(
    {"user_id": "Liam", "contents": "I'm very witty"})

# Read from secondary, due to "read_preference" above.
user_posts = list(posts_collection.find({"user_id": "Liam}))

MongoClient sends the insert command to the primary, which by default awaits acknowledgment from a majority of servers: in a three-server replica set, a majority is the primary plus one secondary. But then, MongoClient sends the find command to a random secondary—perhaps the other secondary, which may not have replicated the new post yet.

Reading from secondaries is unpredictable. You can’t reliably read your writes. You can’t do monotonic reads either: as you execute a series of reads on secondaries, you’ll use secondaries with different amounts of replication lag, so your data will seem to randomly jump back and forth in time. Secondary reads give you only the weakest guarantee, eventual consistency.

Causal Consistency

We want you to be able to read from secondaries with reasonable consistency, so in MongoDB 3.6 we introduced causal consistency, which guarantees read-your-writes and monotonic reads. As we wrote in Implementation of Cluster-wide Logical Clock and Causal Consistency in MongoDB,

Causal Consistency is defined as a model that preserves a partial order of events in a distributed system. If an event A causes another event B, causal consistency provides an assurance that every other process in the system observes the event A before observing event B. Causal order is transitive: if A causes B and B causes C then A causes C. Non causally ordered events are concurrent.

You enable causal consistency with a session:

# Note read_concern, which isn't in the code above.
posts_collection = client.db.get_collection(
    "posts",
    read_preference=ReadPreference.SECONDARY,
    read_concern=ReadConcern("majority"))

# start_session() has causal_consistency=True by default.
with client.start_session() as s:
    posts_collection.insert_one(
        {"user_id": "Liam", "contents": "I'm very witty"}, session=s)

    # Read your previous write, even from a secondary!
    user_posts = list(posts_collection.find(
        {"user_id": "Liam}, session=s))

    # This query returns data at least as new as the previous
    # query, even if it chooses a different secondary.
    n = posts_collection.count_documents({}, session=s)

You have to pass the session parameter with every command. This is too easy to forget—I forgot it when I first wrote this example!

In a causally consistent session, you’ll read your writes and get monotonic reads from secondaries. Both read concern and write concern must be set to “majority”, as explained in the MongoDB manual. “Majority” is the default write concern, so I configured only the read concern explicitly.

How does MongoDB ensure causal consistency? It uses a logical clock (aka a Lamport clock) called clusterTime to partially order events across all servers in a replica set or sharded cluster. Whenever the client sends a write operation to a server, the server advances its logical clock and returns the new clock value to the MongoClient. Then, if the MongoClient’s next message is a query, it passes the afterClusterTime parameter, which asks the server to return data including all writes up to that clusterTime. If the server is a lagged secondary, it waits until has sufficiently caught up:

If you query a secondary that hasn’t yet caught up to that point in time, according to the logical clock, then your query blocks until the secondary replicates to that point. (Yes, the parameter is called afterClusterTime, but the secondary only needs to replicate up to that clusterTime, not after it.)

The Fine Print

As I said, causal consistency requires write concern “majority” (the default) and read concern “majority” (not the default). Other configurations give weaker guarantees, see the thorough documentation.

Your client-side session object tracks, in memory, an ever-increasing clusterTime value, which it exchanges with every MongoDB server it talks to. If you lose the clusterTime, you lose causal consistency. Thus you must use the same MongoClient and the same session object throughout. Except, it’s possible to…

Transfer the clusterTime

Here’s the trick we should’ve publicized: you can transfer the clusterTime from one session to another. The sessions can’t be concurrent—you must wait for one session to end before you use the next one—but even if the sessions belong to different MongoClients, or different processes, or they run on different machines, they’ll form a causally consistent chain.

Continuing the example above, inside the “with start_session” block, capture the session’s clusterTime and operationTime:

with client.start_session() as s:
    posts_collection.insert_one(
        {"user_id": "Liam", "contents": "I'm very witty"}, session=s)
    cluster_time = s.cluster_time
    operation_time = s.operation_time

Now you have logical clock values from the primary after it inserted the post. You can create a new session, optionally on a different client or even a different machine, and fast-forward its logical clock:

client2 = MongoClient("mongodb://srv1,srv2,srv3/?replicaSet=rs")

# Same get_collection args as before.
posts_collection2 = client2.db.get_collection(
    "posts",
    read_preference=ReadPreference.SECONDARY,
    read_concern=ReadConcern("majority"))

with client2.start_session() as s2:
    s2.advance_cluster_time(cluster_time)
    s2.advance_operation_time(operation_time)
    user_posts2 = list(
        posts_collection2.find({"user_id": "Liam"}, session=s2))

After you call advance_cluster_time and advance_operation_time, subsequent operations on that session (don’t forget the session parameter!) are guaranteed to reflect all changes up to that time.

See the MongoDB Manual’s example code for transferring logical clocks with each driver.

Not So Convenient

Why do you need to transfer two clock values between sessions, instead of one? It’s bad API design; it reveals implementation details that could have and should have been hidden. I was in a rush during the MongoDB 3.6 cycle and I didn’t take the time to understand our logical clocks and propose a convenient API. It’s hard to change now, millions of people depend on the current APIs, but some MongoDB engineers are pushing to fix it.

Anyway, transferring two values isn’t so bad, the real inconvenience is piping these values through the layers of your application stack. If you use MongoDB in your web application, you probably have something like this:

Guaranteeing causal consistency with this architecture is toilsome. When Liam posts his witticism, he clicks a button in the frontend (a Javascript web app in this example), which sends the post to the middle tier, which calls insert with the MongoDB driver. Then the middle tier must capture the session’s clusterTime and operationTime, and return them to the frontend, which saves them in web storage. When Liam refreshes the page, the frontend must load the clusterTime and operationTime from web storage and send them with its request to the middle tier, which uses them to call advance_operation_time and advance_cluster_time on its session before executing find on a secondary. This guarantees Liam sees his post, but what a pain in the tuchus!

In the years since MongoDB 3.6, we could’ve documented this process better, and encouraged framework authors to build it into their application frameworks to ease the burden on developers. We didn’t do that. As far as I know, causal consistency is rarely used. It’s a shame, since it’s performant and conceptually simple.

It’s not too late to make causal consistency popular. If you want to help (especially if you maintain a multi-tier app framework), please write to me! But I now guess that consistent secondary reads without application logic are the real solution. I hope to research it later this year.

Monks at Holy Cross, part 2

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 22 Apr 2024 16:26:06 +0000

April 15, 2024. Monks at Holy Cross, an Anglican Benedictine monastery in the Hudson Valley. See part 1 here.

Jyakuen's Shuso Hossen

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 24 Mar 2024 22:08:13 +0000

March 24, 2024. A shuso hossen ceremony at the Village Zendo. Sherri Jyakuen Posey gave her first dharma talk and became a senior Zen student.

Monks at Holy Cross

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 16 Mar 2024 10:19:46 +0000

March 5, 2024. Monks at Holy Cross, an Anglican Benedictine monastery in the Hudson Valley. See part 2 here.

Bodhisattvas Always Smile

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 13 Mar 2024 22:42:22 +0000

I gave a talk at the New Paltz Zen Center against doomerism. We vow to bear witness to all the suffering in the world, but there’s no reason to be gloomy about it. Let’s be cheerful bodhisattvas and enjoy our lives!

Here’s three images. When you read the news, or you think about the state of the world, which one are you? What’s your state of mind?

This is Edvard Munch’s “The Scream”, painted in 1893. There’s an emaciated figure, holding its head in its hands and its mouth is open like it’s howling, it’s in a disorienting, apocalyptic landscape with a red-streaked sky.

Munch wrote about the inspiration for this painting: “One evening I was walking along a path, the city was on one side and the fjord below. I felt tired and ill. I stopped and looked out over the fjord – the sun was setting, and the clouds turning blood red. I sensed a scream passing through nature; it seemed to me that I heard the scream. I painted this picture, painted the clouds as actual blood. The color shrieked. This became The Scream.”

The second image is a comic strip. There’s a dog sitting at a table with a cup of coffee, and all around it the house is on fire, and a cloud of smoke hangs under the ceiling. The dog is wearing a little bowler hat, it has a stupid smile on its face, and it’s saying, “This is fine,” as huge flames close in around it.

This was drawn in 2013 by an artist named K.C. Green for his webcomic, it’s part of a 6-panel comic where the dog catches fire as he says, “I’m okay with the events that are unfolding currently”, “that’s okay, things are going to be okay”. Then his face melts off.

The first two panels became an Internet meme that people use to express, “You’re in denial. There’s a crisis of some sort and you should be panicking.”

This is an image of Avalokiteshvara, painted on a cave wall in Nepal in the 12th Century. Avalokiteshvara’s name means literally “down-looking lord”. But, I learned that the name was originally Avalokitasvara—slightly different—and it meant “perceiving sound”, which is how we think of the Bodhisattva, as a being that hears the cries of the suffering world. This being was first mentioned in the Avatamsaka Sutra, written about 2000 years ago, where he appeared as male. In the Lotus Sutra, slightly later, Avalokiteshvara has many forms, both male and female.

In China he or she is called Guanyin and can be male or female. This statue of Guanyin is at the Met, it was made in the 10th or 11th Century in China.

Here’s a contemporary statue. Here she’s pouring the water of compassion from a container.

In Japan, Guanyin is called Kanzeon or Kwanon or Kannon. The Canon camera company is named after her, by the way. Here’s an early Canon logo.

What I want you to notice is, even though Kanzeon hears the cries of all the suffering beings of the world, she’s smiling. Or at least, she looks peaceful. At the Village Zendo we are all on the Bodhisattva Path, we chant the Bodhisattva Vows, bodhisattvas like Kanzeon are our role models. So let’s ask ourselves: when we hear the cries of the suffering beings, are we smiling, or not?

Zenshin Caplow gave a terrific talk at the zendo a couple weeks ago, she focused on the courage of a bodhisattva. Please listen to that talk, it’s super insightful. Today, I’m not talking about courage, but about the cheerfulness of a bodhisattva!

Consider those three images: The Scream, or the oblivious dog in the burning house, or smiling Kanzeon. Which way are you?

I hear depression or exhaustion from a lot of people in the Village Zendo, and a lot of friends, and a lot of people who care about justice and democracy and the climate. What I don’t hear is people making it their goal to cheer up and enjoy their lives, despite the daily crises.

So I’m going to try to convince you to make the last image, the smiling Kanzeon, your role model. I want you to make it your goal to be cheerful and enjoy your life no matter what world events you hear about. I have three reasons why you should do this:

The world isn’t going to hell.
You’re a more effective bodhisattva if you’re cheerful.
Emptiness.

Number one, the world isn’t going to hell. Do you think it is? Do you think we’re all going to die of global warming, or people are starving, or that the world is getting more violent? You can slice statistics in infinite ways, but overall, life has improved for nearly everyone, and it’ll probably continue to improve for a long time.

Global poverty is declining rapidly. There’s a statistic called the International Poverty Line, defined as living on less than $2.15 a day. The number of people below that line has decreased from nearly 40% of people in 1990 to less than 10% today, in other words the share of the population in extreme poverty is a quarter of what it was in 1990. In the same period infant mortality fell by half. Deaths from famine have become extremely rare compared to past decades.

As I say this, I’m aware that people are dying of hunger in Gaza right now. I’m not minimizing that. The opposite: I’m saying we must act to end hunger everywhere, and we should be encouraged, knowing that it’s possible, because we’ve nearly accomplished it.

Of course, the big negative trend is global warming. But it doesn’t mean the planet is going to burn up or we’re all going to die. The Intergovernmental Panel on Climate Change says the most likely scenario is 2 or 3 degrees Celsius of warming by the year 2100. That means we’re going to be hotter than the goal that they set, but not by too much, and it’s far better than the worst-case scenario. A lot of climate activists will cite scenarios where Manhattan is underwater and all of Africa is a desert, but that’s based on the IPCC’s worst-case scenario, and it’s now clear that won’t happen, partly because people are taking action! The US’s total carbon emissions, for example, have been falling since 2007. Plus, as countries get richer, they tend to use more efficient energy sources. And as countries get richer their population growth slows. The global population will probably peak in the year 2100 or sooner, and start shrinking after that, which should make it easier to reduce carbon emissions. Taking everything together, the IPCC projects that over the next century, despite global warming, global poverty will keep falling, hunger will be rarer, average lifespan will increase, standards of living will improve. Global warming is bad, but humanity’s progress continues anyway.

Of course there are crises. The last few years have been fairly violent, with wars between Russia and Ukraine, and between Israel and Hamas. And there have been a few other wars. Just as an example of how distorted our view can be, the Tigray War in Ethopia a few years ago was much bigger than the Israel-Hamas war but it was hardly covered by US media.

Kanzeon doesn’t lose her cool when a war breaks out. She knows there have been far more violent years in the 70s and 80s, and of course those years are dwarfed by WWI and WWII. Overall, wars between countries have become very rare compared to the 20th Century. And anyway, Kanzeon understands human nature, we’re prone to violence because we have greed, anger, and ignorance. So she expects there to be some wars from time to time. It doesn’t change her mission. Kanzeon just keeps working to wake us up.

If you think the world is going to hell, that’s your view, it’s arising in your mind. The world seems to be going to hell because of what you choose to focus on, and that is probably influenced by how you consume the news. And the news is influenced by media companies’ desparate need for your attention, it’s what they need to survive. Of course they focus on the negative! Global poverty keeps falling year after year, but media companies know that stories about continuous improvement don’t generate clicks. Be aware of this distortion and try to correct it, seek out good news, and seek out information about big, long-term trends, not the brief crises of the day.

Kanzeon sees the whole world over the long term, that’s why she keeps smiling through each crisis.

Number two, you’re a more effective bodhisattva if you’re cheerful.

If we imagine that today’s wars are unusually bad, it can seem like we’re losing the struggle for peace, and that’s discouraging. Or if we think we’re all doomed on a burning planet, that’s super discouraging. I hear people say there’s no point bringing children into a doomed world. Have children or don’t, I have no opinion, but make an informed decision: their lives will probably be better than ours. If we’re aware of the progress we’re making, towards peace, towards a carbon-neutral economy, towards global prosperity, that encourages us to keep trying!

There’s a story about Kanzeon, which of course has many versions, so here’s mine: Once upon a time, early in Kanzeon’s career, he had just one head and two arms. But he worked furiously hard and liberated countless beings from suffering. He paused, he looked around, and the number of suffering beings still seemed overwhelming. He was so frustrated his head exploded. Buddha put his head back together in a different form: it became 11 faces. A peaceful face to calmly gaze in each of the 10 directions, so Kanzeon wouldn’t lose track again. And on top of these 10 faces, Buddha placed an 11th face, a wrathful face for the suffering beings who need a kick in the ass to liberate them.

So Kanzeon went back into action. But trying to save all beings at once, he moved his arms so fast they shattered. So Buddha did some magical upgrades, and gave Kanzeon 1000 arms, and in each of his 1000 palms Buddha placed an eye, so Kanzeon could give individual attention to every being simultaneously.

But you and I are not 11-faced and 1000-armed. Generally we have one face each, and at most two arms. If I tried to respond to every crisis simultaneously, my head would explode and my arms would shatter. I need to remember that this body, this brain, is just one of Kanzeon’s hands and eyes. Or maybe I’m one of her fingers, maybe I’m one of her cells. I can’t do it all. So I focus where I think I’m effective: I lead the sangha at Sing Sing, I donate 10% of my income to GiveWell, which directs it to the most effective charities, like vitamins and vaccines to people in Africa. I support the Village Zendo and New Paltz Zen Center. I focus on a few things, and I’m satisfied. I am doing enough!

Kanzeon responds to all the cries of the world, no one and nothing is left out, and my actions are a part of that.

Since I’m doing enough, I don’t read too much news. I don’t need to know the ins and outs of US politics. I don’t need to follow the daily events of every war. It’s okay. I need to be informed, but I don’t need daily information to do my bodhisattva work. If I got daily information it would be bad for me.

If you take one piece of advice from this whole talk, it’s this: turn off news alerts on your phone. If Shakyamuni Buddha were here today, he’d make all his monks and nuns turn off their news alerts before he shaved their heads. It’s essential.

If you have an iPhone, here’s how to do it. Open Settings, search for whatever news app is sending you alerts, like maybe the NYTimes app, then tap Notifications and turn them off. If you have an Android or a smartwatch, I’m sure it’s equally easy. It is so important that you do this.

You might be wondering, how can a bodhisattva ignore the daily news? What about bearing witness? Bearing witness is one of the tenets of the Zen Peacemaker Order, which we belong to. We occasionally recite these tenets and vow to follow them. So am I allowed to focus on a few things, and turn off the news? Yes I am! Bearing witness is intimate. It’s being fully present with someone. You have to enter their world and experience it with them. You can’t do that if you’re being interrupted by New York Times headlines. The headlines are a media company’s interpretation of the world. Bearing witness is a direct experience. The two things could not be more different. If you turn off the news it gives you space to bear witness, and I find that when I do bear witness, it fulfills me and gives me courage to act, cheerfully!

So that was number two, a cheerful bodhisattva is an effective bodhisattva. My third and final argument why you should try to be cheerful is: emptiness. Nothing is separate or solid, it only briefly appears that way. Beings appear and disappear constantly, like waves rise up from the ocean and fall back into it. Fairly soon, everyone in this room and this Zoom call will die. Later, the sun’s fusion reaction will use up all its hydrogen, and it will start fusing helium instead; it will heat up and destroy the earth. If humanity escapes the earth, we could survive a while longer, but some day the whole universe will run out of energy and become cold and silent and everything will die.

What I like about Buddhism is that we face the facts. We practice accepting death. Unlike the oblivious dog saying “this is fine”, we do see the flames and we know what’s going to happen. But we accept it, because the beings that appear and disappear are just waves in the ocean. They are empty of meaning as individuals, they are inseparable parts of the magnificent whole. The knowledge of this fact is called prajna paramita, “wisdom perfected”. As the Heart Sutra tells us, Avalokiteshvara practices prajna paramita! The paradox of being a bodhisattva is to value each individual and care about their well-being, without ever forgetting that we are each just a momentary expression of the whole shebang, never really separate. When we see this, our own peace of mind is stable, it doesn’t depend on anything, certainly not on the daily news cycle.

As Samantabadhra Bodhisattva says in the Avatamsaka Sutra:

Even though the void of space, together with the states of beings, with the karmas of beings, with the sorrows of beings, though all these will end, yet my approval and joy in the merits of all beings will not end.

Samantabadhra isn’t depressed by the thought of the end of the world, actually he sounds ecstatic, because he knows his joy doesn’t depend on anything but his vow.

So I’ve argued that the bodhisattva path is cheerful action and enjoying our lives. The bodhisattva vow isn’t vowing to be depressed about the state of the world, it’s vowing to be like Kanzeon—both in our attitude and in our action. Think again about those three images, the scream, the oblivious dog, the smiling bodhisattva. If you feel more like screaming than smiling, ask yourself: what about my practice is having this result? What might I do differently to have a different result?

The hardest news for me to handle, lately, is the likelihood that Trump will be reëlected. But I do not intend to be anxious about it. My method is, instead of obsessively reading news coverage, I’m just doing what I can. Keishin and I will lead a fundraiser for Biden this fall, we did this in 2020 and helped raise tens of thousands of dollars, and we’ll do that again. And I’ll take September and October off work and look for ways to volunteer for Democrats. My local congressman is a Democrat. We’re in a swing district here, he won by one percent last year, so working for his reëlection is a high-leverage way to make a difference. Electing Democrats seems like the most effective way to combat global warming and protect human rights. We all know Democrats are far from perfect, but it looks like the bodhisattva path to me.

It’s my experience that obsessing about politics is depressing, but participating in politics is really fun and satisfying. So I’ll do what I can during this campaign, and then I’ll move on and enjoy my life. A bodhisattva is never stuck, a bodhisattva’s action is always flowing.

If Trump wins, I don’t intend to be depressed about it. I’ll keep up with the news as much as is useful, and that’s it. I’ll cheerfully work to contain the damage however I can, and I’ll move on and enjoy my life. I won’t get stuck, and I won’t fall apart.

I’ll leave you with a verse by Enkyo Roshi, from her book “A Little Bit of Zen”:

We can’t lose our heads.
Oh, Bodhisattva!
Losing your head doesn’t help—
Instead, extend your hand.
Offer your words, your wisdom, your effort.
That’s what a Bodhisattva does, day in and day out.
Serving all of life.

Ordination at Christ Episcopal

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 11 Mar 2024 22:43:28 +0000

March 5, 2024. An ordination ceremony for Emily Carter at Christ Episcopal Church in Poughkeepsie.

Climbing in Puerto Rico

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 15 Feb 2024 16:16:36 +0000

February 2024. Photos of my friends (and one friendly stranger) sport-climbing on the limestone crags near San Juan (Monagas and Flying Coconut).

Review: Antipode: Enforcing Cross-Service Causal Consistency in Distributed Applications

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 29 Jan 2024 21:02:14 +0000

Ptolemy Map, 1482

In 2015 some Facebook researchers threw down a gauntlet, challenging anyone who dared to provide stronger consistency in big, heterogeneous systems. In 2023, some researchers (mostly Portuguese) responded with Antipode: Enforcing Cross-Service Causal Consistency in Distributed Applications. Antipode defines an interesting new consistency model, cross-service causal consistency, and an enforcement technique they claim is practical in such systems.

Motivating Example

Here’s an example from the paper. A social network is composed of services:

My simplification of the paper’s Figure 2

This system has the following problematic workflow:

Author uploads a post; it’s received by the Post-Upload service in Region A.
Post-Upload sends the post to Post-Storage,
… which saves the post to a local replica of its datastore.
Post-Upload tells the post-id to Notifier,
… which saves the post it a local replica of its (separate) datastore.
Both datastores eventually replicate to Region B in arbitrary order.
In Region B, as soon as Notifier replicates the notification,
… it triggers Follower-Notify,
… which retrieves the post from Post-Storage,
… which retrieves it from its local replica of its datastore.
Once Follower-Notify has the post, it relays it to Follower.

The problem, as astute readers can predict, is if Notifier’s datastore replicates sooner than Post-Storage’s. In that case, Follower-Notify will learn about the post too soon; it’ll try to retrieve it from Post-Storage but the post won’t be there yet.

This is a consistency violation of some sort—the paper will define it exactly, in a moment. We could prevent the anomaly by making all datastores replicate synchronously. In that case, once Post-Storage has acknowledged storing the post in Region A, it has also replicated it to Region B, so it’s certainly there by the time Follower-Notify tries to retrieve it. But this kills all parallelism. Maybe there’s a better way?

Causal Consistency Isn’t Enough

What about Lamport-style causal consistency with logical clocks? This wouldn’t prevent the anomaly. The paper doesn’t explain in detail why, so I’ll try.

In Lamport’s algorithm, each process has a clock value (perhaps just an integer), which is incremented and propagated whenever processes exchange messages. Lamport clocks could prevent the anomaly if we had one replicated datastore:

Post-Storage in Region A saves the post, gets a Lamport clock value of 42 from the datastore.
Post-Storage in Region A directly notifies Post-Storage in Region B and tells it the clock value.
Post-Storage in Region B reads from its replica of the datastore. It tells the datastore to wait until it’s replicated up to clock value 42 before executing the query:

Many datastores (including MongoDB) support causal consistency this way, and it would prevent the anomaly described above. This doesn’t work in the example from the paper’s Figure 2, however. The problem is, there are two datastores replicating concurrently in Figure 2. Causal consistency is a only a partial order, not total; it allows the Post-Storage’s or the Notifier’s datastore to replicate first. With multiple replicated datastores, the anomaly is allowed by causal consistency, so we have to define a stricter consistency level.

Cross-service Causal Consistency

The Antipode authors define a new consistency level that prohibits the anomaly: “cross-service causal consistency”. They abbreviate it “XCY”, which perhaps makes sense in Portuguese. Cross-service causal consistency includes several ideas:

Lineage: A DAG of operations, partially ordered with Lamport’s “happens-before” relation. A lineage begins with some “start” operation, such as a client request or a cron job, and proceeds until each branch completes with a “stop” operation.

In Figure 2 there are two lineages: one is spawned when Author uploads a post. I’ll call this Lineage A. It has two branches (leading to Post-Storage and Notifier), and it includes concurrent replication of two datastores to Region B. The other lineage, which I’ll call Lineage B, is spawned when Follower-Notify in Region B receives the notification. Lineage B then reads from Post-Storage in Region B, and notifies Follower.

The authors use a data set from Alibaba, where lineages are hairy: “User requests typically form a tree, where more than 10% of stateless microservices fan out to at least five other services, and where the average call depth is greater than four. Additionally, this tree contains, on average, more than five stateful services,” i.e. services with datastores.

Reads-from-lineage: An operation b reads-from-lineage L if b reads a value written by an operation in L.

Cross-service causal order: This is denoted with the squiggly arrow ⤳. For two operations a and b, if a happens-before b or b reads-from-lineage L, where L includes a, then a ⤳ b. Cross-service causal order is a transitive partial order, like happens-before.

XCY: This is the paper’s new consistency level. An execution obeys XCY if you can find a serial order of operations obeying cross-service causal order.

XCY is the consistency level that Figure 2 violates! When Follower-Notify tries to read Author’s post in Region B, that should happen after all the events in Lineage A, including the replication of the post to Region B.

My Feelings About Lineages

I feel uncomfortable, as if there’s a purer mathematical concept obscured by the specifics of microservice architectures. Why are the borders between Lineages A and B drawn where they are? Could we split these operations into more than two lineages, or combine them into one?

I think that a lineage is a general concept (“a DAG of operations”), but Antipode finds it convenient for microservice architectures to split lineages thus: Operations in a lineage are connected by happens-before. When a service reads a value from storage, this operation does not join the lineage that wrote the value. Instead, it’s connected by reads-from-lineage. The goal of “cross-service causal consistency” is to make a partial order of lineages, such that replicated data stores appear not-replicated. (I was confused about this until I read the paper’s appendix. You should read the appendix, too.)

I think there’s a more general idea of “recursive” or “nested” causal consistency trying to be born. This general idea would include lineages, defined however you want, and lineages could contain nested lineages. Cross-service causal consistency is a specialization of this general idea.

Tracking And Enforcing Cross-Service Causal Consistency

This paper describes a system for enforcing XCY, called “Antipode”, which means “opposite side”; maybe this refers to end-to-end consistency guarantees across geographic regions. Or maybe it refers to mythical beings with reversed feet for some reason.

Anyway, whenever services exchange messages as part of the regular functioning of the application, Antipode piggybacks lineage information. Since microservice architectures already piggyback info for distributed tracing, Antipode doesn’t add much coding-time or runtime burden. Additionally, Antipode places shims in front of all datastores; the shims add lineage information to reads and writes. (Antipode borrows the technique from Bolt-on Causal Consistency.) Lineage info accumulates along causal branches within a lineage, and gets dropped whenever a branch ends.

Developers can customize lineage tracking; they can explicitly add or remove dependencies. If one lineage depends on another in a way that Antipode doesn’t detect, a developer can transfer lineage info between them.

(MongoDB drivers let you transfer causality info between sessions, too, although it’s basically undocumented; I explain it here.)

Antipode could enforce XCY automatically, on each read operation, but instead it provides an explicit barrier operation that developers must call to wait for dependencies to be satisfied. This seems error-prone, but it sometimes permits developers to reduce latency by carefully choosing where to place their barrier calls. The authors write,

One argument that can be made against barrier is that it is as explicit as today’s application-level solutions, since both of them require the developer to manually select its locations. What makes Antipode’s approach better suited is not only barrier, but its combination with the implicit/explicit dependency tracking, which keeps services loosely coupled and does not require end-to-end knowledge of what to enforce.

This bit about “loose coupling” is insightful: you can place your barrier call somewhere, and if you later add dependencies, barrier will enforce them without code changes. On the other hand, having one barrier call for all dependencies requires you to wait for all of them at once, including those you don’t need yet.

How does barrier know how long to wait?

Antipode’s barrier API call enforces the visibility of a lineage. It takes a lineage as an argument and will block until all writes contained in the lineage are visible in the underlying datastores. Internally, a barrier will inspect the write identifiers in the lineage and contact the corresponding datastores. For each datastore, barrier will call the datastore-specific wait API, which will block until the write identifier is visible in that datastore. Note that wait is datastore-specific because visibility depends on the design choices and consistency model of the underlying datastore. Once wait has returned for all identifiers in the lineage, barrier will return.

In our example, this means that before Follower-Notify retrieves Author’s post from Post-Storage, it calls barrier, which queries the Post-Storage datastore and waits until it’s sufficiently up-to-date.

This is an extra round trip (red arrow) even if the datastore is already up-to-date. I think this could be optimized away with something like MongoDB’s afterClusterTime, but Antipode’s API would have to change. Luckily, you can limit the consistency check to nearby replicas:

We implemented a practical optimization strategy specifically tailored for geo-replicated datastores. This involves implementing the wait procedure to enforce dependencies only from replicas that are co-located with its caller, thereby avoiding (whenever the underlying datastore allows it) global enforcement.

I don’t fully understand barrier from the paper’s description. If it’s waiting for all writes from Lineage A to be visible in Region B, how does it know about writes that Lineage A hasn’t even started yet? Must it wait for all branches of Lineage A to finish? If so, how? And what if an operation in Lineage A crashes or hangs?

Their Evaluation

The authors evaluate Antipode with three benchmarks and a dozen brands of datastore, and ask 1) would there be XCY violations without Antipode, and 2) what is the cost of preventing them? The answer to question 1 is yes. For question 2:

Lineage info adds 200 bytes to 14 kb per message in the authors’ benchmarks (developers might need to explicitly prune lineages in their own systems).
Waiting for consistency increases latency, by definition.
Enforcing XCY decreases throughput by 2-15%.

My Evaluation

Cross-service causal consistency is a neat concept. The chief argument for it, buried in the middle of the paper, is decoupling: it permits microservices to read consistently from multiple replicated data stores, without knowing the details of the microservices that wrote to them. This limits the impact of changes to any part of your system. With Death Star architectures like Alibaba or AWS, decoupling is crucial.

The AWS microservices “death star” in 2008

Antipode’s API is higher-level than manually enforcing cross-service dependencies. I think it could be useful as part of an even higher-level “cloud programming language” that automatically decomposes, distributes, and parallelizes high-level logic, while detecting consistency requirements and enforcing them. I’m aware of cloud programming language projects like Unison, Wing, Goblins, Hydro, Dedalus, Gallifrey, and so on. They’re at various stages of development and levels of abstraction. If this paper’s definition of lineage were generalized to encompass more kinds of causal relations among operations, it could express the constraints of a variety of constructs in high-level cloud programming languages, and something like Antipode could enforce them.

Review: Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 25 Jan 2024 11:35:03 +0000

Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency, Cary G. Gray and David R. Cheriton, 1989. Old and good. I read this simple paper because it seems to be ground zero for timed leases in distributed systems, in which I’m now intensely interested.

The Protocol

The authors discuss a distributed file system, e.g. for a network of diskless workstations connected to a shared file server. (The file server itself is a single machine, not replicated. Yes, it’s a single point of failure. This is 1989.) The workstations can read or write files by exchanging messages with the server.

Each workstation is a write-through cache: when it reads a file, it caches it in its local RAM for future reads.

When a workstation updates files in its RAM, it synchronously updates the server’s copy.

Of course, all sorts of race conditions can cause inconsistency. Workstation A could read and cache a file, then workstation B updates it, then workstation A re-reads the file from its now-stale cache and sees outdated data.

The authors want to provide consistency: “By consistent, we mean that the behaviour is equivalent to there being only a single (uncached) copy of the data except for the performance benefit of the cache.” Their solution is a timed lease.

A cache using leases requires a valid lease on the datum (in addition to holding the datum) before it returns the datum in response to a read, or modifies the datum in response to a write. When a datum is fetched from the server (the primary storage site of the datum), the server also returns a lease guaranteeing that the data will not be written by any client during the lease term unless the server first obtains the approval of this leaseholder. If the datum is read again within the term of the lease (and the datum is still in the cache), the cache provides immediate access to the datum without communicating with the server. After the lease expires, a read of the datum requires that the cache first extend the lease on the datum, updating the cache if the datum has been modified since the lease expired. When a client writes a datum, the server must defer the request until each leaseholder has granted approval or the term of its lease has expired.

So before Workstation B can write to a file, it must acquire from the server a lease on that file, with a timeout that will expire some time in the future. Workstation B completes its writes while holding the lease, and either extends the lease or allows it to expire. When Workstation A then re-reads the file from its cache, it must get a new lease on it, thus discovering that the file has changed, and refresh its copy from the server. After this, Workstation A can keep reading the file until its lease expires. Meanwhile, no other workstation can modify it.

The server itself can read and write files, and it needs a lease to do so, the same as a workstation.

It seems that the system grants multiple shared leases for reading, or one exclusive lease for writing. The authors don’t say so, but they imply it. E.g., the passage above mentions “each leaseholder”, indicating there could be more than one per file. These days a paper like this would crush all ambiguity with pseudocode, a TLA+ specification, a formal proof, and a link to an open-source implementation on GitHub. I guess in 1989 you could mumble a few pages about leases and the ACM would publish it.

If a workstation or the server wants access to a file that’s already leased to another workstation, the lease-wanter can wait, or ask the leaseholder to relinquish the lease early. A leaseholder can proactively relinquish an unexpired lease that it doesn’t need anymore.

If many nodes want a lease on the same file, the server enqueues them. The paper doesn’t specify the scheduling policy, except to mention in a footnote that writers take priority.

Fault Tolerance

If a workstation gets disconnected from the other nodes, it might have crashed, or it might be partitioned and still reading from its cache any files for which it has valid leases. Thus to guarantee consistency, the server must wait for the disconnected workstation’s leases to expire before it can grant more on the same files. Since the workstation is practically stateless, crash recovery is trivial.

If the server crashes and restarts, it must remember all the leases it granted before. It could durably record each lease it grants, but that might make disk I/O a bottleneck. The authors propose that the server durably records only the maximum expiration time. When it restarts, it waits for that maximum expiration to pass before granting new leases.

The protocol is resilient to delayed or lost messages, but it does require all nodes’ clocks to advance at the same speed, modulo a small, known epsilon.

Optimal Lease Terms

Short leases minimize recovery time after a workstation or server failure. They also minimize “false sharing”: when a node has to wait for a lease, although the leaseholder is no longer using it. Long leases reduce lease-requesting traffic and latency at the workstations.

The large portion of the paper constructs an analytical model of these tradeoffs. I’ll ignore this section since it’s fairly specific to the authors’ system. Plus I’m generally skeptical of analytic performance modeling, compared to simulation.

There’s an interesting discussion, however, of the optimal lease terms for different sorts of files and workstations. The server should grant longer leases to more distant workstations to compensate for network latency. Operating system files are very frequently read and almost never written, so the server should grant very long read leases on them. In fact, it should just grant bulk read leases on whole directories of OS files, and proactively issue read lease extensions to all workstations so long as no write to the OS files is pending. On the other hand, the server should grant short leases on a frequently-written file. The authors say “a heavily write-shared file might be given a lease term of zero”, which I don’t understand—how can a workstation use a lease that’s already expired by the time the workstation knows it has acquired it? Anyway, the authors conclude, “a server can dynamically pick lease terms on a per file and per client cache basis using the analytic model, assuming the necessary performance parameters am monitored by the server.”

Their Evaluation

The authors evaluate performance not with a real-life test, but by applying their analytic model to some real-world data. They use a trace of file accesses from one workstation to one file server, while the workstation recompiled a program. There’s no contention in the trace (since there’s one workstation), so they simulate various levels of contention in their evaluation. Network latencies were measured in separate tests, then fed to the model. This is so many steps removed from reality, it wouldn’t fly in a modern research journal. However, I do appreciate this abstracted approach; it can be more revealing than testing a fully-implemented system with its adventitious complexity and noise.

The authors measure server load (i.e., the number of messages it must process) for different numbers of workstations (the “sharing level”, S) and different lease terms. They set the server load to 1 with 1 workstation and zero-term leases, and measure other configurations relative to that one. The workload is 96% reads, so longer terms reduce server load because workstations can mostly read from cache. I’d be curious to see latency at the workstations, too. I’d expect longer terms to decrease latency in this read-heavy workload, and increase it in a write-heavy workload.

My Evaluation

A short and sweet paper, worth reading for historical interest. If I publish a paper about leases I’ll need to know what’s in this one so I can cite it. I wish they had described their protocol more precisely, perhaps in pseudocode, before they rushed to model it analytically and draw charts. But all is forgiven: this is the paper that coined the term “lease” and introduced the world to an elegant consistency technique.

Review: Exploiting a Natural Network Effect for Scalable, Finegrained Clock Synchronization

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 22 Jan 2024 16:30:27 +0000

Christiaan Huygens by Caspar Netscher, 1671.

This is a review of Exploiting a Natural Network Effect for Scalable, Finegrained Clock Synchronization, from Stanford and Google researchers. It describes Huygens, a protocol for very accurate clock synchronization, plus (an undersold benefit) very accurate measurements of one-way network latency. I read it because the Huygens protocol is used by Nezha, which I reviewed last week. The Huygens and Nezha papers share two authors.

Motivation

As Barbara Liskov wrote in 1991, there are practical uses of clocks in distributed systems, such as establishing an order of events on different servers without communication. In an especially insightful paragraph, the authors write:

In order to achieve external consistency, a write-transaction in Spanner has to wait out the clock uncertainty period, T, before releasing locks on the relevant records and committing. Spanner can afford this wait time because T is comparable to the delay of the two-phase-commit protocol across globally distributed data centers. However, for databases used by real-time, single data center applications, the millisecond-level clock uncertainty would fundamentally limit the database’s write latency, throughput and performance. Thus, if a low latency database, for example, RAMCloud, were to provide external consistency by relying on clock synchronization, it would be critical for T to be in the order of 10s of nanoseconds so as not degrade the performance.

NTP, the usual clock-sync protocol, is only accurate to a few milliseconds. More accurate protocols require specialized hardware. The Huygens protocol (“HOY-gons”, named for the inventor of the pendulum clock) gives nanosecond accuracy in ordinary data centers. The authors summarize it thus: “First, coded probes identify and reject impure probe data (data captured by probes which suffer queuing delays, random jitter, and NIC timestamp noise). Next, Huygens processes the purified data with Support Vector Machines, a widely-used and powerful classifier, to accurately estimate one-way propagation times and achieve clock synchronization to within 100 nanoseconds. Finally, Huygens exploits a natural network effect (the idea that a group of pair-wise synchronized clocks must be transitively synchronized) to detect and correct synchronization errors even further.”

The Buddy System

Clock-sync protocols like NTP estimate network round-trip time (RTT) thus: server A sends a message called a “probe” to server B, which responds with an “ack”. Both messages are timestamped by the sending and receiving network interface cards (NICs). Server A averages the durations between probes and acks to estimate RTT. But some messages experience random queueing delays. Huygens wants to find probe-ack pairs that passed between the servers in minimum time without queueing delays; the authors call these “pure” probes. They use only pure probes for RTT and clock-skew estimation.

Huygens distinguishes pure and impure probes with a crafty little algorithm: Server A sends a probe, waits a small time s according to A’s clock, then sends a second probe. (For some reason these probe pairs are called “coded” probes.) If the time between the probes on the receiving server B is very close to s, Huygens calls both probes pure. (The duration s is small enough that the two servers’ differential clock drift is irrelevant.)

The Forbidden Zone

Huygens uses pure probes to calculate the largest and smallest possible clock discrepancies between servers. For example, if server A sends a probe at time 0 according to A’s clock, and B receives the probe at time 5 according to B’s clock, B’s clock can’t be more than 5 units ahead of A’s: otherwise the message would’ve arrived before it was sent, in absolute time. Thus a message from A to B reveals the upper bound of B’s clock skew relative to A’s.

A message going the opposite direction reveals the lower bound.

Any of these numbers could be negative: if the message leaves from A at time 0 and arrives at B at time -5, B’s clock can’t be more than -5 units ahead of A’s (i.e. it must be at least 5 units behind).

Even “pure” probes exhibit some random variation in latency. Quicker probes give tighter bounds: if the message leaves at time 0 and arrives at time 4, we’d know B’s clock can’t be more than 4 units ahead of A’s. But we don’t know how much of that difference is clock skew and how much is network latency—not yet.

Over the course of seconds, as the servers’ clocks drift towards or away from each other, the bounds change, producing this difficult and delightful chart:

Figure 4 from the paper.

The quickest messages give the tightest bounds, visible as dense blue and green lines. Slowdowns seem to be randomly distributed, making sparse fields of blue dots above the least upper bound, and green dots below the greatest lower bound.

The handful of dots between the bounds are in the “forbidden zone”. They seem to be unusually quick messages that provide tighter bounds than the other dots, but it’s a lie. In fact, they’re an artifact of NIC timestamp noise: when a server sends a message, there’s an occasional delay of 10s or 100s of nanoseconds before it records the transmit timestamp, making the transmission time seem shorter than it is. This is small enough that these probes are considered “pure”, but large enough to hurt Huygens’s accuracy. Huygens uses a very well-known statistical method called a support vector machine to find the dense lines that border the forbidden zone and filter out the samples inside it.

Detecting Asymmetric Delays

Other protocols like NTP assume symmetric network delays: they assume the one-way delay (OWD) from server A to server B is practically the same as vice versa. The Huygens authors find that this is mostly true in their data centers, but slight asymmetries violate this assumption enough to hurt clock synchronization. They exploit a natural network effect (hence the paper’s title): additional servers help detect asymmetries. For example (from the paper), say that servers A and B think that A’s clock is 20 units ahead of B’s, but due to asymmetry they’re wrong, A’s clock is only 10 units ahead of B’s:

If these were the only servers, the error would be undetectable. But say there’s a server C that thinks it’s 15 units behind B and 5 units ahead of A:

This is impossible, because summing the offsets A→B→C→A we find a “loop offset surplus” of 10 units (20 + -15 + 5). Huygens evenly distributes the surplus among the pairwise offsets, improving accuracy. In this example, the offset from A to B is now estimated to be 16.7; not the accurate value of 10, but better than the wrong value of 20.

In a real data center, “each server probes 10-20 others, regardless of the total number of servers in the network”, and Huygens analyzes the entire graph of pairwise synchronized clocks.

Their Evaluation

The authors built two testbeds, each with several racks and switches, and dozens of servers. One testbed represents, according to the authors, a “state-of-art data center”, and the other a “low-end commodity data center”. The two testbeds have different timestamping features in their NICs, and different network latencies.

I was wondering how they’d evaluate clock synchronization—after all, the problem they’re trying to solve is the lack of a perfect clock to compare to. They evaluate pairwise synchronization with a NetFPGA-CML board, which has four ethernet ports and one clock. They attach two separate VMs to two of the ports, and connect them to the rest of the servers in the low-end testbed. The two VMs are banned from talking directly to each other, they must sync their clocks via intermediate servers. Since the actual clock discrepancy between these VMs is zero, the authors can measure how closely Huygens has synchronized their clocks. They get a mean error of 13.4 ns and a 99th-percentile error of 30.2 ns; this is tiny, and the low variance is especially impressive.

There are additional evaluations that show that:

If each server probes K other servers, clock error falls as K grows.
Network load hurts synchronization, but not too much: a few probes can still get through without delay, and Huygens correctly identifies these as “pure” and uses them for clock sync.
Huygens is orders of magnitude better than NTP.

My Evaluation

The results in this paper seem exceptionally useful for distributed systems. Huygens appears to achieve incredibly tight clock synchronization, plus two features that the paper undersells: Huygens also measures the clock error bounds, and the one-way delay between servers. These three measurements could enable countless futuristic distributed protocols. Furthermore, the explanations in the paper are thorough and clear. The only part I found intimidating was their description of the loop-analysis algorithm, but I think that’s inherently complex.

Unfortunately, I can’t tell from the paper whether Huygens works in public clouds. Do customers like me have the required access to NIC timestamps? Do we need it? If Huygens works in public clouds, I wish this paper had evaluated it there. The Huygens code is closed-source, so third parties can’t test it themselves.

Several of the authors founded Clockwork, which “runs on all major clouds”, synchronizes clocks, and measures clock error bounds and one-way delays. Clockwork claims “sync accuracy as low as 5 – 10 ns with hardware timestamps, 100s of ns – a few μs with software timestamps.” It looks to me like they ported Huygens to public clouds using software timestamps, and its accuracy there is reduced but still great.

Review: Nezha: Deployable and High-Performance Consensus Using Synchronized Clocks

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 14 Jan 2024 08:25:53 +0000

This is a review of Nezha: Deployable and High-Performance Consensus Using Synchronized Clocks, from NYU and Stanford researchers last year. To understand this paper I had to relearn how quorums work in Paxos and Fast Paxos, so we’ll start there.

Classic Paxos Quorums

In Paxos there are proposers, acceptors, and learners. Each server usually plays all three roles, but the protocol’s often described as if the roles are separate (confusingly, in my opinion). The protocol is like:

A client sends value v to a proposer.
The proposer sends “prepare” to the acceptors, with a unique current round number.
The proposer hears “promise” replies from a majority of acceptors.
The proposer knows that only it can propose a value for this round.
The proposer sends “accept” to the acceptors with the value v for this round.
A learner hears “learn” messages from a majority of acceptors, it knows v is the value for this round.
The learner sends an acknowledgment to the client.

Real systems use MultiPaxos, in which the “prepare”/“promise” exchange is done once to establish a “distinguished proposer”, which then drives a series of “accept”/“learn” exchanges indefinitely. Optimized MultiPaxos is nearly the same as Raft, it’s just explained much worse.

Fast Paxos Quorums

Before any proposals, a proposer sends “accept any” to the acceptors with no value for this round.
The proposer hears “promise” replies from a majority of acceptors.
Fast track: the client sends value v directly to the acceptors!
A learner hears “learn” messages from 3/4 of the acceptors, with the same value.
The learner sends an acknowledgment to the client.

Fast Paxos saves a single one-way message delay compared to regular Paxos. But it risks collisions: several clients can send several values to the acceptors in the same round. Fast Paxos uses a larger “fast quorum” size to check for collisions. If no value is chosen by a fast quorum, the system goes into a slow error-recovery mode. Fast Paxos is safe—it will never accept multiple values in a round—but its performance is brittle; it’s slow under contention.

Quorum Sizes

A classic Paxos quorum must be a majority, so that any two quorums Q₀ and Q₁ share at least one node. For example with 9 acceptors, a quorum is at least 5:

Thus if a minority of acceptors fails, at least one survivor remembers accepting v. (If a majority of acceptors fails, the system won’t accept more values.)

The rule for fast quorums is, any classic quorum Q must share at least one node with any two fast quorums R₀ and R₁. With 9 acceptors, a classic quorum is at least 5 as before, and a fast quorum is at least 7:

If a fast quorum accepts v and then a minority of acceptors fails, at least one survivor remembers v, and no other value could’ve been accepted by a fast or classic quorum. In classic Paxos, “no other value” is guaranteed by the “prepare” phase, where one proposer hears a majority promise to stop accepting other proposers’ values with earlier round numbers. But in Fast Paxos all the failed acceptors could’ve accepted some other value. We need a bigger fast quorum to know that we’ve chosen a unique fault-tolerant value.

There are various quorum sizes that satisfy this property, and there are tradeoffs when you choose a quorum size. See Lamport’s Fast Paxos paper for details.

Nezha

Now I can describe the Nezha paper. As author Jinkun Geng mentions in a podcast interview, Nezha is a Chinese diety with three heads and six arms, “so he has wonderful fault-tolerance.”

Nezha bridges the gap between protocols such as MultiPaxos and Raft, which can be readily deployed, and protocols such as NOPaxos and Speculative Paxos, that provide better performance, but require access to technologies such as programmable switches and in-network prioritization, which cloud tenants do not have.

So the authors’ motivation is to make a high-performance consensus protocol that can be deployed by cloud customers in public clouds, without requiring special access to the hardware. Nezha improves performance using tightly-synchronized clocks, which increasingly are available to cloud tenants, especially on AWS.

Nezha is like Fast Paxos plus Deadline-Ordered Multicast, speculative execution, and a stateless proxy.

Deadline-Ordered Multicast (DOM)

Just like Fast Paxos, Nezha has a fast path and a slow path, and it’s crucial to take the fast path as often as possible. The authors say that message reordering is the most common reason for taking the slow path: a sequence of messages from the proxy take different network paths to the same server and arrive out of order. DOM reduces reordering thus:

The sender attaches a deadline to each message: the sender’s clock time + one-way-latency estimate.
The receiver rejects any message received after its deadline.
The receiver executes each message after its deadline passes (according to the receiver’s clock).
The receiver executes messages in deadline order.

The authors evaluate DOM in Google Cloud:

The reordering score is the percent of messages in a sequence that are not in the longest ordered subsequence. E.g., if there are 100 messages and the longest ordered subsequence is 80 messages long, the reordering score is 20. Higher is worse. Without DOM, they found in Google’s cloud, under high contention, that the reordering score was over 40%.

DOM estimates the distribution of one-way latencies, using a technique called Huygens from another paper. Once it knows the distribution of one-way latencies, it knows the 50th-percentile one-way latency, the 75th, and so on. So the “50p” bar in the chart above means the authors configured DOM to set a deadline that was the sender’s clock plus the 50th-percentile one-way-latency. A longer delay further reduces reordering, but the receiver waits longer on average before executing each message. So the delay is a tunable parameter; there are tradeoffs and some optimum that you have to find.

I was surprised to read that the Nezha authors choose the 50th percentile. This means half of messages arrive after their deadlines! However, not all late messages force Nezha to take the slow path, only messages that are late and out of order. If a sequence of messages are all late, they can still be ordered. As the chart indicates, configuring DOM so that half of messages are late reduces the reordering score to barely 20%.

Speculative Execution

In regular Paxos, servers don’t execute a client’s command (they don’t update their state machines) until they know the command has been logged by a quorum. But the Nezha leader executes and acknowledges a command as soon as its deadline passes. The client accepts the execution result once it hears confirmation from a fast quorum. This reduces latency. If the leader is deposed before committing the command, the client rejects the result and retries the command.

Stateless Proxy

Nezha includes a proxy that encapsulates some Nezha logic. The proxy runs the Huygens protocol to estimate one-way latency and to tightly synchronize its clock with the other proxies and the servers. The proxy is basically stateless and horizontally scalable; it isn’t responsible for any ordering guarantees. Smart proxies permit dumb clients.

The Nezha Protocol

Fast Path

First a request comes from a client.

The proxy assigns a deadline from the proxy’s reasonably-synchronized clock, plus a delay that’s a percentile of the one-way-latency estimate. The proxy sends the message to all the servers.
The leader and followers get the message, and if its deadline hasn’t passed, the message goes in their “early” buffers to wait. This is the fast path, if the message is late we take the slow path, which I’ll describe later.
Soon after the deadline, each server removes the message from the early buffer and logs it. Servers process messages in deadline order.
The leader executes the command…
…and returns the result to the proxy. The followers send acknowledgments to the proxy without any result, because they don’t have state machines and they don’t execute commands, they only have logs. (So how can a follower become a leader? Read the paper.)

Each acknowledgment includes a hash of the whole log, so the proxy knows whether all the servers in the quorum have the same log. We know the messages are ordered correctly by timestamp, but some servers could be missing messages.

If the proxy hears a fast quorum of replies with the same hash, including from the leader, it accepts the result. It knows it’s durable.

So that’s the fast path. It saves some latency, because the proxy sends the message to all servers at once, and all servers respond directly to the proxy, instead of routing messages through the leader like in MultiPaxos or Raft. It might incur some latency though, if you find that you have to configure the DOM delay to a high percentile.

What about the slow path? Let’s look especially at messages that arrive too late.

Slow Path

A request comes from a client again.

The proxy assigns a deadline and sends it to the servers, the same as before.
This time the message arrives late and goes to the late buffer.
The leader eventually modifies the message’s deadline, from a past time to a future time, and puts it in the early buffer!
Once the new deadline passes, the leader logs…
…and executes the message…
… and sends the result to the proxy.
Concurrently, the leader also sends the message’s ID and its new deadline…
…to the followers. Note how this is slower than the fast path: it requires leader-follower communication, unlike the fast path.
Luckily the followers don’t need the whole message, they have it in their late buffers, so they just retrieve it from there and log it in the proper position with its new deadline. If a follower didn’t receive the message at all, then it has to fetch it from another server, which is even slower.
Finally the followers acknowledge the message…
…and the proxy hears from a majority (a slow quorum) of servers, all with the same hash, and accepts the result.

The authors claim that the slow path is still faster than some competing protocols like MultiPaxos, because of speculative execution at the leader. They say the slow path is only one message delay slower than the fast path.

Of course, a message could arrive before its deadline on some servers, and late on other servers. There are worse cases, if messages are dropped between the proxy and the servers, or between the leader and the followers, or if a replica fails and rejoins, or there’s a new leader. The paper handles these scenarios and I will not.

I’ll summarize everything so far: Typical consensus protocols route through the leader to guarantee ordering, but this costs some network hops and makes the leader a bottleneck. Nezha uses Deadline-Ordered Multicast and large quorums to guarantee ordering, so it can parallelize more.

Commutativity Optimization

Messages are commutative if they contain commands operating on different keys. Nezha relaxes the rule for a message to enter the early buffer.

Naïve rule: the message’s deadline must be after the last message released from the early buffer.
Optimized rule: its deadline must be after the last non-commutative released message.

I see how this preserves per-key linearizability, but I think it violates whole-database linearizability, also known as strict serializability. That’s ok, I believe Nezha only promises per-key linearizability, and it’s the right choice for many users.

Their Evaluation

The authors ran experiments in Google Cloud with various configurations. I’ll concentrate on the open-loop workload with 3 replicas, 5 proxies, and 10 clients. The Huygens clock sync protocol is installed on the proxies and servers, it gets a p99 clock error of only 50 ns! They use a minimal application that processes messages with no command logic. They have Poisson arrivals, 50% reads/writes, and a somewhat skewed access pattern with some hot keys and some cold ones.

As expected, Nezha outperforms everything, because this is a paper about Nezha. But this is an even more dramatic chart than usual in evaluation sections.

NOPaxos (“Network-Order Paxos”) might be at a disadvantage here, because NOPaxos wants low-level access to the network and they don’t have it in Google’s public cloud. The Nezha authors say “we use the implementation from the NOPaxos repository with necessary modification: we change switch multicast into multiple unicasts because switch multicast is unavailable in cloud. We use a software sequencer with multi-threading for NOPaxos because tenant-programmable switches are not yet available in cloud.” NOPaxos-Optim is their enhancement of the published NOPaxos code; they relieved a bottleneck with multithreading. The authors also wrote a technical report with more benchmarks; NOPaxos-Optim outperforms Nezha in one test there.

To measure latency, we use median latency because it is more robust to heavy tails. We attempted to measure tail latency at the 99th and 99.9th percentile. But we find it hard to reliably measure these tails because tail latencies within a cloud zone can exceed a millisecond.

I want to see Nezha’s tail latency, and I don’t understand this explanation for omitting it. Cloud network latencies are indeed unpredictable, which is a big risk for a protocol like Nezha, which is optimistic and explicitly designed for public clouds. The authors’ justification sounds to me like, “This problem is so bad we decided to ignore it.”

The paper includes an “ablation study”, a neat bit of jargon for studying the effects of removing optimizations individually.

Deadline-Ordered Multicast is obviously important. Without it (red line), Nezha is practically the same protocol as MultiPaxos and performs about the same: it’s usually on the slow path due to message reordering. “Quorum-Check Offloading” to the proxy or client is important, because it removes some work from the leader, which is otherwise a bottleneck (gray). It seems that the leader’s CPU is saturated and quorum-checking contributes to that. Commutativity is apparently not important for throughput and it only slightly improves latency (purple vs. green). Perhaps that’s because DOM is so good at message ordering that commutativity doesn’t help much, or perhaps it’s because their workload skewness means many messages are non-commutative?

My Evaluation

The paper is written for users like me and MongoDB Atlas: public cloud users without special hardware or network access. I appreciate this focus.
I worry about performance variability in an optimistic protocol with fast and slow paths. How much does Nezha suffer when there’s contention and/or clock skew?
The authors write, “Nezha does not assume the existence of a worst-case clock error bound”, but that’s just for safety. Performance does rely on tightly synchronized clocks. Ever since Metastable Failures in Distributed Systems, I dislike optimizations that work sometimes. Therefore I wish they’d benchmarked tail latency, not just median latency, and tested more adverse conditions like workload skewness, clock skew, and network latency variability.
Deadline-Ordered Multicast is slick, and useful. If it’s tuned, it should be nearly free. But how does DOM fare when latencies to different nodes differ?
The commutativity optimization is clever, although its usefulness is overshadowed by DOM here.
Some of my colleagues were annoyed that the Nezha authors don’t credit the Tempo paper and other precedents.
Synced clocks in public clouds are real now. We can use them in distributed protocols. This is a superb example.

I learned a lot from this paper, especially since I presented it to the DistSys Reading Group and wrote this review. I want to see more of this kind of research: the use of public cloud features for distributed protocols.

Sojourner

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 09 Jan 2024 15:18:24 +0000

January 1, 2024. Our young Chinese dwarf hamster Sojourner Truth Armstrong Davis.

Bonticou Crag and Table Rocks

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 07 Jan 2024 12:42:40 +0000

November 25, 2023. Hiking the Bonticou Crag and Table Rocks loop in the Gunks, upstate New York. Portra 400 with my Mamiya 7.

Dickie Barre

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 07 Jan 2024 10:35:53 +0000

December 1, 2023. Climbing with Jennifer in the Dickie Barre area of Peter’s Kill, the Gunks. Portra 400, Mamiya 7.

Village Zendo's Winter 2023 Meditation Retreat

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 06 Jan 2024 19:15:04 +0000

December 2023. The Village Zendo’s annual year-end Zen retreat, at the Garrison Institute (a former Catholic monastery) on the Hudson River. This year I was a monitor of the meditation hall, responsible for everyone staying awake, sitting up straight and silent and still, finding a comfortable posture, and keeping to the schedule. We had a lot of inexperienced meditators who were clearly in pain after the first day. I met with them and discussed options for sitting on a chair, or a seiza bench, or a cushion, and how to position their backs and shoulders. The meditation hall settled down noticeably for the rest of the week.

I photographed some meditation sessions on Portra 400 with my Mamiya 7. The room was dark and the photos generally underexposed, so the colors are muddy in several of these. Next year I’ll bring ISO 800 film and a tripod.

Java Modelling Tools

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 04 Jan 2024 18:55:38 +0000

In 2022, Marc Brooker argued that formal methods like TLA+ can check distributed systems’ correctness but not their performance. Since then, I’ve been searching for good performance modeling tools. Queue theory seems like a foundation for performance modeling, so I learned some queue theory, although I read the wrong book. That book tried to teach me to analyze queue networks by solving intricate equations, but for most queue networks the equations can’t be solved, and for the rest I can’t remember how to solve them. I concluded that equations aren’t practical for me, and simulation is the right method. I looked for an off-the-shelf queue network simulator, and found the Java Modelling Tools.

JMT is from Politecnico di Milano and Imperial College London. It was begun in 2002 and it’s still actively developed; the last release was November 2023. The two main developers, Giuliano Casale and Giuseppe Serazzi, have written and maintained a thorough user manual, and when I asked a question on the project forum last year they both responded quickly and in detail.

JMT is a suite of Java applications:

JSIMwiz: Wizard interface for JSIM, a discrete event simulator, for modeling a queue network.
JSIMgraph: Same, but you draw the queue network.
JMVA: Exact and approximate analysis of queue networks with restricted features.
JMCH: Simulates a single node, animates the Markov chain.
JABA: Find bottlenecks in closed queue networks.
JWAT: Reads log files, clusters customers into workload classes.

I played with JMCH briefly and JSIMgraph for a couple days.

JMCH

JMCH displays the states and transition probabilities for a single-node queue network, represented as a Markov chain. It seems to be for teaching how Markov chains relate to queue theory.

This is a simple M/M/1 queue. The diagram in the middle is a Markov chain, where each node represents a state with a certain queue length: e.g., if the system is in state 3, that means there are 3 customers enqueued. Each state is labeled with its long-term probability. So in the long run, the system spends half its time in state 0 (empty queue). You can watch the blue queue grow and shrink at the bottom, and the system transition from state to state, as random events occur. If I’d had this available when I started reading my queue theory book it might’ve helped me understand Markov chain analysis.

JSIMgraph

This is the tool I want. I can draw a queue network, with nodes and sources and sinks connected by directed edges, and simulate the system, watching the animation and observing average values. Here’s an example included with JMT:

A network needs one or more classes of customers (or jobs or tasks or whatever). Classes were a niche topic in the queue theory book because we usually assumed one class, but classes are a big deal in JMT.

Every class needs a reference station. I have read the manual’s explanation of reference stations five times and I don’t understand, perhaps I need a PhD.

JMT offers a huge number of probability distributions for arrivals, service times, etc. You can choose the “Replayer” distribution which reads values from a file; that could be a trace from your real system or numbers generated by another program.

The goal of running the simulation is to measure some performance indices. Here the indices are the response time for the whole system, and the utilization of each disk.

Let’s run the simulation and see those performance indices’ average values:

By default, JSIM runs until each performance index has converged to a stable value within some confidence interval (the red lines converge around the blue lines).

When I showed this to some friends they wondered, “What if the Join0 node waited for a majority of disks instead of all of them?” This was satisfyingly easy to answer. I changed the “join strategy” from “standard” to “quorum”:

As expected, re-running the simulation shows the same disk utilization but much faster response time: 19 seconds average instead of 34.

Answering a queue theory question

Let’s take a queue theory problem from the book “Performance Modeling and Design of Computer Systems” and contrast the analytic approach to simulation. Here’s a figure from the book:

The book’s Exercise 2.1 asks,

Maximum Outside Arrival Rate
For the network-of-queues with probabilistic routing given in Figure 2.3, suppose that each server serves at an average rate of 10 jobs/sec; that is, μ_i = 10, ∀i. Suppose that r₂ = r₃ = 1. Suppose that p₁₂ = p_2,out = 0.8, p₂₃ = p₁₃ = 0.2, p₁,out = 0, and p₃₁ = 1. What is the maximum allowable value of r₁ to keep this system stable?

Let’s add those parameters to the figure:

To answer the question, “what’s the max r₁”, or “how fast can jobs arrive at Server 1 without overloading the system?”, the book taught me to solve a system of simultaneous equations. I arrived at r₁ ≤ 5.2.

I also wrote a 72-line Python simulation (using just the standard library) to confirm this number.

Now that I’ve forgotten how to answer this question with paper and pencil, can I use JMT instead? I drew this uglitude in JSIMgraph:

I tried to emulate the book’s figure, with a “source” node for r₁, r₂, and r₃, but this produced strange behavior. Maybe I need to understand “reference stations”. Instead I made three classes of customer (each with a different arrival rate), generated them all at the same source, forked them to the three servers, and collected them in a “sink” node.

I set Class 1 to 10 arrivals per second, which we know is too high; it should be at most 5.2. The others are 1 per second as in Exercise 2.1. Since we have 3 classes, we have to configure how each server handles each class. First I set the service times to 0.1 for each class (since the exercise says they serve 10 jobs per second), then the routing rules.

Configuring a queue network is a quickly-growing chore; I have to point and click and enter a number for all classes multiplied by servers multiplied by routes. Worse, there’s no single place to see all the parameters and verify them. I have to double-click each node to open its config dialog, and click among the tabs checking each number.

To find the maximum r₁ value I configure a “What-If” analysis, trying a range of values:

JMT’s simulator assumes by default that the system is stable and its performance indices will converge. In this case, if r₁ is too big the system is unstable and JMT runs until it’s out of memory. So I configure the simulation to run for a limited time:

What performance indices will tell me whether the system is stable? If queues grow or a server is fully utilized, those are bad signs, so I’ll measure queueing time and utilization.

Finally, I actually run the analysis:

As I had hoped, between the tested r₁ values of 5.1 and 6.1, things go haywire. Queueing time takes off and Server 1’s utilization hits 100%. Configuring this was a lot of work, but it’s gratifying to see a clear result. JMT lets me easily measure more performance indices, or tweak the model and see the effects, more easily than in Python and much more easily than paper and pencil.

My evaluation

Gripes:

JSIMgraph requires a bunch of pointing, clicking, and typing to set up a queue network with a few classes and nodes. The Python interface looks like a slick way to construct really big networks, but medium networks are a chore.
Since JSIMgraph expects stable systems by default, doing “what-if” analysis to determine whether a set of params is stable is tricky.
There’s no single place to see all configuration values, so I have difficulty trusting that I’ve set up a network correctly.

But in JMT’s favor, if your needs fit its features, it’s more convenient than writing a Python simulation, and it overcomes the total impracticality of solving equations. I may play with it some more to learn its features better, and try answering some actual day-job questions about distributed systems.

West Trapps Bouldering

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 02 Nov 2023 11:50:27 +0000

October 11, 2023. Boulderers in the West Trapps area of the Gunks. Shot with a Mamiya 7 on Portra 400.