Programming on A. Jesse Jiryu Davis

This Senior Staff Engineer Vibe-Coded for the First Time, What Happened Next Will Shock You

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 19 Nov 2025 02:53:40 +0000

Actually it won’t shock you. Spoiler alert: Coding with AI is a skill. It’s annoying but efficient. My prior experience handwriting code helped.

By Gemini, obviously. This is the first and only post I’ll ensloppify with AI “art.”

I started programming seriously when I was in college, in the late 90s. Basic autocomplete in Microsoft Visual C++ was the state of the art back then. I spent the next couple decades meticulously laying out my code like a 16th-Century typesetter, crafting each line with an artisan’s pride.

How My Blog Handles Math and Images in HTML, Atom, and Email in 2025

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 31 Aug 2025 08:43:59 +0000

I want to explain computer science on my blog, and show photos, and I want every article to look exquisite. My goal is to display math and images (especially SVGs) as beautifully as possible for people who read my articles on my site, and via an Atom feed and email. This is hard with today’s technology, plus I don’t have real frontend or design skills.

I’ve recently started a series of articles about epistemology and distributed systems, which includes diagrams and equations that broke my existing publication system. I came up with a new stack of kludges that works for me. I’ll write them down here so I remember, and perhaps you’ll learn something you can use.

Optimizing a Meditation Retreat with a SAT Solver

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 29 Jul 2025 22:00:52 +0000

Well, a SAT solver can optimize the retreat schedule. A meditation retreat as a whole is a living creature, beyond any individual’s control. But I can provide a schedule that functions smoothly and reduces distractions. I used Python and integer programming, a technique for mathematical optimization.

The problem

The Village Zendo holds two weeklong Zen meditation retreats every summer, and one every winter. An essential part of Zen retreats is interviews (“dokusan” or “daisan”) between students and teachers, where students can ask questions or present their understanding of koans. At a Village Zendo retreat, we want each student to see one teacher per day. We assign students to groups, and call each group out of the meditation hall to see a teacher at the start of a meditation period. Certain students serve as jishas; they’re in charge of calling a group of students and escorting them to the door of the interview room. The students in the group line up, and enter the room one at time for a short conversation with the teacher.

From Python Programmer to Distributed Systems Researcher in 10 Years Without a PhD

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 30 Jun 2025 22:29:35 +0000

Phil Eaton wrote “From Web Developer To Database Developer In 10 Years” recently. I was inspired to write my own version.

In high school, I wanted to be a modern dancer, with a company like Pilobolus or Streb. But I also kind of wanted to be a programmer. Pixar’s Toy Story, the first 3D-animated feature film, was released when I was a high school sophomore. I was fascinated by this simulated world, revealed by simulated light to a simulated camera. How did they make it? I somehow learned (however I learned things before I had Internet access) that 3D graphics were rendered with linear algebra and C++. So I got books about both from my neighborhood library in Shaker Heights, Ohio. I don’t recall much about the linear algebra book, but the C++ book is a core memory: a giant glossy volume, with a copy of Borland Turbo C++ for DOS on a CD in a slip attached to the inside back cover. I installed it on the family computer, a Tandy with a 386 processor, which looked like this:

Taming Google Scholar Alerts

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 23 Jun 2025 17:09:48 +0000

Since I joined MongoDB’s Distributed Systems Research Group a few years ago, I’ve relied on Google Scholar alerts to tell me when an author in my specialty publishes a new paper, or (much more often) when a new paper cites an author I’m interested in. Google Scholar can send me email alerts, but their frequency isn’t configurable: daily only. And if the same new paper is included in multiple alerts (e.g., it cites several authors I follow), Google Scholar won’t deduplicate it. It’s common for me to get a half-dozen emails in the morning about overlapping sets of papers.

Conformance Checking at MongoDB: Testing That Our Code Matches Our TLA+ Specs

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 03 Jun 2025 09:03:01 +0000

Cross-posted from the MongoDB engineering blog.

At MongoDB, we design a lot of distributed algorithms—algorithms with lots of concurrency and complexity, and dire consequences for mistakes. We formally specify some of the scariest algorithms in TLA+, to check that they behave correctly in every scenario. But how do we know that our implementations conform to our specs? And how do we keep them in sync as the implementation evolves?

This problem is called conformance checking. In 2020, my colleagues and I experimented with two MongoDB products, to see if we could test their fidelity to our TLA+ specs. Here’s a video of my presentation on this topic at the VLDB conference. (It’ll be obvious to you that I recorded it from my New York apartment in deep Covid lockdown.) Below, I write about our experience with conformance checking from 2025’s perspective. I’ll tell you what worked for us in 2020 and what didn’t, and what developments there have been in the field in the five years since our paper.

Jesse's Notes from the Antithesis BugBash Conference

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 20 May 2025 22:33:34 +0000

Antithesis has pioneered “autonomous testing”, a term they coined for their platform, which combines a deterministic hypervisor with coverage-guided fuzzing and fault injection. My company MongoDB was one of their first customers—I evaluated their platform for MongoDB in 2020. At the time we had several very rare replication bugs, including one that manifested once in many years of stress testing (tens of thousands of hours of compute), and Antithesis found them reliably in about 24 hours.

Are We Serious About Using TLA+ For Statistical Properties?

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 10 May 2025 10:59:24 +0000

At this year’s TLA⁺ community event, I tentatively proposed adding features to make the language useful for performance modeling. Here’s the video, and a written version is below.

Half Our Problems

In 2022, I saw Marc Brooker give a talk called “Formal Methods Only Solve Half My Problems” at HPTS. He published a blog post about this too. He said that TLA⁺ can check correctness (safety and liveness), but not performance characteristics. “What I want is tools that do both: tools that allow development of formal models … and then allow us to ask those models questions about design performance.” He acknowledges that correctness is important. But it’s not enough to say nothing bad ever happens, and something good eventually happens—we want to know how quickly something good happens!

Jesse's 2025 TLA+ Community Event Notes

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 08 May 2025 21:47:33 +0000

Here are my notes from the 2025 TLA⁺ Community Event. See also my colleague Murat Demirbas’s notes (he was one of the event’s organizers). The talks were all recorded, and the videos, slides, and abstracts are posted.

Some years there are TLA⁺ “conferences” co-located with industry conferences, and some years there are “community events” co-located with academic conferences. This year was a community event, the first TLA⁺ community event in North America. There were about 20 participants, evenly divided between academia and industry. I knew most of them already. If I recall correctly, the last TLA⁺ thing I attended was the 2021 conference; it had more participants, perhaps because it was co-located with the huge StrangeLoop industry conference.

How Long Must I Test?

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 07 Apr 2025 22:36:05 +0000

When you’re hunting for bugs with a nondeterministic test, how many times should you run the test? The program might be deterministic: the same input produces the same behavior. Presuming you’re confident this is the case, you could fuzz the input for a while, hoping to catch bugs. But if your program is nondeterministic, you must not only give it random inputs, you must re-test it on the same input many times, hoping to achieve high branch coverage and catch rare bugs.

Join Me At NYC Systems October 17

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 10 Oct 2024 17:36:40 +0000

New York’s hottest club is NYC Systems. I’m speaking there October 17 with my colleague Matthieu Humeau. We’ll describe our experiment with predictive auto-scaling in MongoDB Atlas. The other presenter that night will be Andrew Werner, talking about instrumentation of compiled code. Unfortunately for you, registrations filled up a half hour after they opened. Join the waitlist! Or watch our presentation on the same topic at the Data Council conference this spring.

NYC Systems is a tech talk series recently begun by Phil Eaton. Phil also created a giant online book club that reads books about software internals. He’s talented at organizing communities, I’ve been observing from the sidelines and I’m impressed.

What's Next for Women in Tech?

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 24 Aug 2024 16:53:51 +0000

The Washington Post says “the movement to diversify Silicon Valley is crumbling”, and the story is particularly dire for gender-diversity nonprofits: Girls In Tech has dissolved, so has Women Who Code, and software companies have sharply cut their grants to others. But were they helping? The article links to this Department of Labor chart which shows women’s share of all STEM employment has grown from 25% in 2000 to only 26% in 2022. For some reason the Post didn’t cite a worse statistic on the same chart: women’s share of “computer occupations” has fallen from 30% to 24%.

How To Use MongoDB Causal Consistency

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 14 May 2024 15:53:22 +0000

MongoDB implemented a consistency level called causal consistency in 2017 (version 3.6). It’s quite a handy consistency level, strong enough for most applications and still performant. I helped design the API for causal consistency, but when we released it I dropped the ball and didn’t publicize or document it well. MongoDB’s causal consistency didn’t get the fame it deserved in 2017; I’ll try to rectify that now.

MongoDB implements a Raft-like consensus protocol. Most people deploy MongoDB as a three-server replica set, with one primary and two secondaries. Only the primary executes writes, which are replicated by secondaries with a small delay. You can read from the primary or the secondaries. Your application talks to the servers via an instance of the MongoClient class.

Pseudocode Is Not Durable

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 01 Apr 2023 17:06:47 +0000

My friend Andrew Helwer argues that the best way to communicate an algorithm is with PlusCal or Python. He has “come to believe that both are superior to an ad-hoc unspecified pseudo-math language,” of the sort that researchers often invent for their papers. He recommends a subset of Python that avoids “fancy features” like list comprehensions, and avoids the standard library. He shows an apt example: a 43-year-old algorithm that was specified in pseudocode and has therefore been plagued with bugs and ambiguities ever since.

Multi-Paxos in Python, tested with Jepsen

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Fri, 10 Dec 2021 15:25:18 +0000

I want to understand Paxos better, especially Multi-Paxos, so I implemented it (badly) in Python. I tested it with Jepsen—it was a chance to play with Jepsen, and a way to check if I’d understood Paxos well enough to code it. I spent about two weeks on the project (one was MongoDB’s periodic “Skunkworks” week). Here’s a rambling report on my experience.

Current and Future Tools for Interactive TLA+

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 05 Oct 2021 17:38:04 +0000

This is a talk I gave at the 2021 TLA+ conference with my MongoDB colleague Samyukta Lanka. The video is below, and a written version below that.

Update: See this Twitter thread for important corrections. Many of the features we asked for are already possible, but obscure.

Precise vs. holistic

In our experience there are two ways to understand a system: precisely or holistically. (This isn’t a super well-developed philosophy, just our observation.) The precise way of understanding involves well-defined questions like, “Does the system obey a particular invariant or property?” There are lots of powerful tools for this; this is what TLA+ and TLC are for.

Paper Review: Programming as Theory Building

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 26 Aug 2021 16:01:58 +0000

If you’re my colleague and you want to trigger my most ferocious pet peeve, ask me to explain something that I’ve already written down. My first response will be a terse link to the docs. I won’t say “RTFM” out loud, but if you listen closely you can hear me think it. If you ask a followup that I’ve also written the answer to, I get sarcastic and rude. Things generally devolve from there.

Choosing the Adventurous Route: The Career Path for Non-Managers

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 26 Feb 2019 11:49:37 +0000

If you’re committed to a lifetime of coding, how do you make the most of it?

Your training, experience, and passion are all in computer programming, but after a few years it can start to feel like you’re being railroaded into management, or out of the industry. Full-time programmers in their 40s or older are few, and the industry doesn’t reward long experience like it should. We can change this. We can find more creative ways to deepen our skills, get promoted, and expand our influence as coders for our entire careers.

Choosing the Adventurous Route: Resources

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Fri, 08 Feb 2019 23:22:04 +0000

Links to articles and papers I mention in my PyTennessee talk about the career path for older coders.

Video and transcript of my PyTennessee talk.

An article I wrote on this subject, aimed at a more general audience.

Old modem sounds and an explanation thereof.

Median age at Google and Amazon is 30.

Median age of American worker is 42.

Stack Overflow developer survey: three quarters of respondents were under 35.

Please Vote for My ConFoo Proposals

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 24 Sep 2018 08:35:26 +0000

I have some ideas I want to share at ConFoo 2019 in Montréal. If you think these talks sound interesting, take a minute and vote for me!

Please accept, as a token of my gratitude, this picture of a damp cat in Montréal (part of a set I shot in 2008).

Motor 2.0

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 11 Jul 2018 04:11:37 +0000

To support multi-document transactions, I had to make breaking changes to Motor’s session API and release a major version bump, Motor 2.0. Since this is a major release I also deleted many helper methods and APIs that had been deprecated over time since Motor 1.0, most notably the old CRUD methods insert, update, remove, and save, and the original callback-based API. Read the Motor 2.0 Migration Guide carefully to upgrade your existing Motor application. Motor 2.0 also includes a Python 3.7 compatibility fix in the MotorChangeStream class returned by MotorCollection.watch.

Video: Writing Winning Proposals for Tech Conferences

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 03 Feb 2018 21:51:21 +0000

I was invited by PyLadies Online to give a February 3, 2018 webinar about proposing talks, for the annual Global Diversity CFP Day. Watch the recording for my tips about advancing your conference-speaking career.

Announcing libbson and libmongoc 1.9.2

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 11 Jan 2018 22:48:08 +0000

I made a mistake in libbson 1.9.1, so I’ve just released 1.9.2 of libbson and libmongoc.

libbson 1.9.2 completes reverting a changed macro definition that broke API compatibility. The revert in 1.9.1 did not completely fix the BC break. Thanks to Petr Písař for finding and fixing the mistake. See CDRIVER-2460 for details. It’s a comedy of errors.

libmongoc 1.9.2 has no changes since 1.9.1, I released it to keep pace with libbson’s version number.

Announcing libbson and libmongoc 1.9.1

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 09 Jan 2018 17:46:11 +0000

I’m pleased to announce version 1.9.1 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

Announcing libbson and libmongoc 1.9.0

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 20 Dec 2017 21:29:51 +0000

I’m pleased to announce version 1.9.0 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

libbson

It is my pleasure to announce Libbson-1.9.0.

New features and bugfixes:

Fix Autotools syntax for OpenBSD and any platform lacking stdint.h.
Fix Android NDK incompatibilities.
Fix a one-byte write past the end of a buffer in bson_decimal128_to_string.
Avoid reading past the end of a string that contains UTF-8 multibyte NIL.
Fix some pedantic warnings in C99 mode.

libmongoc

It is my pleasure to announce mongo-c-driver 1.9.0. This version drops support for MongoDB 2.4 and adds support for MongoDB 3.6 features:

Motor 1.2.0

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 18 Dec 2017 15:47:17 +0000

I’m excited to announce version 1.2.0 of Motor, my async Python driver for MongoDB. Motor works with Python 2.7 and 3.4+, and it supports async MongoDB applications using either Tornado or asyncio. Version 1.2.0 is not substantially changed from the release candidate I announced last week, please read that post for all the details about this version of Motor.

Install Motor with pip:

python -m pip install -U motor

If you find issues, file a bug and I’ll respond promptly. But if it works for you, don’t be silent!—tweet at me and tell me.

Announcing Motor 1.2 Release Candidate, With MongoDB 3.6 Support

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 13 Dec 2017 07:01:32 +0000

MongoDB 3.6 was released December 5. Today I’ve uploaded a release candidate for version 1.2 of Motor, the async Python driver for MongoDB. This will be a big release so I hope you try the release candidate and tell me if it works for you or if you find bugs.

Install the release candidate with pip:

python -m pip install motor==1.2rc0

Compatibility Changes

MongoDB: drop 2.4, add 3.6
Python: drop 2.6, continue to support 2.7 and 3.3+
Tornado: drop 3, continue to support Tornado 4.5+
aiohttp: support 2.0 and later, drop older version

See the Compatibility Matrix for the relationships among Motor, Python, Tornado, and MongoDB versions.

Announcing libbson and libmongoc 1.8.2

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 18 Nov 2017 10:24:03 +0000

I’m pleased to announce version 1.8.2 of libbson and libmongoc, the libraries constituting the MongoDB C Driver. Sadly, this is the final version that I worked on with Hannes Magnusson. He’s moved on to fight new bugs on new battlefields.

libbson

No change since 1.8.1; released to keep pace with libmongoc’s version.

libmongoc

This release fixes the following bugs:

Announcing libbson and libmongoc 1.8.1

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 11 Oct 2017 22:34:57 +0000

I’m pleased to announce version 1.8.1 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

libbson

This release removes a syntax error in the configure script that affects some shells, and fixes the encoding of the NEWS file.

libmongoc

This release fixes the following bugs:

Remove a syntax error in the configure script that affects some shells.
The configure script respects –with-zlib=system and –with-snappy=system.
The internal mongoc_server_description_t struct is properly reinitialized after a network error.
Fix the encoding of the NEWS file.

Links:

Thanks to Jeremy Mikola for his contributions to this release.

Announcing libbson and libmongoc 1.8.0

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 13 Sep 2017 18:02:07 +0000

I’m pleased to announce version 1.8.0 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

libbson

New features and bugfixes:

Make symbols bson_get_major_version, bson_get_minor_version, bson_get_micro_version, bson_get_version, and bson_check_version available to C++ programs.
New CMake option ENABLE_MAINTAINER_FLAGS.
Crash iterating over invalid code with scope.

libmongoc

The zLib and Snappy compression libraries are bundled if not available. Wire protocol compression is enabled on Windows.
mongoc_collection_find_and_modify_with_opts now respects a “writeConcern” field in the “extra” BSON document in its mongoc_find_and_modify_opts_t.
These command functions now ignore “read_prefs”:
mongoc_client_read_write_command_with_opts,
mongoc_database_read_write_command_with_opts
mongoc_collection_read_write_command_with_opts
These functions are both deprecated, use mongoc_database_write_command_with_opts instead. A guide to creating an index using that function has been added.
mongoc_collection_create_index
mongoc_collection_create_index_with_opts
Use select, not WSAPoll, on Windows.
Always mark a server “Unknown” after a network error (besides a timeout).
mongoc_client_pool_t sends platform metadata to the server; before, only a single mongoc_client_t did.
New stream method mongoc_stream_timed_out.
Wire version checks introduced in 1.8.0 will prevent the driver from connecting to a future MongoDB server version if its wire protocol is incompatible.
New CMake option ENABLE_MAINTAINER_FLAGS.

Links:

Thanks to everyone who contributed to this release.

Announcing libbson and libmongoc 1.7.0

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 09 Aug 2017 12:51:14 +0000

I’m pleased to announce version 1.7.0 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

The main new features are an updated JSON-BSON codec, more resilient network layer, and convenient linking.

libbson

New features and bug fixes:

Changes to JSON encoding and decoding:
New functions bson_as_canonical_extended_json and bson_as_relaxed_extended_json convert BSON to canonical and relaxed extended JSON according to MongoDB Extended JSON Spec. Output for the existing bson_as_json function has not been changed.
When parsing JSON type wrappers like “$timestamp”, any missing or extra keys are an error.
The JSON format for BSON regular expressions is now “$regularExpression”: {“pattern”: “…”, “options”: “…”}. The old format {"$regex": “…”, “$options”: “…”} is still parsed.
The JSON format for BSON binary elements is now “$binary”: {“base64”: “…”, “subType”: “…”}. The old format {"$binary": “…”, “$type”: “…”} is still parsed.
BSON dates can be parsed from “$date” as an ISO8601 date or “$numberLong” as milliseconds since the epoch: “t”: {"$date": {"$numberLong": “1234”}}. Dates can no longer be formatted as raw integers. bson_as_json writes a BSON date after 1970 as “$date” with an ISO8601 string, and dates before 1970 as negative milliseconds wrapped in “$numberLong”. bson_as_canonical_extended_json always writes dates with “$numberLong”. bson_as_relaxed_extended_json always writes dates with “$date”.
The non-numbers NaN, Infinity, and -Infinity are now recognized (regardless of case) when parsing JSON.
CMake build now installs .pc files for programs that link to libbson using pkg-config. Both the CMake and Autotools build systems now install .cmake files for programs that link to libbson using CMake. Linking to libbson statically or dynamically is now much more convenient.
New CMake option, “ENABLE_STATIC”, defaults to ON.
Minimum required CMake version has been increased to 3.1.
CMake remains experimental on non-Windows platforms and issues a warning now
New functions
bson_strcasecmp, a portable equivalent of strcasecmp.
bson_iter_as_double, cast the current value to double.
bson_iter_init_from_data, creates an iterator from BSON string.
bson_validate_with_error, checks a document like bson_validate does but also reports which key was invalid
New convenience macros
BSON_ITER_HOLDS_INT, checks if iterator holds int32 or int64
BSON_ITER_HOLDS_NUMBER, checks if iterator holds int32, int64 or double
Raised BSON recursion limit to 200

libmongoc

New features and bug fixes:

The Noisiest Block in the Neighborhood: Analyzing NYC Data with Mongolite

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 15 Mar 2017 07:11:15 +0000

Back in 2012, I ended my seven years of residence on the Lower East Side and moved into an apartment with my girlfriend in Stuyvesant Town a mile north. I’d loved the LES when I first came to New York, because it was where my Jewish ancestors had first lived when they came to America a century ago, and their history still showed on every corner. I lived on Orchard Street, across from a Jewish corset shop, around the corner from the salmon specialists at Russ & Daughters. Half a block in the other direction was the Roumanian-American Synagogue, huge and old, once so famous for its Hebrew singing it was called the “Cantor’s Carnegie Hall.”

Announcing libbson and libmongoc 1.6.1

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 07 Mar 2017 10:10:22 +0000

I’m pleased to announce version 1.6.1 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

libbson

This is a bugfix release that resolves GCC 7 compiler warnings, improves HP-UX compatibility, and avoids a test failure on 32-bit MIPS.

Debugging the MIPS bug was a small saga. I don’t have a MIPS machine and I’ve only partly learned how to simulate one with QEMU; therefore it wasn’t until the Debian Autobuilder Network emailed me that I knew the libbson test suite had begun failing on 32-bit MIPS.

Monitoring MongoDB Driver Events In Motor

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 04 Feb 2017 15:13:45 +0000

Do you want to know every MongoDB query or command your program sends, and the server’s reply to each? How about getting a notification whenever the driver detects a primary failover, or when a new secondary joins the replica set? Over the last year, MongoDB drivers have implemented these monitoring features in all our supported programming languages. Here’s how to use monitoring in Motor, my Python async driver.

Announcing libbson and libmongoc 1.6.0

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 01 Feb 2017 21:23:48 +0000

I’m pleased to announce version 1.6.0 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

libbson

New features and bug fixes:

Use jsonsl instead of libyajl as our JSON parsing library, parse JSON more strictly, fix minor parsing bugs.
Extended JSON documents like {"$code": "...", "$scope": {}} are now parsed into BSON “code” elements.
ISO8601 dates now allow years from 0000 to 9999 inclusive. Before, years before 1970 were prohibited.
BSON floats and ints are now distinguished in JSON output.
The library is now built and continuously tested with MinGW-W64 on Windows.
The documentation is ported from Mallard XML to ReStructured Text, the HTML documentation is restyled, and numerous man page syntax errors fixed.
All public functions now have the __cdecl calling convention on Windows.

libmongoc

It is my please to announce mongo-c-driver 1.6.0.

Announcing libbson and libmongoc 1.5.4

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 30 Jan 2017 10:19:34 +0000

I’m pleased to announce version 1.5.4 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

There is no change to libbson since 1.5.3; we released it to keep pace with libmongoc’s version.

The libmongoc release fixes an error in cursor iteration when a readConcern is set. Thanks to Jeremy Mikola and Hannes Magnusson.

Links:

Peace,
— A. Jesse Jiryu Davis

Announcing libbson and libmongoc 1.5.3

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 11 Jan 2017 18:21:41 +0000

I’m pleased to announce version 1.5.3 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

libbson

This is a patch release that fixes a build error with MinGW on Windows.

libmongoc

This release fixes the following bugs:

CDRIVER-1964: Windows CA stores should be opened with read-only flag
CDRIVER-1970: Conflicting symbols MAX and MIN on FreeBSD
CDRIVER-1971: Missing exports of some GridFS functions
CDRIVER-1975: Mixed $ and non-$ query operators should be allowed

There was a glitch on the road from 1.5.1 to this release, 1.5.3. We tried to update the driver to connect to IPv6-only MongoDB servers by hostname, (CDRIVER-1972). For example, if the server listening on example.com only accepts IPv6 connections, we should be able to connect to it with the URI mongodb://example.com. Unfortunately we messed this up: if the server only accepts IPv4, we’d try IPv6 first, time out, and we did not then fall back to IPv4 (CDRIVER-1988).

Two Dramatic Performances About getaddrinfo

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 04 Jan 2017 21:25:09 +0000

I just discovered this: a couple weeks ago the BSD Now podcast featured my article about making getaddrinfo concurrent in Python BSD. They were doing a roundup of stories about ancient BSD lore and included my tale about fixing an ancient bug. If you want to watch the two hosts cracking each other up with quotes from my silly story, watch the video starting at minute 13.

The BSD Now hosts issued a challenge to their listeners: someone should do a dramatic reading of my entire saga. And the host of BSD Synergy, Mason Egger, has done it. Not only did he perform it with heroic tones fitting the subject matter, he even illustrated it with paintings from King Arthur and Tolkien. He intuited the exact atmosphere I aimed for when I wrote the story.

Announcing libbson and libmongoc 1.5.1

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 17 Dec 2016 14:25:39 +0000

I’m pleased to announce version 1.5.1 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

libbson

No change since 1.5.0; released to keep pace with libmongoc’s version.

libmongoc

This is a bugfix release:

Fix SEGFAULT with performance counters on NUMA (thanks to Jonathan Wang).
Prevent rare assertion error in mongoc_cluster_stream_for_server.
Improve error messages from auth failure.
Escape quotes when appending CFLAGS to handshake metadata.
Fix OpenSSL header lookups in non-default paths.
Fix build failure with LibreSSL.

Links:

Thanks to everyone who contributed to this release.

Making getaddrinfo Concurrent in Python On Mac OS and BSD

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 05 Dec 2016 11:50:25 +0000

Tell us about the time you made DNS resolution concurrent in Python on Mac and BSD.

No, no, you do not want to hear that story, my friends. It is nothing but old lore and #ifdefs.

But you made Python more scalable. The saga of Steve Jobs was sung to you by a mysterious wizard with a fanciful nickname! Tell us!

Gather round, then. I will tell you how I unearthed a lost secret, unbound Python from old shackles, and banished an ancient and horrible Mutex Troll.

Announcing Motor 1.1 For MongoDB 3.4

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 29 Nov 2016 22:58:25 +0000

MongoDB 3.4 was released this morning; tonight I’ve released Motor 1.1 with support for the latest MongoDB features.

Motor 1.1 now depends on PyMongo 3.4 or later. (It’s an annoying coincidence that the latest MongoDB and PyMongo versions are the same number.)

With MongoDB 3.4 and the latest Motor, you can now configure unicode-aware string comparison using collations. See PyMongo’s examples for collation. Motor also supports the new Decimal128 BSON type. The new MongoDB version supports write concern with all commands that write, so drop_database, create_collection, create_indexes, and all the other commands that modify your data accept a writeConcern parameter.

Announcing libbson and libmongoc 1.5.0

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 27 Nov 2016 22:13:46 +0000

I’m pleased to announce version 1.5.0 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

libbson

We have a brand-new BSON Type, Decimal128, which MongoDB 3.4 and all drivers now implement.

The BSON “code” and “code with scope” types are better supported now: bson_append_code_with_scope now preserves the “code with scope” type if scope is an empty, non-NULL BSON document, and the “code” and “code with scope” types are properly translated to JSON like this:

The Saga of Concurrent DNS: My Article in the MongoDB Engineering Journal

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 17 Nov 2016 12:15:26 +0000

Earlier this year I updated CPython to allow concurrent DNS resolution on Mac and BSD. My patch was trivial, but it took me weeks of archeological research to prove it was correct. In this article for the MongoDB Engineering Journal, I sing the ballad of my quest:

The Saga of Concurrent DNS in Python,
and the Defeat of the Wicked Mutex Troll

If silly fantasy stories are your thing, enjoy! If not, you can just read the report on bugs.python.org.

Announcing Motor 1.0

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 02 Nov 2016 18:23:39 +0000

More than four years ago, I was sitting through a particularly distracted period of meditation. It was a Saturday morning and I was sitting on a meditation cushion next to my girlfriend in her Brooklyn apartment, staring at the wall. I suddenly thought of a technique for turning PyMongo into an async driver without rewriting it from scratch, by using greenlets. That weekend I forked PyMongo and demonstrated a proof of concept to my boss.

Announcing Motor 0.7

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 26 Oct 2016 07:16:27 +0000

Three weeks after I released the beta, I’m proud to present Motor 0.7.

For asynchronous I/O Motor now uses a thread pool, which is faster and simpler than the prior implementation with greenlets. It no longer requires the greenlet package, and now requires the futures backport package on Python 2. Read the beta announcement to learn more about the switch from greenlets to threads.

Motor 0.7 Beta With Pymongo 2.9 And A Threaded Core

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 03 Oct 2016 08:28:36 +0000

Please try the beta release of Motor 0.7 and let me know how it works for you:

python -m pip install motor==0.7b0

Documentation:

In two ways, Motor 0.7 paves the way for Motor 1.0: first, its PyMongo dependency is upgraded from PyMongo 2.8 to 2.9. Second, I have abandoned my greenlet-based core in favor of a faster, simpler core built on a thread pool. Let's talk about threads first.

Announcing libbson and libmongoc 1.4.2

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Fri, 30 Sep 2016 17:26:03 +0000

I'm pleased to announce version 1.4.2 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

libbson

No change since 1.4.1; released to keep pace with libmongoc's version.

Announcing libbson and libmongoc 1.4.1

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 20 Sep 2016 16:16:32 +0000

Hannes Magnusson and I are pleased to announce version 1.4.1 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

Announcing libbson and libmongoc 1.4.0

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 10 Aug 2016 18:46:01 +0000

I'm pleased to announce version 1.4.0 of libbson and libmongoc, the libraries constituting the MongoDB C Driver. This is a very big release a long time coming.

The headline feature is support for the TLS libraries included in Mac and Windows, so you no longer need OpenSSL there. Hannes Magnusson built this feature off contributions by Samantha Ritter and Mark Benvenuto. He overcame months of frustrations and roadblocks to achieve a secure, production-quality implementation; it's an extraordinary achievement.

Announcing libbson and libmongoc 1.4.0-beta1

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 27 Jul 2016 23:06:58 +0000

I'm pleased to announce version 1.4.0-beta1 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

libbson

New features and bug fixes:

bson_reader_reset seeks to the beginning of a BSON buffer.
bson_steal efficiently transfers contents from one bson_t to another.
Fix Windows compile error with BSON_EXTRA_ALIGN disabled.
Potential buffer overrun in bson_strndup.
bson_oid_to_string optimization for MS Visual Studio
bson_oid_is_valid accepts uppercase hex characters.
bson_json_reader_read aborted on some invalid Extended JSON documents.
All man page names now begin with "bson_" to avoid install conflicts.
Fix bug where error messages were sometimes truncated at 63 chars.

This release tentatively supports the new BSON decimal type. This feature may change between now and version 1.5. To try it now, build the library like:

Talk Python to Me: "Write an Excellent Programming Blog"

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 26 Jul 2016 20:50:50 +0000

Michael Kennedy and I talked about writing about programming. What kind of writing is most valuable, how do you choose a topic, improve your writing, find an audience, and find the time to write? Listen to the podcast on the Talk Python To Me site.

I've talked with Michael before: Episode 2 of "Talk Python to Me" was about MongoDB and Python.

By the way: Michael's a Python expert and a master teacher. Join his "Python for Entrepreneurs" course on Kickstarter now for early access to the course at a deeply discounted price.

How To Write Resilient MongoDB Applications

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 14 Jul 2016 23:28:07 +0000

Update: This technique has been superseded by MongoDB’s built-in retryable writes, introduced in 2017. Retryable writes are far simpler and superior to the techniques described here. For even stronger guarantees, use transactions, released in 2018.

Once, on a winter afternoon in early 2012, I met a MongoDB customer who was very angry.

He'd come to our regular "MongoDB Office Hours" at our office in Soho, and he had one question: "How can I make my application resilient in the face of network errors, outages, and other exceptions? Can I just retry every operation until it succeeds?" He demanded to know why we hadn't published a simple, smart strategy that would work for all applications.

"Smart Strategies for Resilient MongoDB Applications", The Screencast

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 29 Jun 2016 13:56:40 +0000

I recorded a screencast of my MongoDB World 2016 talk. It's about handling network exceptions and other errors in a correct and resilient fashion. If you want a simple, reliable method for programming a MongoDB application, watch this:

72% Of The People I Follow On Twitter Are Men

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 23 Jun 2016 22:44:47 +0000

At least, that's my estimate. Twitter does not ask users their gender, so I have written a program that guesses based on their names. Among those who follow me, the distribution is even worse: 83% are men. None are gender-nonbinary as far as I can tell.

Come to MongoDB World. We Need To Talk.

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 09 Jun 2016 14:39:58 +0000

Please come see me at MongoDB World. It's in New York on June 28 and 29. I'll give you 25% off, I just need to talk to you.

You see, for four years, I've had this guilt hanging over me. It goes back to a winter afternoon in early 2012, when I met a MongoDB customer who was very angry. He'd come to our regular "MongoDB Office Hours" at our old spot in Soho, and he had one question: "When a MongoDB driver handles a network error, what should my code do?" He demanded to know why I didn't have a simple answer for him.

My Two Talks at Open Source Bridge

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 08 Jun 2016 09:54:04 +0000

My two favorite talks at Open Source Bridge this year aren't mine: they're my friends talks about accessibility, and about a parallel testing system written in Go.

Nevertheless, I'm excited about the talks I'm going to give. I hope to see you there: Register for Open Source Bridge, June 21-24.

My Two Favorite Talks at Open Source Bridge

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 07 Jun 2016 19:49:00 +0000

This year there are two talks I've had a hand in. The first is by Rebecca Jennings, an ASL interpreter and accessibility expert. What she knows is so fascinating that I nagged her for two years to speak about it at Open Source Bridge. Finally she's coming to teach us about accessible information systems.

"Write An Excellent Programming Blog" at PyCon 2016

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 02 Jun 2016 08:30:33 +0000

Video of my talk at PyCon 2016 in Portland. I talked about writing about programming.

Voluminous related information, including a written version of this material, a screencast that I captioned, and links to further reading:

The "Write an Excellent Programming Blog" Page

Say Useful Things To An Audience That's Listening: 6 Tips For Delivering A PyCon Talk

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 29 May 2016 08:34:31 +0000

Sylvia Pankhurst didn't waste time on Q & A.

Ten minutes before my first PyCon talk ever, I was standing at the podium. I had got up there as soon as the previous speaker stepped off—I had to be sure I was ready in time for my 3:15pm slot! The nice session runner gave me a water bottle, the A/V guy wired me up, and then they left me to stand and sweat at the lectern. I stared at the audience as they filed in. The waiting was unbearable. At the moment my laptop's clock flipped from 3:14 to 3:15, I began.

Captioning Myself And 6 Other Ways I'll Prepare In The 24 Hours Before I Speak At Pycon

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 26 May 2016 13:34:15 +0000

At 1:10pm Pacific Time this Wednesday, I'll be in Portland, sitting in a lecture room at PyCon, judging the right moment to walk on stage and start my talk. I have a regimen for the 24 hours before that primes me to give you the best talk I can.

This whole series of articles about conference speaking is inspired by my friend Sasha Laundy's 24 hours checklist, which is the definitive guide for public speakers. I'll add a few more tips for you to choose among, particularly my method to ensure my talk is accessible to Deaf people.

I Saw You Fall So I'm Helping

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 25 May 2016 09:10:15 +0000

Women have been systematically excluded from computer programming for decades. But it wasn't this way in the past, and it doesn't need to be in the future. The software industry, it seems, is waking up overnight to how unfairly we've treated women and minorities, and we're beginning to take action to fix it. A talk I gave at the Village Zendo on May 22, 2016.

A koan from The Hidden Lamp:

The Biggest Gift You Can Give Your Mentor

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 15 May 2016 17:25:09 +0000

I did not say, "It's nice to see you out of jail." I was buying lunch at a grocery store in Austin, and I had recognized the cashier. He was my age—I was 22—scraggy and blond. A few weeks before, he was in the meditation class I taught at the county jail.

Reminding Myself to Back Up My Drives, With Google Sheets

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 09 May 2016 08:33:19 +0000

My photo archive is the only data I have that's precious enough to back up, and too big to back up in the cloud. Nevertherless, I procrastinate about making backups. I especially procrastinate about swapping my home backups with the offsite ones at my office. Until now.

My Rig

The photo archive lives on a Seagate 4 TB hard drive, and I back it up to two other 4 TB drives. Yet two more 4 TB drives are Time Machine backups for the system drive on my laptop. I swap these drives in and out of a Mediasonic HF2-SU3S2 enclosure with 4 bays. It's cheap, remarkably small, and does the job.

Ubuntu 16.04 Includes The MongoDB C Driver

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 02 May 2016 13:42:05 +0000

I've been working with an expert Debian developer, Roberto Sanchez, to package libbson and libmongoc for Debian. Our efforts paid off: the MongoDB C Driver is now included in Debian Unstable and in Ubuntu 16.04 Xenial Xerus. If you run Xenial, get the driver as easily as:

apt-get install libmongoc-1.0

Roberto's patience and passion to teach me about Debian packaging has made it a joy for me.

If you're on a RedHat-like system, you're in luck. Remi Collet maintains excellent RPMs for the C Driver. Commands to enable Remi's repository and install libmongoc depend on your exact system. For example, on CentOS 6:

Please! Vote For Me At Open Source Bridge

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 26 Apr 2016 22:07:56 +0000

I love Open Source Bridge so much that I couldn't restrain myself: I submitted four talks. But I'm not embarrassed. My love for this conference must proclaim itself abundantly. So help me get accepted! Please add stars and comments to the four talks I proposed.

Update: "Black Pipe Testing" and "March to Triumph as a Mentor" have been accepted. My friend Rebecca Friedman's talk "The Ability to Disable: Who Did You Forget When You Designed Your UI?" was also accepted, and "Massively Parallel Testing at MongoDB" by my colleagues Shraya Ramani, Kyle Erf. See you in Portland!

Code Podcast: Event Loop & Coroutines

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 14 Apr 2016 21:59:02 +0000

It was a treat to talk with Saúl Ibarra Corretgé and Andrey Salomatin for Episode Three of the Code Podcast. We discussed async event loops and coroutines in Python 3 and libuv.

Andrey doesn't simply broadcast an interview: he carefully edits his episodes to tell a story about a specific subject, setting him apart from all the other podcasts about software. Less like Charlie Rose, more like Radiolab. I'm eager to hear the next one.

My First Two Articles For The PSF

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 05 Apr 2016 16:24:24 +0000

I'm delighted to tell you, I've started writing on the Python Software Foundation's blog. The PSF's communications head Diana Clarke asked me last month if I could blog for them, to share duties with their inveterate writer Mary Ann Sushinsky.

So far, I've talked with Damien George and Nicholas Tollervey about the effort to port Python to the new micro:bit, a tiny computer the BBC is distributing to a million school children in the UK. And I interviewed data scientist Jackie Kazil and PSF vice-chair Lynn Root about Jackie's career and her leadership of the PyLadies annual charity auction.

Announcing libbson and libmongoc 1.3.5

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 30 Mar 2016 17:14:02 +0000

I'm pleased to announce version 1.3.5 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

libbson

No change since 1.3.4; released to keep pace with libmongoc's version.

libmongoc

This release fixes a crash in mongoc_cleanup when an allocator had been set with bson_mem_set_vtable.

It also introduces a configure option MONGOC_NO_AUTOMATIC_GLOBALS which prevents code built with GCC from automatically calling mongoc_init and mongoc_cleanup when your code does not. This obscure, GCC-specific behavior was a bad idea and we'll remove it entirely in version 2.0. Meanwhile, we're letting you explicitly opt-out.

Announcing libbson and libmongoc 1.2.4 and 1.3.4

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 14 Mar 2016 23:21:32 +0000

I'm pleased to announce versions 1.2.4 and 1.3.4 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

libbson

The new versions of libbson have no changes; they're released to keep pace with libmongoc.

libmongoc

The MongoDB C Driver releases fix a security vulnerability: when a mongoc_client_t uses SSL and is disconnected, it failed to re-verify the server certificate after reconnecting. This flaw affects single clients, not pooled ones.

How To Play Foul With getaddrinfo()

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 13 Mar 2016 22:28:40 +0000

I'm the referee for a road rally. You have to drive from New York to San Francisco in 48 hours, but—here's the catch—I'm going to start my stopwatch before you can look at the map. Worse, there are hundreds of other drivers who need the map, and only one driver can use it at a time. If you're unlucky, you could spend the whole 48 hours waiting in line.

Announcing Motor 0.6

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 06 Mar 2016 16:41:34 +0000

Motor is the asynchronous Python driver for MongoDB. It is compatible with Tornado and asyncio.

This is just a patch release with six bugfixes, but some fixes required tiny API changes that might affect you, so I'm bumping the version from 0.5 to 0.6, instead of to 0.5.1. Read below for the changes—if you make it to the bottom you'll learn about an abstruse asyncio optimization in Python 3.5.1!

Cat-Herd's Crook published in the MongoDB Engineering Journal

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 03 Mar 2016 11:27:59 +0000

At MongoDB we write database drivers in ten programming languages. We try to make them all behave the same, but inevitably they vary. How can we herd all these cats along the same path? For years we failed, and each false start left us more discouraged. But now we're on track to standardize all our drivers. Samantha Ritter and I gave a talk about this at Open Source Bridge last year.

Performance, Persuasion, Structure: What A Speaking Coach Did For My PyCon Talk

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 28 Feb 2016 23:31:00 +0000

In the months leading to PyCon, I'm refining my talk three ways: rehearsing it alone, rehearsing it for friends, and rehearsing it for a professional speaking coach. Guess which is the most productive?

Yesterday I did a session with Melissa Collom, an opera singer, speaking coach, and friend. She advised me on my performance, persuasion, and structure. We made my talk twice as good, I bet she could help with your talk, too.

Announcing WinKerberos

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Fri, 26 Feb 2016 04:46:29 +0000

My colleague Bernie Hackett has published a new Python extension module called WinKerberos. It provides native Kerberos support to Python applications on Windows. It's a drop-in replacement for the popular PyKerberos package, but it uses Microsoft's own Kerberos implementation, the Security Support Provider Interface (SSPI), and supports some Windows specific options.

Motivation

A number of MongoDB customers have requested support for GSSAPI authentication with PyMongo on Windows, so they can use Kerberos with Python and MongoDB.

Do Your Slides At The Last Minute: 8 Steps To Writing Your PyCon Talk

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 10 Feb 2016 12:23:37 +0000

Jennie Baines didn't take questions at the end.

PyCon accepted my talk "Write an Excellent Programming Blog". If you got in, too, congratulations! Now we have to write our talks.

Plan your time.

Plan to begin your talk late, and not to take questions.

Announcing libbson and libmongoc 1.3.3

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Fri, 05 Feb 2016 09:11:06 +0000

I'm pleased to announce version 1.3.3 of libbson and libmongoc, the libraries constituting the MongoDB C Driver.

libbson

No change since 1.3.2; released to keep pace with libmongoc's version.

libmongoc

Fixes a bug where a slightly-oversized bulk write operation was not split into batches; instead, it was sent whole to the server, which rejected it.

Links:

Peace,
— A. Jesse Jiryu Davis

Announcing libbson and libmongoc 1.3.2

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 01 Feb 2016 20:53:38 +0000

I'm pleased to announce version 1.3.2 of libbson and libmongoc, the libraries constituting the MongoDB C Driver. We fixed a critical bug and a few minor ones.

libbson

man pages couldn't be built from a distribution tarball.

libmongoc

A socket is properly discarded after a network error from a command.
mongoc_database_get_collection copies the database's read preferences, read concern, and write concern, instead of copying the client's.
The mongoc_cursor_t private struct allows a negative limit.

Links

Thanks to everyone who contributed to this release.

getaddrinfo on Mac OS X

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 01 Feb 2016 05:31:29 +0000

In 2015 and 2016 I plunged deep into an investigation of getaddrinfo on Mac OS X.

Image: Isabella Cantanea Parasole, Italian 1565–1625, embroidery design. From Fashion and Virtue: Textile Patterns and the Print Revolution 1520–1620, by Femke Speelberg.

How To Hobble Your Python Web-Scraper With getaddrinfo()

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 31 Jan 2016 22:54:39 +0000

This is the second article in what seems destined to be a four-part series about Python's getaddrinfo on Mac. Here, I discover that contention for the getaddrinfo lock makes connecting to localhost appear to time out.

Network Timeouts From asyncio

A Washington Post data scientist named Al Johri posted to the MongoDB User Group list, asking for help with a Python script. His script downloaded feeds from 500 sites concurrently and stored the feeds' links in MongoDB. Since this is the sort of problem async is good for, he used my async driver Motor. He'd chosen to implement his feed-fetcher on asyncio, with Motor's new asyncio integration and Andrew Svetlov's aiohttp library.

Video of My Coroutines Live-Coding Demonstration, at SCALE14x

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 25 Jan 2016 15:41:14 +0000

Here's video of the live-coding demo I did Sunday at SCALE14x: in under an hour I built a Python 3 async framework. I began by demonstrating non-blocking sockets, callbacks, and an event loop. This implementation is very efficient and rather ugly. In the second half, I build Python coroutines out of a Future class, generators, and a Task class. Coroutines are the best of both worlds: as scalable as callbacks and as straightforward as regular blocking code.

Announcing libbson and libmongoc 1.3.1

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 18 Jan 2016 15:09:01 +0000

I'm pleased to announce version 1.3.1 of libbson and libmongoc, the libraries constituting the MongoDB C Driver. This is a run-of-the-mill bugfix release for you, unless you use GridFS with write concern, read concern, or read preference—in that case read carefully before upgrading.

libbson

bson_strnlen is off by one on Windows.
BSON_HAVE_STRNLEN config check used incorrectly.
Incompatibility with older CMake versions.
Wrong-sized allocation in bson_json_reader_new.

libmongoc

mongoc_client_get_gridfs now copies the client's read preferences, read concern, and write concern to the newly created mongoc_gridfs_t. Before this fix, GridFS operations were always executed with the default config: data was read from the primary, with the read concern level "local", and written with write concern "acknowledged". Now, if you have configured any of these options on the mongoc_client_t, they are respected by the mongoc_gridfs_t.
CMakeLists.txt now includes and installs the pkg-config files.

Links

Thanks to everyone who contributed to this release.

Experiences Migrating to Lektor

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sat, 16 Jan 2016 10:04:13 +0000

Over the last few weeks I've ported this blog, with over 400 articles, to Armin Ronacher's new static site generator Lektor. Lektor will grow and mature for years to come, but it isn't too early to write up my experience rebuilding a substantial site with it.

These observations fall into two categories. One is the comparison between Lektor and my homemade blog software, Motor-Blog. The other is the comparison between running a dynamic server versus deploying a static site.

This Site Is Built With Lektor

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 10 Jan 2016 13:48:20 +0000

In the last week I've ported this blog and my /photography to Armin Ronacher's new static site generator, Lektor.

In the beginning, I blogged on WordPress. WordPress is a wonder of the world. It powers a quarter of the web. The WordPress developers have figured out how to auto-update their millions of users (please follow that link, the story is fascinating), and this auto-updating makes it a remarkably secure and robust platform. But I'm somehow not smart enough to make WordPress fast.

How To Deadlock Your Python With getaddrinfo()

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 21 Dec 2015 09:23:30 +0000

What happens if you run this code?:

import os
import socket
import threading


def lookup():
    socket.getaddrinfo('python.org', 80)

t = threading.Thread(target=lookup)
t.start()
if os.fork():
    # Parent waits for child.
    os.wait()
else:
    # Child hangs here.
    socket.getaddrinfo('mongodb.org', 80)

On Linux, it completes in milliseconds. On Mac, it usually hangs. Why?

Journey To The Center Of The Interpreter

Anna Herlihy and I tackled this question a few months ago. It didn't look like the code example above—not at first. We'd come across an article by Anthony Fejes reporting that the new PyMongo 3.0 didn't work with his software, which combined multithreading with multiprocessing. Often, he'd create a MongoClient, then fork, and in the child process the MongoClient couldn't connect to any servers:

Announcing PyMongo 3.2

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 07 Dec 2015 21:24:46 +0000

Our Python team released PyMongo 3.2 today. This time I wasn't involved at all; Bernie Hackett, Anna Herlihy, and Luke Lovett developed this version, with a contribution from Felix Yan.

Version 3.2 implements the new server features introduced in MongoDB 3.2. (It's a coincidence that the version numbers are the same.)

Support for ReadConcern.
WriteConcern is now applied to find_one_and_replace(), find_one_and_update(), and find_one_and_delete().
Support for the new bypassDocumentValidation option in write helpers.
Reading and write raw BSON with RawBSONDocument—this feature is very exciting, show us what you can do with it!
We now prefer the package "Monotime" to the outdated "monotonic" to provide a safe clock.

Some MongoClient properties, like client.is_mongos, will now block until a connection is established or raise ServerSelectionTimeoutError if no server is available.

Announcing libbson and libmongoc 1.3.0

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 07 Dec 2015 21:09:33 +0000

I just released version 1.3.0 of libbson and libmongoc, the C libraries that compose the MongoDB C Driver. The changelist is massive; the highlights are support for new MongoDB 3.2 features, and a widespread cleanup of GridFS.

Links:

libbson

Since the release candidate libbson 1.3.0-rc0, the only changes have been fixes for compiler warnings and errors on various platforms.

Announcing libbson and libmongoc 1.3.0-rc0

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 02 Dec 2015 15:13:43 +0000

The MongoDB C Driver beta I announced two weeks ago has ripened into a release candidate. Here is what's changed between 1.3.0-beta0 and today's release, 1.3.0-rc0.

libbson

Parse DBRefs correctly from JSON.
CMake option to disable building tests: you can turn off ENABLE_TESTS.
Fix build warnings on some platforms.
The build system is refactored to declare the current version and latest release in one place.

libmongoc

Features:

Black Pipe Testing: In Summary

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 01 Dec 2015 08:44:56 +0000

Every conversation I hear about testing seems to go thus:

"Here's a useful way to test!"

"But if you only test that way, you won't catch all the bugs."

I would like us, the open source community, to move on from this conversation. Each testing method is useful for some kinds of software, and each accomplishes some goals. But none is plenipotent. There is no silver bullet.

Announcing libbson and libmongoc 1.2.2

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 30 Nov 2015 21:46:56 +0000

This morning I released version 1.2.2 of libbson and libmongoc, the libraries that compose the MongoDB C Driver.

This release fixes an edge case where the driver can query hidden secondaries unintentionally. The bug manifests rarely: the hidden node must be in the seed list, and your application must be reading with a non-primary read preference while no primary is available.

(If the hidden node is not in the seed list it is never discovered. If it is in the seed list but a primary is available, the driver trusts the primary's host list, which omits the hidden member.)

Announcing Motor 0.5 with asyncio, Python 3.5, and "async" / "await"

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 30 Nov 2015 09:31:39 +0000

Welcome back, I hope you enjoyed Thanksgiving. I certainly did; among everything for which I give thanks this year, I am grateful for the contributions Rémi Jolin, Andrew Svetlov, and Nikolay Novik made to Motor's asyncio integration. Their help is the greatest gift any project of mine has received.

And now, it's official! Motor 0.5 is released. Install it with:

python -m pip install motor

This version of Motor is compatible with Python 2.6 through 3.5, and it can use either Tornado or asyncio. This means pip no longer automatically installs Tornado.

Announcing libbson and libmongoc 1.3.0-beta0

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 18 Nov 2015 17:42:50 +0000

I've just released a beta of the MongoDB C driver 1.3.0, with support for new features in the upcoming MongoDB 3.2. The driver is compatible with MongoDB 2.4 and later. Please try it out and file a ticket in Jira if you see any issues.

Links:

New features and changes:

mongoc_collection_find_and_modify will now apply the mongoc_collection_t's write_concern_t when talking to MongoDB 3.2.
Support for MongoDB 3.2's "readConcern" feature for queries, counts, and aggregations. The option "readConcernLevel" is now accepted in the MongoDB URI. New struct mongoc_read_concern_t, and functions operating on it:
- mongoc_client_get_read_concern
- mongoc_client_set_read_concern
- mongoc_database_get_read_concern
- mongoc_database_set_read_concern
- mongoc_collection_get_read_concern
- mongoc_collection_set_read_concern
- mongoc_read_concern_copy
- mongoc_read_concern_destroy
- mongoc_read_concern_get_level
- mongoc_read_concern_new
- mongoc_read_concern_set_level
- mongoc_uri_get_read_concern
Support for MongoDB 3.2's "bypassDocumentValidation" option for writes.
New struct mongoc_bulk_write_flags_t and related functions:
- mongoc_bulk_operation_set_bypass_document_validation
- mongoc_bulk_operation_set_flags
New struct mongoc_find_and_modify_opts_t and related functions:
- mongoc_find_and_modify_opts_new
- mongoc_find_and_modify_opts_destroy
- mongoc_find_and_modify_opts_set_sort
- mongoc_find_and_modify_opts_set_update
- mongoc_find_and_modify_opts_set_fields
- mongoc_find_and_modify_opts_set_flags
- mongoc_find_and_modify_opts_set_bypass_document_validation
- mongoc_collection_find_and_modify_with_opts
Configurable wait time on tailable cursors with MongoDB 3.2:
- mongoc_cursor_get_max_await_time_ms
- mongoc_cursor_set_max_await_time_ms
Support for MongoDB 3.2 wire protocol: use commands in place of OP_QUERY, OP_GETMORE, and OP_KILLCURSORS messages.
To explain a query plan with MongoDB 3.2, you must now call the "explain" command, instead of including the "$explain" key in a mongoc_collection_find query. See the mongoc_collection_find documentation page for details.
Use constant-time comparison when verifying credentials
Combine environment's CFLAGS with configure options when building.
Improved man page output and "whatis" entries

Extensive bugfixes and improvements in GridFS, including:

Tech Talks Are Campfires: Jesse's Three Reasons You Should Speak At PyCon

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 15 Nov 2015 22:16:45 +0000

The PyCon call for proposals has been open a few weeks now—why should you submit a talk? Why should you speak at software events in general?

Build the campfire

Programmers like to meet. It's surprising, with our reputation for introversion—or perhaps it is to overcome our introversion that we have so many events. In any case, our demand for software conferences and Meetups is voracious. PyCon, for example, has grown from 400 attendees in 2006 to 3000 people last year. Past PyCon chair Diana Clarke tells me they could have easily sold a thousand more tickets in recent years if they'd raised the cap.

Motor 0.5 Beta: asyncio, async and await, simple aggregation

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Tue, 10 Nov 2015 23:52:06 +0000

Today is a good day: I've published a beta of Motor, my async Python driver for MongoDB. This version is the biggest upgrade yet. Help me beta-test it! Install with:

python -m pip install --pre motor==0.5b0

Motor 0.5 still depends on PyMongo 2.8.0 exactly. That PyMongo version is outdated, I know, but I've decided not to tackle that issue right now.

You'll forgive me, because this Motor release is huge:

This Blog Runs MongoDB 3.2 rc2

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Sun, 08 Nov 2015 22:11:02 +0000

This blog is my equivalent of the hotrod an auto-mechanic has sitting in the garage: very custom, chopped down, and jagged at the edges. I've built it in Python, of course, with Tornado and Motor, my async driver for MongoDB. The original intent of the blog was to eat my own hamster food by building substantial software on top of Motor. Nowadays it's just the roadster I tinker with on the weekend.

Test MongoDB Failure Scenarios With MockupDB

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Wed, 04 Nov 2015 09:46:01 +0000

This is the fifth article in my series on "black pipe" testing. Traditional black box tests work well if your application takes inputs and returns output through one interface: the API. But connected applications have two interfaces: both the API and the messages they send and receive on the network. I call the validation of both ends a black pipe test.

In my previous article I described black pipe testing in pure C; now we return to Python.

PyMongo 3.1 Works Around A Funny Performance Flaw In Python 2

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 02 Nov 2015 18:14:46 +0000

Bernie Hackett, Anna Herlihy, Luke Lovett, and I are pleased to announce the release of PyMongo 3.1. It adds features that conform to two new cross-language driver specs: it implements the Command Monitoring Spec to help you measure performance, and it adds a GridFSBucket class to match our new GridFS Spec.

A few of our users reported that PyMongo 3 used five or ten percent of their CPU while idle, and recorded a couple hundred context switches per second. I investigated and found a slapstick performance flaw in Python 2's condition variable that was interacting badly with my concurrency design in PyMongo 3.

Announcing libbson and libmongoc 1.2.1

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Thu, 29 Oct 2015 21:25:58 +0000

It is my pleasure to announce the 1.2.1 release of libbson and libmongoc, the C libraries that compose the MongoDB C Driver.

This release includes critical bugfixes for SSL connections with mongoc_client_pool_t, and for Unix domain socket connections.

The documentation is updated for a change introduced in version 1.2.0: mongoc_client_set_ssl_opts and mongoc_client_pool_set_ssl_opts now configure the driver to require an SSL connection to the server, even if "ssl=true" is omitted from the MongoDB URI. Before, SSL options were ignored unless "ssl=true" was included in the URI.

Black Pipe Testing A Connected Application In C

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Mon, 26 Oct 2015 23:20:08 +0000

This is the fourth article in my series on "black pipe" testing. Here I describe testing libmongoc (the MongoDB C Driver) as a black pipe.

Like any network client library, libmongoc cannot be fully tested as a black box. Traditional black box tests enter some input and check the output—this only validates one side of the system at a time. But libmongoc has two sides, working in concert. One side is its public API, its structs and functions and so on. The other is its communication over the network with the MongoDB server. Only by treating it as a black pipe can we fully test its two sides.

Cross-linking Mallard HTML Pages With yelp-build

jesse@emptysquare.net (A. Jesse Jiryu Davis) — Fri, 23 Oct 2015 11:37:08 +0000

From its start, the MongoDB C Driver has been split into two projects: libbson and libmongoc. Each has its own reference manual, each comprising hundreds of pages, written in the Mallard XML format. We use yelp-build to convert the Mallard to HTML and put it online. For example, here's the reference page for libmongoc's mongoc_collection_find:

mongoc_collection_find

mongoc_cursor_t *
mongoc_collection_find (mongoc_collection_t       *collection,
                        mongoc_query_flags_t       flags,
                        uint32_t                   skip,
                        uint32_t                   limit,
                        uint32_t                   batch_size,
                        const bson_t              *query,
                        const bson_t              *fields,
                        const mongoc_read_prefs_t *read_prefs);

Parameters

collection

Programming on A. Jesse Jiryu Davis

This Senior Staff Engineer Vibe-Coded for the First Time, What Happened Next Will Shock You

How My Blog Handles Math and Images in HTML, Atom, and Email in 2025

Optimizing a Meditation Retreat with a SAT Solver

The problem

From Python Programmer to Distributed Systems Researcher in 10 Years Without a PhD

Taming Google Scholar Alerts

Conformance Checking at MongoDB: Testing That Our Code Matches Our TLA+ Specs

Jesse's Notes from the Antithesis BugBash Conference

Are We Serious About Using TLA+ For Statistical Properties?

Half Our Problems

Jesse's 2025 TLA+ Community Event Notes

How Long Must I Test?

Join Me At NYC Systems October 17

What's Next for Women in Tech?

How To Use MongoDB Causal Consistency

Pseudocode Is Not Durable

Multi-Paxos in Python, tested with Jepsen

Current and Future Tools for Interactive TLA+

Precise vs. holistic

Paper Review: Programming as Theory Building

Choosing the Adventurous Route: The Career Path for Non-Managers

Choosing the Adventurous Route: Resources

Please Vote for My ConFoo Proposals

Motor 2.0

Further Reading About What To Expect When You're Expiring

Video: Writing Winning Proposals for Tech Conferences

Announcing libbson and libmongoc 1.9.2

Announcing libbson and libmongoc 1.9.1

Announcing libbson and libmongoc 1.9.0

libbson

libmongoc

Motor 1.2.0

Announcing Motor 1.2 Release Candidate, With MongoDB 3.6 Support

Compatibility Changes

Announcing libbson and libmongoc 1.8.2

libbson

libmongoc

Announcing libbson and libmongoc 1.8.1

libbson

libmongoc

Links:

Announcing libbson and libmongoc 1.8.0

libbson

libmongoc

Links:

Announcing libbson and libmongoc 1.7.0

libbson

libmongoc

The Noisiest Block in the Neighborhood: Analyzing NYC Data with Mongolite

Announcing libbson and libmongoc 1.6.1

libbson

Monitoring MongoDB Driver Events In Motor

Announcing libbson and libmongoc 1.6.0

libbson

libmongoc

Announcing libbson and libmongoc 1.5.4

Links:

Announcing libbson and libmongoc 1.5.3

libbson

libmongoc

Two Dramatic Performances About getaddrinfo

Announcing libbson and libmongoc 1.5.1

libbson

libmongoc

Links:

Making getaddrinfo Concurrent in Python On Mac OS and BSD

Announcing Motor 1.1 For MongoDB 3.4

Announcing libbson and libmongoc 1.5.0

libbson

The Saga of Concurrent DNS: My Article in the MongoDB Engineering Journal

Announcing Motor 1.0

Announcing Motor 0.7

Motor 0.7 Beta With Pymongo 2.9 And A Threaded Core

Announcing libbson and libmongoc 1.4.2

libbson

Announcing libbson and libmongoc 1.4.1

Announcing libbson and libmongoc 1.4.0

Announcing libbson and libmongoc 1.4.0-beta1

libbson