A. Jesse Jiryu Davis

Announcing libbson and libmongoc 1.1.5

I've released versions 1.1.5 today of libbson and libmongoc. libbson is a C library for creating, parsing, and manipulating BSON documents. libmongoc is the C Driver for MongoDB, a library for building high-performance applications [...]

Sea Smoke

I've released versions 1.1.5 today of libbson and libmongoc.

libbson is a C library for creating, parsing, and manipulating BSON documents. libmongoc is the C Driver for MongoDB, a library for building high-performance applications that communicate with MongoDB in the C language. It also serves as the base for drivers in some higher-level languages.

Release tarballs are available for download:

This is a patch release with small bug fixes. In libbson:

  • Fix link error "missing __sync_add_and_fetch_4" in GCC on i386 - the functions bson_atomic_int_add and bson_atomic_int64_add are now compiled and exported if needed in i386 mode
  • Fix version check for GCC 5 and future versions of Clang
  • Fix warnings and errors building on various platforms

In libmongoc:

  • The fsync and j write concern flags now imply acknowledged writes
  • Prevent using fsync or j with conflicting w=0 write concern
  • Obey socket timeout consistently in TLS/SSL mode
  • Return an error promptly after a network hangup in TLS mode
  • Prevent crash using SSL in FIPS mode
  • Always return NULL from mongoc_database_get_collection_names on error
  • Fix version check for GCC 5 and future versions of Clang
  • Fix warnings and errors building on various platforms
  • Add configure flag to enable/disable shared memory performance counters
  • Minor docs improvements and fix links from libmongoc to libbson docs

For further information:

With this release, I abandon the convention that odd-numbered patch versions indicate unstable releases. I am switching to simple semantic versioning: 1.1.5 is a stable release with bug fixes since 1.1.4. During subsequent development the libmongoc and libbson versions will be "1.1.6-dev".

This is my first release of libbson and libmongoc; I needed a lot of help and I received it. Thanks to those who contributed:

  • Christian Hergert
  • Hannes Magnusson
  • Jason Carey
  • Jeremy Mikola
  • Jeroen Ooms
  • Paul Melnikow

Image: Kristopher Wilson/ US Navy

Announcing PyMongo 3.0.2

Bernie Hackett and I are pleased to announce PyMongo 3.0.2. This release fixes bugs reported since PyMongo 3.0.1—most importantly, a bug that could route operations to replica set members that are not in primary or secondary [...]

Hepatica leaf

Bernie Hackett and I are pleased to announce PyMongo 3.0.2. This release fixes bugs reported since PyMongo 3.0.1—most importantly, a bug that could route operations to replica set members that are not in primary or secondary state when using read preference PrimaryPreferred or Nearest.

For the full list of bugs fixed in PyMongo 3.0.2, please see the release in Jira.

If you use PyMongo 3.0.x, upgrade.

If you are on PyMongo 2.8.0, upgrade to yesterday's bugfix release PyMongo 2.8.1 instead. Read the changelog for major API changes in PyMongo 3, and test your application carefully with PyMongo 3.0.x before deploying.

Announcing PyMongo 2.8.1

PyMongo 2.8.1 is a bugfix release that addresses issues discovered since PyMongo 2.8 was released, primarily related to authentication and metadata operations on replica sets. If you're on PyMongo 2.8 and not ready to update your code [...]

Vines

PyMongo 2.8.1 is a bugfix release that addresses issues discovered since PyMongo 2.8 was released, primarily related to authentication and metadata operations on replica sets. If you're on PyMongo 2.8 and not ready to update your code for the new APIs and behaviors of the PyMongo 3.0 line, please upgrade to PyMongo 2.8.1 promptly.

  • PYTHON-842 - Unable to specify 'ssl_cert_reqs' option using URI style connection string
  • PYTHON-864 - Fully support RFC-3339 offset format for $date
  • PYTHON-893 - Wrong wrapping function called for CommandCursor
  • PYTHON-903 - Properly handle network errors in auth
  • PYTHON-913 - UserWarning with read preference and command using direct connection
  • PYTHON-915 - secondaryAcceptablelatencyMS should accept 0
  • PYTHON-918 - Auth err from resyncing member prevents primary discovery
  • PYTHON-920 - collection_names, options, and index_information prohibited on direct connection to secondary with MongoDB 3.0+
  • PYTHON-921 - database_names prohibited on direct connection to secondary with MongoDB 3.0+

For the full list of bugs fixed in PyMongo 2.8.1, please see the release in Jira.

Announcing Motor 0.4.1

I received an extraordinarily helpful bug report yesterday from Brent Miller, who showed me that Motor's replica set client hangs if it tries two operations at once, while it is setting up its initial connection. He sent a script that not [...]

Motor

I received an extraordinarily helpful bug report yesterday from Brent Miller, who showed me that Motor's replica set client hangs if it tries two operations at once, while it is setting up its initial connection. He sent a script that not only reproduces the hang, but diagnoses it, too, by regularly dumping all threads' stacks to a file.

A report this generous made my work easy. I found that I'd caused this bug while fixing another one. In the previous bug, if Motor's replica set client was under load while reconnecting to your servers, it could start multiple greenlets to monitor your replica set, instead of just one. (Eventually, Motor will be designed to start multiple greenlets and monitor all servers in parallel, the same as PyMongo 3, but for now, starting multiple monitor greenlets is a bug.)

I fixed that bug overzealously: now if you start multiple operations on a replica set client as it connects, it does not start the monitor greenlet at all, and deadlocks. Motor 0.4.1 gets it right. It starts one and only one monitor greenlet as it connects to your replica set. Get it from PyPI:

pip install motor==0.4.1

Download Stats By Python Version: PyMongo, Motor, Tornado

After PyCon last month, Python packaging saint Donald Stufft generously tweeted: If you’re interested to see Py2 vs Py3 breakdowns for a Python package, let me know while I still have the data set loaded (~300GB DB). He had the [...]

After PyCon last month, Python packaging saint Donald Stufft generously tweeted:

If you’re interested to see Py2 vs Py3 breakdowns for a Python package, let me know while I still have the data set loaded (~300GB DB).

He had the database loaded for his diverting article on a Year of PyPI Downloads, but I was curious about three packages I own or contribute to.


PyMongo

Here's PyMongo's downloads this year, by Python version:

PyMongo downloads

Python 2.7 dominates, and it is growing at the expense of 2.6. Python 2.4 is absent, and PyMongo downloads for Python 2.5 vanished last year, which validates our decision to drop Python 2.4 and 2.5 from the latest release, PyMongo 3.0.

Donald made a second chart isolating the Python 3 downloads:

PyMongo downloads for Python 3

As one expects, people who use Python 3 use the latest one, Python 3.4. When we released PyMongo 2.8.0 at the end of January, the download spike was entirely Python 3.4 users.

I expect Python 3.5 will soon dominate among Python 3 users, and Python 2.6 will continue to decline asymptotically, while the Python 2 versus 3 ratio overall will stay steady for a few more years.

Motor

Something is wrong with Donald's chart for Motor, but it suggests that Motor users are like PyMongo users: 10% run the latest Python 3, 50% run Python 2.7, and the others run a smattering of other Pythons. (Motor has never supported Python before 2.6.)

Motor downloads

In Donald's Python 3 chart, it seems only Python 3.4 is represented:

Motor downloads for Python 3

Tornado

Motor's potential user base includes all Tornado users, so I was curious about Tornado's overall distribution:

Tornado downloads

I have no explanation for the spike of Python 2.6 downloads last fall; Tornado's release schedule doesn't strongly correlate with it. Tornado's user base is distributed similarly to PyMongo's, though more inclined to stay on Python 2.6.

Again, Tornado's Python 3 users hold steady at ten percent, but they switched to Python 3.4 quickly after it was released.

Tornado downloads for Python 3

Conclusion

The takeaways are: Python 2.4 is dead, and 2.5 is effectively dead for PyMongo, Motor, and Tornado users. It's important to keep packages working in Python 2.6—mainly for enterprises with long-term support contracts for Linux versions that shipped with 2.6—but adding features or optimizations that only work in Python 2.7 is reasonable now.

It's critical we keep testing Python 3.5 alphas and betas as they come out, because the 10% of people who run Python 3 will migrate to 3.5 rapidly. Python 3.2 is nearly dead, and 3.3 will also vanish soon.

Review of "Autotools: A Practitioner's Guide"

My colleague Amalia Hawkins and I are choosing interns for our project this summer: a new Lua driver for MongoDB. Looking over their resumes I have to chuckle. Kid, you're 20 years old, you're not proficient in C, C++, Java, Javascript, and [...]

Autotools calcote

My colleague Amalia Hawkins and I are choosing interns for our project this summer: a new Lua driver for MongoDB. Looking over their resumes I have to chuckle. Kid, you're 20 years old, you're not proficient in C, C++, Java, Javascript, and PHP. Maybe you did some homework in a language and you know some syntax, but it does not make you "proficient." I am proficient in one language: Python. It took me a decade.

When I began learning Python as a professional, what I didn't know that I didn't know was how to maintain a cross-platform open-source package like PyMongo. I didn't know how to write source code compatible with Python 2 and 3. I didn't know all the grotty little details about individual Python versions. I didn't know, for example, that assigning to a threadlocal isn't thread-safe in Python 2.6. I did not know how to design Python C extensions to run in mod_wsgi sub interpreters. I did not know that Python 2.7's unittest framework introduced assertRaisesRegexp and that it was reintroduced to Python 3.1 as assertRaisesRegex without the "p", then backported to Python 2.6 via unittest2, again without the "p". I did not know how to make a package installable on Linux, Mac, and Windows, with and without a C compiler.

I tell you this, not to brag about what I have learned—although to be honest I am proud of how far I have come. I just want to explain how high my standards for "proficient" have risen, as a result of my experiences working on PyMongo.

Now that I am taking over libbson and libmongoc, the C libraries for MongoDB client applications, I am humbled by how much I do not know. I am even more humbled by the unknown unknowns. I expect they are primarily in the distribution tools, not the language. After all, I "learned C" in college—or so I thought when I was 20 years old. But all I learned was its syntax. What I did not learn was how to maintain a cross-platform open-source distribution in C. So when I was charged with becoming an expert C programmer, I did not pick up a book on syntax or algorithms. I looked for a book on the Autotools.

Autotools: A Practioner's Guide to GNU Autoconf, Automake, and Libtool, by John Calcote, seems virtually the only modern book on the subject. Compared to all the C++ books out there, or all the "Learn C in 21 Days"-type introductions to the language, there is very little written about the Autotools. But a huge proportion of open source software for Linux and Unix uses the Autotools for packaging and distribution. Becoming an expert open source C programmer requires enough familiarity with the Autotools to create distributions and debug problems in them.

Calcote's book tackles this tiresome subject in a stylishly written and carefully organized manner.

We get off to a bumpy start, however. Calcote must begin by listing the parts of the Rube Goldberg device: primarily Automake, Autoconf, and Libtool. But there are wheels within wheels, like autom4te, aclocal, and a whole gearbox of little-known cogs. We barely understand the purpose of the Autotools, so we read the list of parts without understanding their functions. It is discouraging at the outset. But you will be alright if you just read the chapter once, do your best, then soldier on ignorantly and hope to muddle through.

It only gets more disheartening when we come to the data-flow diagrams:

Data flow

Data can flow in so many directions among the Autotools that the diagrams, although we desperately want them, are almost meaningless: every file depends on all the others, it seems, and the scripts all call each other. The system is so complex you need to read much of the book before the diagrams in Chapter One can make sense. Indeed, I have read the whole book, then reread the first chapter, and I still do not understand it. But the round trip brings a bit more clarity, and more hope.

The next chapters are tractable, since they explain the contraption one gear at a time, beginning at the lowest-level tools and working backwards from there. Due to the Autotools' design-by-accretion, each chapter discusses a tool, like make, and the script it executes, like Makefile, and then the tool that generates that script from a higher-level script. As we climb the levels of script-generating scripts we arrive, gasping, at the pinnacle: Automake. Except, of course, that the levels are not so cleanly separated at all, but are incomprehensibly intertwined. Calcote makes a heroic effort to focus each chapter on a specific aspect of the toolkit, but by necessity refers upwards and downwards, in the same way the tools themselves do.

In Chapters Eight and Nine, Calcote describes the monstrous undertaking of converting a large project from hand-written Makefiles to the Autotools. Here I skimmed quickly, but my admiration deepened for the author's patience. If I am ever willing to make such an expedition myself, I will want Calcote beside me. As A. E. Housman put it:

It will do good to heart and head
When your soul is in my soul's stead;
And I will friend you, if I may,
In the dark and cloudy day.

The last two chapters best fit the archetype of a "missing manual" for the Autools. In Chapter Ten, Calcote teaches us to use the M4 macro language in the context of Autoconf. I had recently been frustrated Googling for something as simple as this: How to execute one M4 expression if a symbol is defined, and another expression if not. The main trouble is that the M4 Manual teaches you to use raw M4, and the Autoconf Manual tells you that it has renamed all of M4's macros, so there is no one place on the Internet that actually tells you how to write Autoconf scripts. Calcote's explanation suffices to get you unstuck.

The final chapter, "A Catalog of Tips and Reusable Solutions," is without apology a bunch of facts that washed to the end of the book without coming to rest in any of the tutorial chapters. Yet the short sections here are more worthwhile than a "catalog" might promise. The opening bit on "Keeping Private Details out of Public Interfaces" teaches you enough about designing stable C APIs to keep you safe until you read David R. Hanson's C Interfaces and Implementations. Later, the section on cross-compilation collects crucial details, not just for people who actually cross-compile libraries, but for maintainers like me who must ensure their libraries are truly portable.

"Autotools: A Practitioner's Guide" is not fun to read, most likely. You read it when you must, and on that day it is indispensable. Calcote completes a tough job with honor and style, and his book testifies to his years of honest effort.

Photos From My Shuso Hossen

My fellow meditator and photographer Stillman Brown was generous enough to take pictures during the shuso hossen ceremony the other week. My teacher Enkyo Roshi allowed me to give my first Zen talk and I became a senior student at the [...]

My fellow meditator and photographer Stillman Brown was generous enough to take pictures during the shuso hossen ceremony the other week. My teacher Enkyo Roshi allowed me to give my first Zen talk and I became a senior student at the Village Zendo.


Jiryu selects 3

Jiryu selects 6

The verse that comes with the koan I spoke on, "Yangshan Plants His Hoe." Calligraphy by Musho.

Jiryu selects 8

Jiryu selects 9

Jiryu selects 10

Jiryu selects 15

Jiryu selects 16

There's a moment of high ritual where I return the koan to my teacher.

Jiryu selects 20

Carrying the short staff back to my seat so I can engage in "dharma combat", the questions and answers that follow my talk.

Jiryu selects 23

Jennifer asks me a question.

Jiryu selects 31

With my teacher Enkyo Roshi.

Jiryu selects 33

Jiryu selects 37

Samantha Ritter And Me At Open Source Bridge 2015

I'm so excited to tell you, my colleague Samantha Ritter and I were accepted to speak in Portland, Oregon this June. Cat-herd's Crook: Enforcing Standards in 10 Programming Languages. Samantha and I helped specify and test how MongoDB [...]

Open source bridge 1

I'm so excited to tell you, my colleague Samantha Ritter and I were accepted to speak in Portland, Oregon this June.

Cat-herd's Crook: Enforcing Standards in 10 Programming Languages.

Samantha and I helped specify and test how MongoDB drivers behave in ten programming languages, and we persuaded dozens of open source programmers to implement these specifications the same. We created a test framework that verified compliance with YAML descriptions of how drivers should act, and we communicated updates to the specs by pushing changes to the YAML files.

We want to show you how we herded all the cats in the same direction.

How Do Python Coroutines Work?

I'll explain asyncio's new coroutine implementation in depth, including the mystical "yield from" statement. You'll know better than any of your peers how this amazing new programming model works. Plus, in a magical and entertaining feat of daring, I plan to live-code an asynchronous coroutine implementation before your very eyes!

Announcing PyMongo 3.0.1

It's my pleasure to announce the release of PyMongo 3.0.1, a bugfix release that addresses issues discovered since PyMongo 3.0 was released a couple weeks ago. The main bugs were related to queries and cursors in complex sharding setups, [...]

Leaf

It's my pleasure to announce the release of PyMongo 3.0.1, a bugfix release that addresses issues discovered since PyMongo 3.0 was released a couple weeks ago. The main bugs were related to queries and cursors in complex sharding setups, but there was an unintentional change to the return value of save, GridFS file-deletion didn't work properly, passing a hint with a count didn't always work, and there were some obscure bugs and undocumented features.

For the full list of bugs fixed in PyMongo 3.0.1, please see the release in Jira.

If you are using PyMongo 3.0, please upgrade immediately.

If you are on PyMongo 2.8, read the changelog for major API changes in PyMongo 3, and test your application carefully with PyMongo 3 before deploying.

Vote For Me At Open Source Bridge!

Update: Two of these talks were accepted, thanks for your support. This is a shameless plug. I'm not ashamed of how much I want to speak at Open Source Bridge this year—it's my favorite conference with my favorite people and I [...]

Portland old town

Update: Two of these talks were accepted, thanks for your support.

This is a shameless plug. I'm not ashamed of how much I want to speak at Open Source Bridge this year—it's my favorite conference with my favorite people and I desperately want to speak there again. So vote for me! Add stars and comments to the three talks I proposed.

Add star

Cat-herd's Crook: Enforcing Standards in 10 Programming Languages.

I'm proposing this talk with my colleague and first-time speaker Samantha Ritter. We helped MongoDB specify and test our specifications for driver APIs and behaviors, and we persuaded dozens of open source programmers to implement these specifications the same. We want to show you how we herded all the cats in the same direction.

Dodge Disasters and March to Triumph as a Mentor.

If you're ambitious to advance in your career, or you care about your junior colleagues' advancement, then it is time for you to learn how to be a great mentor. Especially if you’re committed to diversity: mentorship is critical to the careers of women and minorities in tech. I have failed at mentoring, then succeeded. Learn from me and march to mentorship triumph.

How Do Python Coroutines Work?

I'll explain asyncio's new coroutine implementation in depth, including the mystical "yield from" statement. You'll know better than any of your peers how this amazing new programming model works. Plus, in a magical and entertaining feat of daring, I plan to live-code an asynchronous coroutine implementation before your very eyes!