PyMongo 3.0 is a partial rewrite of the Python driver for MongoDB. More than six years after the first release of the driver, this is the biggest release in PyMongo's history. Bernie Hackett, Luke Lovett, Anna Herlihy, and I are proud of its many improvements and eager for you to try it out. I will shoehorn the major improvements into four shoes: conformance, responsiveness, robustness, and modernity.

(This article will be cross-posted on the MongoDB Blog.)

Conformance

The motivation for PyMongo's overhaul is to supersede or remove its many idiosyncratic APIs. We want you to have a clean interface that is easy to learn and closely matches the interfaces of our other drivers.

CRUD API

Mainly, "conformance" means we have implemented the same interface for create, read, update, and delete operations as the other drivers have, as standardized in Craig Wilson's CRUD API Spec. The familiar old methods work the same in PyMongo 3, but they are deprecated:

• save
• insert
• update
• remove
• find_and_modify

These methods were vaguely named. For example, update updates or replaces some or all matching documents depending on its arguments. The arguments to save and remove are likewise finicky, and the many options for find_and_modify are intimidating. Other MongoDB drivers do not have exactly the same arguments in the same order for all these methods. If you or other developers on your team are using a driver from a different language, it makes life a lot easier to have consistent interfaces.

The new CRUD API names its methods like update_one, insert_many, find_one_and_delete: they say what they mean and mean what they say. Even better, all MongoDB drivers have exactly the same methods with the same arguments. See the spec for details.

One Client Class

In the past we had three client classes: Connection for any one server, and ReplicaSetConnection to connect to a replica set. We also had a MasterSlaveConnection that could distribute reads to slaves in a master-slave set. In November 2012 we created new classes, MongoClient and MongoReplicaSetClient, with better default settings, so now PyMongo had five clients! Even more confusingly, MongoClient could connect to a set of mongos servers and do hot failover.

As I wrote earlier, the fine distinctions between the client classes baffled users. And the set of clients we provided did not conform with other drivers. But since PyMongo is among the most-used of all Python libraries we waited long, and thought hard, before making major changes.

The day has come. MongoClient is now the one and only client class for a single server, a set of mongoses, or a replica set. It includes the functionality that had been split into MongoReplicaSetClient: it can connect to a replica set, discover all its members, and monitor the set for stepdowns, elections, and reconfigs. MongoClient now also supports the full ReadPreference API. MongoReplicaSetClient lives on for a time, for compatibility's sake, but new code should use MongoClient exclusively. The obsolete Connection, ReplicaSetConnection, and MasterSlaveConnection are removed.

The options you pass to MongoClient in the URI now completely control the client's behavior:

>>> # Connect to one standalone, mongos, or replica set member.
>>> client = MongoClient('mongodb://server')
>>>
>>> # Connect to a replica set.
>>> client = MongoClient(
...     'mongodb://member1,member2/?replicaSet=my_rs')
>>>
>>> # Load-balance among mongoses.
>>> client = MongoClient('mongodb://mongos1,mongos2')

This is exciting because PyMongo applications are now so easy to deploy: your code simply loads a MongoDB URI from an environment variable or config file and passes it to a MongoClient. Code and configuration are cleanly separated. You can move smoothly from your laptop to a test server to the cloud, simply by changing the URI.

Non-Conforming Features

PyMongo 2 had some quirky features it did not share with other drivers. For one, we had a copy_database method that only one other driver had, and which almost no one used. It was hard to maintain and we believe you want us to focus on the features you use, so we removed it.

A more pernicious misfeature was the start_request method. It bound a thread to a socket, which hurt performance without actually guaranteeing monotonic write consistency. It was overwhelmingly misused, too: new PyMongo users naturally called start_request before starting a request, but in truth the feature had nothing to do with its name. For the history and details, including some entertaining (in retrospect) tales of Python threadlocal bugs, see my article on the removal of start_request.

Finally, the Python team rewrote our distributed-systems internals to conform to the new standards we have specified for all our drivers. But if you are a Python programmer you may care only a little that the new code conforms to a spec; it is more interesting to you that the new code is responsive and robust.

Responsiveness

PyMongo 3's MongoClient can connect to a single server, a replica set, or a set of mongoses. It finds servers and reacts to changing conditions according to the Server Discovery And Monitoring spec, and it chooses which server to use for each operation according to the Server Selection Spec. David Golden and I explained these specs in general in the linked articles, but I can describe PyMongo’s implementation here.

Replica Set Discovery And Monitoring

In PyMongo 2, MongoReplicaSetClient used a single background thread to monitor all replica set members in series. So a slow or unresponsive member could block the thread for some time before the thread moved on to discover information about the other members, like their network latencies or which member is primary. If your application was waiting for that information—say, to write to the new primary after an election—these delays caused unneeded seconds of downtime.

When PyMongo 3's new MongoClient connects to a replica set it starts one thread per mongod server. The threads fan out to connect to all members of the set in parallel, and they start additional threads as they discover more members. As soon as any thread discovers the primary, your application is unblocked, even while the monitor threads collect more information about the set. This new design improves PyMongo's response time tremendously. If some members are slow or down, or you have many members in your set, PyMongo's discovery is still just as fast.

I explained the new design in Server Discovery And Monitoring In Next Generation MongoDB Drivers, and I'll actually demonstrate it in my MongoDB World talk, Drivers And High Availability: Deep Dive.

Mongos Load-Balancing

Our multi-mongos behavior is improved, too. A MongoClient can connect to a set of mongos servers:

>>> # Two mongoses.
>>> client = MongoClient('mongodb://mongos1,mongos2')

The behavior in PyMongo 2 was "high availability": the client connected to the lowest-latency mongos in the list, and used it until a network error prompted it to re-evaluate their latencies and reconnect to one of them. If the driver chose unwisely at first, it stayed pinned to a higher-latency mongos for some time. In PyMongo 3, the background threads monitor the client's network latency to all the mongoses continuously, and the client distributes operations evenly among those with the lowest latency. See mongos Load Balancing for more information.

Throughput

Besides PyMongo's improved responsiveness to changing conditions in your deployment, its throughput is better too. We have written a faster and more memory efficient pure python BSON module, which is particularly important for PyPy, and made substantial optimizations in our C extensions.

Robustness

Disconnected Startup

The first change you may notice is, MongoClient's constructor no longer blocks while connecting. It does not raise ConnectionFailure if it cannot connect:

>>> client = MongoClient('mongodb://no-host.com')
>>> client
MongoClient('no-host.com', 27017)

The constructor returns immediately and launches the connection process on background threads. Of course, foreground operations might time out:

>>> client.db.collection.find_one()
AutoReconnect: No servers found yet

Meanwhile, the client's background threads keep trying to reach the server. This is a big win for web applications that use PyMongo—in a crisis, your app servers might be restarted while your MongoDB servers are unreachable. Your applications should not throw an exception at startup, when they construct the client object. In PyMongo 3 the client can now start up disconnected; it tries to reach your servers until it succeeds.

On the other hand if you wrote code like this to check if mongod is up:

>>> try:
...     MongoClient()
...     print("it's working")
... except pymongo.errors.ConnectionFailure:
...     print("please start mongod")
...

This will not work any more, since the constructor never throws ConnectionFailure now. Instead, choose how long to wait before giving up by setting serverSelectionTimeoutMS:

>>> client = MongoClient(serverSelectionTimeoutMS=500)
>>> try:
...     client.admin.command('ping')
...     print("it's working")
... except pymongo.errors.ConnectionFailure:
...     print("please start mongod")

One Monitor Thread Per Server

Even during regular operations, connections may hang up or time out, and servers go down for periods; monitoring each on a separate thread keeps PyMongo abreast of changes before they cause errors. You will see fewer network exceptions than with PyMongo 2, and the new driver will recover much faster from the unexpected.

Thread Safety

Another source of fragility in PyMongo 2 was APIs that were not designed for multithreading. Too many of PyMongo's options could be changed at runtime. For example, if you created a database handle:

>>> db = client.test

...and changed the handle's read preference on a thread, the change appeared in all threads:

>>> def thread_fn():
...     db.read_preference = ReadPreference.SECONDARY

Making these options mutable encouraged such mistakes, so we made them immutable. Now you configure handles to databases and collections using thread-safe APIs:

>>> def thread_fn():
...     my_db = client.get_database(
...         'test',
...         read_preference=ReadPreference.SECONDARY)

Modernity

Last, and most satisfying to the team, we have completed our transition to modern Python.

While PyMongo 2 already supported the latest version of Python 3, it did so tortuously by executing auto2to3 on its source at install time. This made it too hard for the open source community to contribute to our code, and it led to some absurdly obscure bugs. We have updated to a single code base that is compatible with Python 2 and 3. We had to drop support for the ancient Pythons 2.4 and 2.5; we were encouraged by recent download statistics to believe that these zombie Python versions are finally at rest.

Motor

Motor, my async driver for Tornado and MongoDB, has not yet been updated to wrap PyMongo 3. The current release, Motor 0.4, wraps PyMongo 2.8. Motor's still compatible with the latest MongoDB server version, but it lacks the new PyMongo 3 features—for example, it doesn't have the new CRUD API, and it still monitors replica set members serially instead of in parallel. The next release, Motor 0.5, still won't wrap PyMongo 3, because Motor 0.5 will focus on asyncio support instead. It won't be until version 0.6 that I update Motor with the latest PyMongo changes.