A. Jesse Jiryu Davis

Toro 0.7 Released

I've just released version 0.7 of Toro. Toro provides semaphores, locks, events, conditions, and queues for Tornado coroutines. It enables advanced coordination among coroutines, similar to what you do in a multithreaded [...]

Toro

I've just released version 0.7 of Toro. Toro provides semaphores, locks, events, conditions, and queues for Tornado coroutines. It enables advanced coordination among coroutines, similar to what you do in a multithreaded application. Get the latest version with "pip install --upgrade toro". Toro's documentation, with plenty of examples, is on ReadTheDocs.

There is one bugfix in this release. Semaphore.wait() is supposed to wait until the semaphore can be acquired again:

@gen.coroutine
def coro():
    sem = toro.Semaphore(1)
    assert not sem.locked()

    # A semaphore with initial value of 1 can be acquired once,
    # then it's locked.
    sem.acquire()
    assert sem.locked()

    # Wait for another coroutine to release the semaphore.
    yield sem.wait()

... however, there was a bug and the semaphore didn't mark itself "locked" when it was acquired, so "wait" always returned immediately. I'm grateful to "abing" on GitHub for noticing the bug and contributing a fix.

A Normal Accident In Python and mod_wsgi

I fixed a glitch in PyMongo last week, the result of a slapstick series of mishaps. It reminds me of the Three Mile Island nuclear accident, which inspired the "Normal Accidents" theory of failure in complex systems: one surprise leads to [...]

Three Mile Island nuclear power plant

I fixed a glitch in PyMongo last week, the result of a slapstick series of mishaps. It reminds me of the Three Mile Island nuclear accident, which inspired the "Normal Accidents" theory of failure in complex systems: one surprise leads to the next, to the next, to an outcome no one anticipated.

It started a couple months ago, when we got a minor bug report about PyMongo. The reporter was using PyMongo with Python 3.2, mod_wsgi, and Apache. Whenever he restarted his application, he saw this error message in his log:

Exception TypeError:
  "'NoneType' object is not callable"
  in <bound method Pool.__del__> ignored

The exception was ignored because it was raised from a "__del__" method, so it didn't affect his application. Still, I needed to understand what was going on. So I made a test environment, and I used Apache to run a Python script like the one in the bug report:

import pymongo

class C:
    pass

C.client = pymongo.MongoClient()

I could reproduce the bug: Whenever I restarted Apache, the PyMongo connection pool's destructor logged "TypeError: NoneType object is not callable."

The pool's destructor makes two method calls, and no function calls:

def __del__(self):
    # _thread_id_to_sock is a dict of sockets.
    for sock in self._thread_id_to_sock.values():
        sock.close()

During interpreter shutdown, None is somehow being called as a function. I'm no expert on Python's shutdown sequence, but I've never heard of a method being set to None. And yet, the only calls in this code are the "values" method and the "close" method. What gives?

I put a "return" statement at the beginning of "__del__" and restarted Apache: the error disappeared. So I moved the "return" statement down a line, before "sock.close()". The next time I restarted Apache, I saw the error again.

While I was hacking directly on the installed PyMongo package, I noticed something funny. The installed code looked like:

def __del__(self):
    # _thread_id_to_sock is a dict of sockets.
    for sock in list(self._thread_id_to_sock.values()):
        sock.close()

Notice the call to "list"? When I installed PyMongo with Python 3.2, the installer ran 2to3 on PyMongo's code, which automatically translates Python 2 syntax to Python 3.

Why did 2to3 decide to wrap the "values" call in "list"? Well, in Python 2, "values" returns a copy, but in Python 3 it returns a dictionary view that's tied to the dict's underlying data. 2to3 worries that I might rely on the old, copying behavior, so in Python 3 it makes a copy of the values by calling "list".

So it must be the call to "list" that raises the TypeError. Sure enough, when I deleted the "list" call from the installed PyMongo code, the exception disappears. Fantastic!

Why don't we see this error all the time, though? Perhaps it has to do with the shutdown sequence. Normally, a pool is referred to by other objects, but not by a class. I hypothesized that the reporter saw the error because he'd made a reference from a class to the MongoClient to the pool, which delayed the pool's destruction until after the "list" builtin had been set to None:

Class refers to pool

To test this theory, I replaced this line:

C.client = pymongo.MongoClient()

...with this:

client = pymongo.MongoClient()

Now the pool is no longer referred to by a class, it's only referred to by a global variable in the module named "mod":

Variable refers to pool

Sure enough the error disappeared.

So far, I understood that the connection pool's destructor ran too late, because it was being kept alive by a reference from a class, and it relied on the "list" builtin, because 2to3 had added a call to "list", so it raised a TypeError. Now, did it only happen with mod_wsgi? I wrote the simplest Python example I could, and I tried to reproduce the TypeError:

# mod.py
class C(object):
    pass

class Pool(object):
    def __del__(self):
        print('del')
        list()

C.pool = Pool()

I could import this module into the Python shell, then quit, and I got no TypeError. Actually I didn't see it print "del" either—the pool's destructor never runs at all. Why not?

A class definition like "C" creates a reference cycle. It refers to itself as the first element in its method resolution order. You can see how "C" refers to itself by printing its method resolution order in the Python shell:

>>> import mod
>>> mod.C.__mro__
(<class 'mod.C'>, <type 'object'>)

When the interpreter shuts down it runs the C function "Py_Finalize", which first does a round of garbage collection to destroy reference cycles, then destroys all modules:

void Py_Finalize(void) {
    /* Collect garbage.  This may call finalizers; it's nice to call these
     * before all modules are destroyed.
     */
    PyGC_Collect();

    /* Destroy all modules */
    PyImport_Cleanup();
}

When "PyGC_Collect" runs, the "mod" module still refers to class C, so the class isn't destroyed and neither is the Pool it refers to:

Class reference cycle

Next, "PyImport_Cleanup" sets all modules' global variables to None. Now class C is garbage: it's in a reference cycle and nothing else refers to it:

Cyclic garbage

But the interpreter is dying and it will never call "PyGC_Collect" again, so class C is never destroyed and neither is the pool.

Great, I understand everything up to this point. But, if the pool is never destroyed when a regular Python interpreter shuts down, why is it destroyed when a mod_wsgi application restarts? I dove into mod_wsgi's source code to see how it manages Python interpreters. (This isn't my first rodeo: I examined mod_wsgi closely for my "Python C Extensions And mod_wsgi" article last year.) I wrote a little C program that runs Python in a sub interpreter, the same as mod_wsgi does:

int main()
{
    Py_Initialize();
    PyThreadState *tstate_enter = PyThreadState_Get();
    PyThreadState *tstate = Py_NewInterpreter();

    PyRun_SimpleString("import mod\n");
    if (PyErr_Occurred()) {
        PyErr_Print();
    }
    Py_EndInterpreter(tstate);
    PyThreadState_Swap(tstate_enter);
    printf("about to finalize\n");
    Py_Finalize();
    printf("done\n");

    return 0;
}

Just like mod_wsgi, my program creates a new Python sub interpreter and tells it to import my module, then it swaps out the sub interpreter and shuts it down with "Py_EndInterpreter". Its last act is "Py_Finalize". And behold! The script quoth:

about to finalize

Exception TypeError:
  "'NoneType' object is not callable"
  in <bound method Pool.__del__> ignored

done

My little C program acts just like the application in the bug report! What is it about this code that makes it throw the TypeError during shutdown, when a regular Python interpreter does not?

I stepped through my program in the debugger and solved the final mystery. What makes this code special is, it calls "Py_EndInterpreter". "Py_EndInterpreter" calls "PyImport_Cleanup", which sets all modules' global variables to None, thus turning class C into cyclic garbage:

Cyclic garbage

"PyImport_Cleanup" even clears the "builtins" module, which includes functions like "list". Any code that tries to call "list" afterward is actually calling None.

Now "Py_Finalize" calls "PyGC_Collect". (It will then run "PyImport_Cleanup" for the second time, but that's not relevant now.) This is the difference between the regular interpreter's shutdown sequence and mod_wsgi's: In the mod_wsgi case, modules have been cleared before the final garbage collection, so class C is destroyed along with the pool. However, since the pool's destructor runs after "PyImport_Cleanup", its reference to "list" is now None, and it throws "TypeError: 'NoneType' object is not callable".

Success! I had traced the cause of the bug from start to finish. To recap: in the bug-reporter's code, he had made a reference from a class to a pool, which made the pool's destructor run very late. And he ran the code in mod_wsgi, which clears modules before the final garbage collection, otherwise the pool's destructor wouldn't have run at all. He was using Python 3, so 2to3 had inserted a call to "list" in the pool's destructor, and since the destructor ran after all modules were cleared, the call to "list" failed.

Luckily, this cascade of failures leads merely to an occasional log message, not to a Three Mile Island meltdown. My boss Bernie came up with an incredibly simple fix. I replace the call to "values":

def __del__(self):
    for sock in self._thread_id_to_sock.values():
        sock.close()

... with a call to "itervalues":

def __del__(self):
    for sock in self._thread_id_to_sock.itervalues():
        sock.close()

(You can view the whole commit here.)

Now that I'm using "itervalues", 2to3 now replaces it with "values" in Python 3 instead of "list(values)". Since I'm no longer relying on the "list" builtin to be available in the destructor, no TypeError is raised.

reStructured Text With Chrome And LiveReload

I've found a useful set of tools for writing RST, when I must. I'll show you how to configure LiveReload and Chrome to make the experience of writing RST's tortured syntax somewhat bearable. (This article is an improvement over the method I [...]

I've found a useful set of tools for writing RST, when I must. I'll show you how to configure LiveReload and Chrome to make the experience of writing RST's tortured syntax somewhat bearable.

(This article is an improvement over the method I wrote about last year.)

LiveReload

I bought LiveReload from the Mac App Store for $10, and opened it. Under "Monitored Folders" I added my project's home directory: I was updating Motor's documentation so I added the "motor/doc" directory.

LiveReload

Next to "Monitoring 44 file extensions" I hit "Options" and added "rst" as a 45th.

LiveReload file extension options

Then I checked "Run custom command after processing changes" and hit "Options". In the popup dialog I added the command for building Motor's documentation. It's a typical Sphinx project, so the build command is:

/Users/emptysquare/.virtualenvs/motor/bin/sphinx-build \
  -b html -d _build/doctrees . _build/html

Note that I specified the full path to the virtualenv'ed sphinx script.

That's all there is to configuring LiveReload. Hit the green box on the lower right of its main window to see the build command's output. Now whenever you change an RST file you should see some Sphinx output scroll by:

LiveReload Sphinx output

Chrome

Next, follow LiveReload's instructions for installing the Chrome extension. Pay attention to LiveReload's tip: "If you want to use it with local files, be sure to enable 'Allow access to file URLs' checkbox in Tools > Extensions > LiveReload after installation."

Now open one of the HTML files Sphinx made, and click the LiveReload icon on your browser to enable it. The difference between "enabled" and "disabled" is damn subtle. This is disabled:

Disabled

This is enabled:

Enabled

The icon plays it close to the chest, but if you hover your mouse over it, it'll admit whether it's enabled or not.

Back at the LiveReload application, you'll now see "1 browser connected."

Try it out! Now you can make changes to your RST and see it live in your browser. I don't think I'll ever learn to type RST's syntax reliably, but at least now, I can see at once whether I've typed it right or not.

Motor 0.3.3 Released

Today I released version 0.3.3 of Motor, the asynchronous MongoDB driver for Python and Tornado. This release is compatible with MongoDB 2.2, 2.4, and 2.6. It requires PyMongo 2.7.1. This release fixes an occasional infinite loop and [...]

Motor

Today I released version 0.3.3 of Motor, the asynchronous MongoDB driver for Python and Tornado. This release is compatible with MongoDB 2.2, 2.4, and 2.6. It requires PyMongo 2.7.1.

This release fixes an occasional infinite loop and memory leak. The bug was triggered when you passed a callback to MotorCursor.each, and Motor had to open a new socket in the process of executing your callback, and your callback raised an exception:

from tornado.ioloop import IOLoop
import motor

loop = IOLoop.instance()

def each(result, error):
    raise Exception()

collection = motor.MotorClient().test.test
cursor = collection.find().each(callback=each)
loop.start()

The bug has been present since Motor 0.2. I am indebted to Eugene Protozanov for an excellent bug report.

Get the latest version with pip install --upgrade motor. The documentation is on ReadTheDocs. View the changelog here. If you encounter any issues, please file them in Jira.

MongoDB Boston

[Source] I'm speaking at MongoDB Boston. My talk is about visualizing weather data using Python, MongoDB, and other open source tools. Here's some links if you saw my talk and want more information. NumPy: The fundamental data format for [...]

Acorn Street

[Source]

I'm speaking at MongoDB Boston. My talk is about visualizing weather data using Python, MongoDB, and other open source tools. Here's some links if you saw my talk and want more information.

NumPy: The fundamental data format for scientific computing in Python: http://www.numpy.org/.

SciPy: A huge collection of fast algorithms for analyzing NumPy data: http://www.scipy.org/.

MatPlotLib: A charting library of vast capabilities, works great with NumPy arrays: http://matplotlib.org/

Monary: This is a high-performance driver that queries MongoDB and stores the data directly in NumPy arrays: https://bitbucket.org/djcbeach/monary/

Our Example App: All the code for loading and analyzing NOAA's weather data: https://github.com/drmirror/century


Postscript: near my hotel, Boston's Trinity Church is absurdly juxtaposed with The Hancock:

Trinity Church and The Hancock

It reminds me of Christ Church Cathedral in Montréal, which I photographed six years ago. It's similarly dominated by a glass box:

Christ Church Montreal

I guess this is inevitable in cities with a history. Skyscrapers move in and all the old buildings are torn down but the churches, dwarves living among young giants.

Buddhists at the People's Climate March

Yesterday I joined the Buddhist section of the giant People's Climate March in Manhattan. I was on assignment for the Shambhala Sun, photographing the Brooklyn Zen Center, so I came to the march with that sangha and then met up with my own [...]

Yesterday I joined the Buddhist section of the giant People's Climate March in Manhattan. I was on assignment for the Shambhala Sun, photographing the Brooklyn Zen Center, so I came to the march with that sangha and then met up with my own group, the Village Zendo.


Brooklyn Zen Center

Members of the Brooklyn Zen Center.


Buddhist sign

Buddhists congregate at their stepping-off point before joining the march.


Hare Krishna

A Hare Krishna on a bicycle dispensing copies of the Bhagavad Gita.


Times Square

Marching toward Times Square.


Enkyo Roshi and members of the Village Zendo

My teacher Enkyo Roshi and members of the Village Zendo.


Signs

Buddhist signs.


Cleanup cart

The ecumenical cleanup cart.

Announcing The Server Discovery And Monitoring Spec

Today I published a draft of the Server Discovery And Monitoring Spec for MongoDB drivers. This spec defines how a MongoDB client discovers and monitors a single server, a set of mongoses, or a replica set. How does the client determine [...]

Spacecraft Discovery from 2001: A Space Odyssey

Today I published a draft of the Server Discovery And Monitoring Spec for MongoDB drivers. This spec defines how a MongoDB client discovers and monitors a single server, a set of mongoses, or a replica set. How does the client determine what types of servers they are? How does it keep this information up to date? How does the client find an entire replica set from a seed list, and how does it respond to a stepdown, election, reconfiguration, or network error?

In the past each MongoDB driver answered these questions a little differently, and mongos differed a little from the drivers. We couldn't answer questions like, "Once I add a secondary to my replica set, how long does it take for the driver to discover it?" Or, "How does a driver detect when the primary steps down, and how does it react?"

From now on, all drivers answer these questions the same. Or, where there's a legitimate reason for them to differ, there are as few differences as possible and each is clearly explained in the spec. Even in cases where several answers seem equally good, drivers agree on one way to do it.

The server discovery and monitoring method is specified in five sections. First, a client is constructed. Second, it begins monitoring the server topology by calling the ismaster command on all servers. (The algorithm for multi-threaded and asynchronous clients is described separately from single-threaded clients.) Third, as ismaster responses are received the client parses them, and fourth, it updates its view of the topology. Finally, the spec describes how drivers update their topology view in response to errors.

I'm particularly excited about the unittests that accompany the spec. We have 37 tests that are specified formally in YAML files, with inputs and expected outcomes for a variety of scenarios. For each driver we'll write a test runner that feeds the inputs to the driver and verifies the outcome. This ends confusion about what the spec means, or whether all drivers conform to it.

The Java driver 2.12.1 is the spec's reference implementation for multi-threaded drivers, and I'm making the upcoming PyMongo 3.0 release conform to the spec as well. Mongos 2.6's replica set monitor is the reference implementation for single-threaded drivers, with a few differences. The upcoming Perl driver 1.0 implements the spec orthodoxly.

Once we have multiple reference implementations and the dust has settled, the draft spec will be final. We'll bring the rest of our drivers up to spec over the next year.

You can read more about the Server Discovery And Monitoring Spec at these links:

We have more work to do. For one thing, the Server Discovery And Monitoring Spec only describes how the client gathers information about your server topology—it does not describe which servers the client uses for operations. My Read Preferences Spec only partly answers this second question. My colleague David Golden is writing an improved and expanded version of Read Preferences, which will be called the Server Selection Spec. Once that spec is complete, we'll have a standard algorithm for all drivers that answers questions like, "Which replica set member does the driver use for a query? What about an aggregation? Which mongos does it use for an insert?" It'll include tests of the same formality and rigor as the Server Discovery And Monitoring Spec does.

Looking farther ahead, we plan to standardize the drivers' APIs so we all do basic CRUD operations the same. And since we'll allow much larger replica sets soon, both the server-discovery and the server-selection specs will need amendments to handle large replica sets. In all cases, we'll provide a higher level of rigor, clarity, and formality in our specs than we have before.

Space Shuttle Discovery

[Space Shuttle Discovery]

Shuso Hossen, Summer 2014

On August 24, 2014, Bokushu gave his first dharma talk at the Village Zendo's summer meditation retreat. I'm next: I'll be the Zendo's practice leader this winter and give my first talk in March, so I received a fan and chanted a poem during [...]

On August 24, 2014, Bokushu gave his first dharma talk at the Village Zendo's summer meditation retreat. I'm next: I'll be the Zendo's practice leader this winter and give my first talk in March, so I received a fan and chanted a poem during the ceremony.

Photos copyright Ian Darson, all rights reserved.

Bokushu

Bokushu with his staff and fan.

Me

Me, chanting the poem.

View the whole photo set on Ian Darson's site.