A. Jesse Jiryu Davis

Read-Your-Writes Consistency With PyMongo

Photo: Thomas van de Vosse A PyMongo user asked me a good question today: if you want read-your-writes consistency, is it better to do acknowledged writes with a connection pool (the default), or to do unacknowledged writes over a single [...]

Quill Photo: Thomas van de Vosse

A PyMongo user asked me a good question today: if you want read-your-writes consistency, is it better to do acknowledged writes with a connection pool (the default), or to do unacknowledged writes over a single socket?

A Little Background

Let's say you update a MongoDB document with PyMongo, and you want to immediately read the updated version:

client = pymongo.MongoClient()
collection = client.my_database.my_collection
collection.update(
    {'_id': 1},
    {'$inc': {'n': 1}})

print collection.find_one({'_id': 1})

In a multithreaded application, PyMongo's connection pool may have multiple sockets in it, so we don't promise that you'll use the same socket for the update and for the find_one. Yet you're still guaranteed read-your-writes consistency: the change you wrote to the document is reflected in the version of the document you subsequently read with find_one. PyMongo accomplishes this consistency by waiting for MongoDB to acknowledge the update operation before it sends the find_one query. (I explained last year how acknowledgment works in PyMongo.)

There's another way to get read-your-writes consistency: you can send both the update and the find_one over the same socket, to ensure MongoDB processes them in order. In this case, you can tell PyMongo not to request acknowledgment for the update with the w=0 option:

# Reserve one socket for this thread.
with client.start_request():
    collection.update(
        {'_id': 1},
        {'$inc': {'n': 1}},
        w=0)

    print collection.find_one({'_id': 1})

If you set PyMongo's auto_start_request option it will call start_request for you. In that case you'd better let the connection pool grow to match the number of threads by removing its max_pool_size:

client = pymongo.MongoClient(
    auto_start_request=True,
    max_pool_size=None)

(See my article on requests for details.)

So, to answer the user's question: If there are two ways to get read-your-writes consistency, which should you use?

The Answer

You should accept PyMongo's default settings: use acknowledged writes. Here's why:

Number of sockets: A multithreaded Python program that uses w=0 and auto_start_request needs more connections to the server than does a program that uses acknowledged writes instead. With auto_start_request we have to reserve a socket for every application thread, whereas without it, threads can share a pool of connections smaller than the total number of threads.

Back pressure: If the server becomes very heavily loaded, a program that uses w=0 won't know the server is loaded because it doesn't wait for acknowledgments. In contrast, the server can exert back pressure on a program using acknowledged writes: the program can't continue to write to the server until the server has completed and acknowledged the writes currently in progress.

Error reporting: If you use w=0, your application won't know whether the writes failed due to some error on the server. For example, an insert might cause a duplicate-key violation. Or you might try to increment a field in a document, but the server rejects the operation because the field isn't a number. By default PyMongo raises an exception under these circumstances so your program doesn't continue blithely on, but if you use w=0 such errors pass silently.

Consistency: Acknowledged writes guarantee read-your-writes consistency, whether you're connected to a mongod or to a mongos in a sharded cluster.

Using w=0 with auto_start_request also guarantees read-your-writes consistency, but only if you're connected to a mongod. If you're connected to a mongos, using w=0 with auto_start_request does not guarantee any consistency, because some writes may be queued in the writeback listener and complete asynchronously. Waiting for acknowledgment ensures that all writes have really been completed in the cluster before your program proceeds.

Forwards compatibility with MongoDB: The next version of the MongoDB server will offer a new implementation for insert, update, and delete, which will diminish the performance boost of w=0.

Forwards compatibility with PyMongo: You can tell by now that we're not big fans of auto_start_request. We're likely to remove it from PyMongo in version 3.0, so you're better off not relying on it.

Conclusion

In short, you should just accept PyMongo's default settings: acknowledged writes with auto_start_request=False. There are many disadvantages and almost no advantages to w=0 with auto_start_request, and in the near future these options will be diminished or removed anyway.

Austin

I'm visiting my old hometown Austin this week. Tuesday night I'll be at the Austin MongoDB User Group talking about how to become a Python contributor. Wednesday night I'm speaking at the Austin Python Meetup. I'll give a shorter version [...]

S Congress Ave Austin TX

I'm visiting my old hometown Austin this week.

Tuesday night I'll be at the Austin MongoDB User Group talking about how to become a Python contributor.

Wednesday night I'm speaking at the Austin Python Meetup. I'll give a shorter version of the same talk about contributing to Python, and I'll present on async frameworks, specifically Tornado: What is it, how does it work, and what is it good for?

So come say hi if you're in Austin, and tell me in the comments: what's new in Austin in the last 10 years that I should check out during my visit?

Day Of The Thread

If you think you’ve found a bug in Python, what’s next? I'll guide you through the process of submitting a patch, so you can avoid its pitfalls and find the shortest route to becoming a Python contributor! This is the final post [...]

Day of the Thread

If you think you’ve found a bug in Python, what’s next? I'll guide you through the process of submitting a patch, so you can avoid its pitfalls and find the shortest route to becoming a Python contributor!

This is the final post in a three part series. In Night of the Living Thread I fixed a bug in Python's threading implementation, so that threads wouldn't become zombies after a fork. In Dawn of the Thread I battled zombie threads in Python 2.6. Now, in the horrifying conclusion, I return to the original bugfix and submit it to the core Python team. Humanity's last hope is that we can get a patch accepted and stop the zombie threads...before it's too late.

The action starts when I open a bug in the Python bug tracker. The challenge is to make a demonstration of the bug. I need to convince the world that I'm not crazy: the dead really are rising and walking the earth! Luckily I have a short script from Night of the Living Thread that shows the zombification process clearly.

Day of the Dead

Next I have to fix the bug and submit a patch. I'm confused here, since the bug is in Python 2.7 and 3.3: do I submit fixes for both versions? The right thing to do is clone the Python source:

hg clone http://hg.python.org/cpython

I fix the bug at the tip of the default branch. The Lifecycle of a Patch doc in the Python Developer's Guide tells me to make a patch with hg diff. I attach it to the bug report by hitting the "Choose File" button and then "Submit Changes."

After this, the Python Developer's Guide is no more use. The abomination I am about to encounter isn't described in any guide: The Python bug tracker is a version of Roundup, hacked to pieces and sewn together with a code review tool called Rietveld. The resulting botched nightmare is covered in scabs, stitches, and suppurating wounds. It's a revolting Frankenstein's monster. (And I thought this was only a zombie movie.)

When I upload a patch to the bug tracker, Roundup, it is digested and spit out into the code review tool, Rietveld. It shows up like this, so a Python core developer can critique my bugfix. Charles-François Natali is my reviewer. He suggests a cleaner bugfix which you can read about in my earlier post, and shows me how to improve my unittest.

Tragically, a week passes before I know he's reviewed my patch. I keep visiting the issue in Roundup expecting to see comments there, but I'm not looking where I should be: there's a little blue link in Roundup that says "review", which leads to Rietveld. That's where I should go to see feedback. Precious time is lost as hordes of zombie threads continue to ravage the landscape.

Day of the Dead street

Even worse, my Gmail account thinks Rietveld's notifications are spam. It turns out that the bug tracker was once breached by spammers and used to send spam in the past, so Gmail is quick to characterize all messages from bugs.python.org as spam. I override Gmail's spam filter with a new filter:

Gmail python filter

Once I make the changes Charles-François suggests, I try to re-upload my patch. Clicking "Add Another Patch Set" in Rietveld doesn't work: it shows a page with a TypeError and a traceback. So I follow the instructions to upload a patch using the upload.py script from the command line and that throws an exception, too. I can't even cry out for help: hitting "reply" to add a comment in Rietveld fails. I tremble in fear.

Day of the Dead hands

Just when humanity's doom seems inevitable, I find a way out: It turns out I must upload my new patch as an additional attachment to the issue in Roundup. Then Roundup, after some delay, applies it to the code review in Rietveld. Finally, I can address Charles-François's objections, and he accepts my patch! Roundup informs me when he applies my changes to the 2.7, 3.3, and default branches.

As the darkness lifts I reflect on how contributing to Python has benefited me, despite the horror. For one thing, I learned a few things about Python. I learned that every module in the standard library imports its dependencies like this example from threading.py:

from time import time as _time, sleep as _sleep

When you execute a statement like from threading import *, Python only imports names that don't begin with an underscore. So renaming imported items is a good way to control which names a module exports by default, an alternative to the __all__ list.

The code-review process also taught me about addCleanup(), which is sometimes a nicer way to clean up after a test than either tearDown or a try/finally block. And I learned that concurrency bugs are easier to reproduce in Python 2 with sys.setcheckinterval(0) and in Python 3 with sys.setswitchinterval(1e-6).

But the main benefit of contributing to Python is the satisfaction and pride I gain: Python is my favorite language. I love it, and I saved it from zombies. Heroism is its own reward.

Day of the Dead final

Dawn Of The Thread

In my previous post, Night of the Living Thread, I described how a dead Python thread may think it's still alive after a fork, and how I fixed this bug in the Python standard library. But what if you need to save your code from the ravenous [...]

Dawn of the thread

In my previous post, Night of the Living Thread, I described how a dead Python thread may think it's still alive after a fork, and how I fixed this bug in the Python standard library. But what if you need to save your code from the ravenous undead in Python 2.6? The bug will never be fixed in Python 2.6's standard library. You're on your own. In Python 2.6, no one can hear you scream.

Recall from my last post that I can create a zombie thread like this:

t = threading.Thread()
t.start()
if os.fork() == 0:
    # We're in the child process.
    print t.isAlive()

isAlive() should always be False, but sometimes it's True. The problem is, I might fork before the thread has added itself to the global _active list, so Python doesn't mark the thread as "stopped" after the fork. To fix this, I need t.start() to wait until the thread is completely initialized, so that I can't fork too soon.

Here's my solution, a SafeThread that will never rise from its grave:

class SafeThread(threading.Thread):
    def __init__(self, *args, **kwargs):
        super(SafeThread, self).__init__(*args, **kwargs)
        self.really_started = threading.Event()

    def start(self):
        super(SafeThread, self).start()
        self.really_started.wait()

    def run(self):
        self.really_started.set()
        super(SafeThread, self).run()

I create an event in the SafeThread's constructor. Then in SafeThread.start(), I call the standard Thread's start method, but I wait for the event before returning. Finally, I trigger the event in run(), which is executed in the new thread. By the time the standard Thread executes run(), it has added itself to _active: thus, we know it's safe to set the event and unblock the thread that called start(). Even if I fork immediately after that, the Safethread is in _active and can't become zombified.

You can imagine more complex scenarios that will still defeat me. For example, Thread A could start Thread B, and Thread C could fork at the wrong moment and leave Thread B zombified. For absolute safety from zombies, upgrade to Python 2.7.6 or 3.3.3 when they're released with my bugfix. Meanwhile, SafeThread is good enough for the common case where the main thread creates the background thread and then forks.

(Now read the gory finale, Day of the Thread, in which humanity's last hope is to figure out the quirky code review system used by the Python Software Foundation.)

Dawn of the Dead

Night Of The Living Thread

What should this Python code print?: t = threading.Thread() t.start() if os.fork() == 0: # We're in the child process. print t.isAlive() In Unix, only the thread that calls fork() is copied to the child process; all other threads are dead. [...]

What should this Python code print?:

t = threading.Thread()
t.start()
if os.fork() == 0:
    # We're in the child process.
    print t.isAlive()

In Unix, only the thread that calls fork() is copied to the child process; all other threads are dead. So t.isAlive() in the child process should always return False. But sometimes, it returns True! It's the....

Night of the Living Thread

How did I discover this horrifying zombie thread? A project I work on, PyMongo, uses a background thread to monitor the state of the database server. If a user initializes PyMongo and then forks, the monitor is absent in the child. PyMongo should notice that the monitor thread's isAlive is False, and raise an error:

# Starts monitor:
client = pymongo.MongoReplicaSetClient()
os.fork()

# Should raise error, "monitor is dead":
client.db.collection.find_one()

But intermittently, the monitor is still alive after the fork! It keeps coming back in a bloodthirsty lust for HUMAN FLESH!

I put on my Sixties scientist outfit (lab coat, thick-framed glasses) and sought the cause of this unnatural reanimation. To begin with, what does Thread.isAlive() do?:

class Thread(object):
    def isAlive(self):
        return self.__started.is_set() and not self.__stopped

After a fork, __stopped should be True on all threads but one. Whose job is it to set __stopped on all the threads that didn't call fork()? In threading.py I discovered the _after_fork() function, which I've simplified here:

# Globals.
_active = {}
_limbo = {}

def _after_fork():
    # This function is called by PyEval_ReInitThreads
    # which is called from PyOS_AfterFork.  Here we
    # clean up threading module state that should not
    # exist after a fork.

    # fork() only copied current thread; clear others.
    new_active = {}
    current = current_thread()
    for thread in _active.itervalues():
        if thread is current:
            # There is only one active thread.
            ident = _get_ident()
            new_active[ident] = thread
        else:
            # All the others are already stopped.
            thread._Thread__stop()

    _limbo.clear()
    _active.clear()
    _active.update(new_active)
    assert len(_active) == 1

This function iterates all the Thread objects in a global dict called _active; each is removed and marked as "stopped", except for the current thread. How could this go wrong?

Night of the living dead

Well, consider how a thread starts:

class Thread(object):
    def start(self):
        _limbo[self] = self
        _start_new_thread(self.__bootstrap)

    def __bootstrap(self):
        self.__started.set()
        _active[self.__ident] = self
        del _limbo[self]
        self.run()

(Again, I've simplified this.) The Thread object's start method adds the object to the _limbo list, then creates a new OS-level thread. The new thread, before it gets to work, marks itself as "started" and moves itself from _limbo to _active.

Do you see the bug now? Perhaps the thread was reanimated by space rays from Venus and craves the flesh of the living!

Night of the living dead 4

Or perhaps there's a race condition:

  1. Main thread calls worker's start().
  2. Worker calls self.__started.set(), but is interrupted before it adds itself to _active.
  3. Main thread calls fork().
  4. In child process, main thread calls _after_fork, which doesn't find the worker in _active and doesn't mark it "stopped".
  5. isAlive() now returns True because the worker is started and not stopped.


Now we know the cause of the grotesque revenant. What's the cure? Headshot?

I submitted a patch to Python that simply swapped the order of operations: first the thread adds itself to _active, then it marks itself started:

def __bootstrap(self):
    _active[self.__ident] = self
    self.__started.set()
    self.run()

If the thread is interrupted by a fork after adding itself to _active, then _after_fork() finds it there and marks it stopped. The thread ends up stopped but not started, rather than the reverse. In this case isAlive() correctly returns False.

The Python core team looked at my patch, and Charles-François Natali suggested a cleaner fix. If the zombie thread is not yet in _active, it is in the global _limbo list. So _after_fork should iterate over both _limbo and _active, instead of just _active. Then it will mark the zombie thread as "stopped" along with the other threads.

def _enumerate():
    return _active.values() + _limbo.values()

def _after_fork():
    new_active = {}
    current = current_thread()
    for thread in _enumerate():
        if thread is current:
            # There is only one active thread.
            ident = _get_ident()
            new_active[ident] = thread
        else:
            # All the others are already stopped.
            thread._Thread__stop()

This fix will be included in the next Python 2.7 and 3.3 releases. The zombie threads will stay good and dead...for now!

(Now read the sequels: Dawn of the Thread, in which I battle zombie threads in the abandoned tunnels of Python 2.6; and Day of the Thread, a post-apocalyptic thriller in which a lone human survivor tries to get a patch accepted via bugs.python.org.)

They keep coming back in a bloodthirsty lust for human flesh!

Oshin and Rebecca

This Sunday, my friends Oshin and Rebecca were married. I photographed. Oshin is a monk at my zendo, and deaf; Rebecca is a sign language interpreter and directs Columbia University's deaf services. They were married in a Jewish-tinged [...]

This Sunday, my friends Oshin and Rebecca were married. I photographed.

Oshin is a monk at my zendo, and deaf; Rebecca is a sign language interpreter and directs Columbia University's deaf services. They were married in a Jewish-tinged Zen service in an Episcopalian chapel. So if you're vulnerable to the spectacle of love uniting people despite differences, you're going to be in trouble....

Oshin rebecca 1

Oshin rebecca 2

Oshin rebecca 3

Oshin rebecca 4

Oshin rebecca 5