Black Pipe Testing A Connected Application In C
This is the fourth article in my series on "black pipe" testing. Here I describe testing libmongoc (the MongoDB C Driver) as a black pipe.
Like any network client library, libmongoc cannot be fully tested as a black box. Traditional black box tests enter some input and check the output—this only validates one side of the system at a time. But libmongoc has two sides, working in concert. One side is its public API, its structs and functions and so on. The other is its communication over the network with the MongoDB server. Only by treating it as a black pipe can we fully test its two sides.
Origin
I began thinking about black pipe testing early this year. I was reading the libmongoc test suite in preparation for taking over the project from Christian Hergert and Jason Carey, and I came across Christian's mock_server_t struct. Test code in C does not ordinarily make lively reading, but I woke up when I saw this. Had he really written a MongoDB wire protocol server in order to test the client library?
If you know Christian Hergert's work, you know the answer. Of course he had. His mock server listened on a random TCP port, parsed the client's network messages, and sent MongoDB responses. At the time, mock_server_t used callbacks: you created a mock server with a pointer to a function that handled requests and chose how to reply. And if you think callbacks are ungainly in Javascript or Python, try them in C.
Despite its awkward API, the mock server was indispensable for certain tests. For example, Christian had a mock server that reported it only spoke wire protocol versions 10 and 11. Since the latest MongoDB protocol version is only 3, the driver does not know how to talk to such a futuristic server and should refuse to, but the only way to test that behavior is by simulating the server.
Besides the protocol-version test, Christian also used the mock server to validate the client's handling of "read preferences". That is, how the client expresses whether it wants to read from a primary server, a secondary, or some subtler criterion. A mock server is required here because a correct client and a buggy one appear the same at the API level: it is only when we test its behavior at the network layer that bugs are caught.
In these two tests I saw the two use cases for "black pipe" testing. First, black pipe tests simulate unusual server behavior and network events. Second, in cases where the client's API behavior can appear correct even when there are bugs at the network layer, black pipe tests validate the network-level logic too.
Evolution: from C to Python
I had not yet taken leadership of libmongoc—I was finishing up some Python work. So, inspired by Christian's idea, I wrote a mock server in Python, called MockupDB. MockupDB is the subject of my earlier article in this series: "Testing PyMongo As A Black Pipe."
Since I was working in my native tongue Python, I could afford to be finicky about MockupDB's interface. I didn't want callbacks, dammit, I wanted to make something nice! As I wrote in the MockupDB article, I came up with a future-based programming interface that let me neatly interleave client and server operations in a single test function:
from mockupdb import MockupDB, Command, go
from pymongo import MongoClient
def test():
server = MockupDB(auto_ismaster={"maxWireVersion": 3})
server.run()
client = MongoClient(server.uri)
collection = client.db.collection
future = go(collection.insert_one, {"_id": 1})
request = server.receives(Command({"insert": "collection"}))
request.reply({'ok': 1})
assert(future().inserted_id == 1)Let's break this down. I use MockupDB's go function to start a PyMongo operation on a background thread, obtaining a handle to its future result:
future = go(collection.insert_one, {"_id": 1})The driver sends an "insert" command to the mock server and blocks waiting for the server response. I retrieve that command from the server and validate that it has the expected format:
request = server.receives(Command({"insert": "collection"}))MockupDB asserts that the command arrives promptly and has the right format before it returns the command to me. I reply to the client, which unblocks it and lets me retrieve the future value:
request.reply({'ok': 1})
assert(future().inserted_id == 1)More evolution: from Python back to C
Once Bernie Hackett and I released PyMongo 3.0, I devoted myself to libmongoc full-time. I set to work updating its mock_server_t with the ideas I had developed in Python. I wrote an example with the API I wanted:
mock_server_t *server;
mongoc_client_t *client;
mongoc_collection_t *collection;
bson_t *document;
bson_error_t error;
future_t *future;
request_t *request;
/* protocol version 3 includes the new "insert" command */
server = mock_server_with_autoismaster (3);
mock_server_run (server);
client = mongoc_client_new_from_uri (mock_server_get_uri (server));
collection = mongoc_client_get_collection (client, "test", "collection");
document = BCON_NEW ("_id", BCON_INT64 (1));
future = future_collection_insert (collection,
MONGOC_INSERT_NONE,/* flags */
document,
NULL, /* writeConcern */
&error);
request = mock_server_receives_command (server, "test", MONGOC_QUERY_NONE,
"{'insert': 'collection'}");
mock_server_replies_simple (request, "{'ok': 1}");
assert (future_get_bool (future));
future_destroy (future);
request_destroy (request);
bson_destroy (document);
mongoc_collection_destroy(collection);
mongoc_client_destroy(client);
mock_server_destroy (server);Alas, C is prolix; this was as lean as I could make it. I doubt that you read that block of code. Let's focus on some key lines.
First, the mock server starts up and binds an unused port. Just like in Python, I connect a real client object to the mock server's URI:
client = mongoc_client_new_from_uri (mock_server_get_uri (server));Now I insert a document. The client sends an "insert" command to the mock server, and blocks waiting for the response:
future = future_collection_insert (collection,
MONGOC_INSERT_NONE,/* flags */
document,
NULL, /* writeConcern */
&error);The future_collection_insert function starts a background thread and runs the libmongoc function mongoc_collection_insert. It returns a future value, which will be resolved once the background thread completes.
Meanwhile, the mock server receives the client's "insert" command:
request = mock_server_receives_command (server,
"test", /* DB name */
MONGOC_QUERY_NONE, /* no flags */
"{'insert': 'collection'}");This statement accomplishes several goals. First, it waits (using a condition variable) for the background thread to send the "insert" command. Second, it validates that the command has the proper format: its database name is "test", its flags are unset, the command itself is named "insert", and the target collection is named "collection".
The test completes when I reply to the client:
mock_server_replies_simple (request, "{'ok': 1}");
assert (future_get_bool (future));This unblocks the background thread. The future is resolved with the return value of mongoc_collection_insert. I assert that its return value was true, meaning it succeeded. My test framework detects if future_get_bool stays blocked: this means mongoc_collection_insert is not finishing for some reason, and this too will cause my test to fail.
Conclusion
When I first saw Christian Hergert's mock_server_t its brilliance inspired me: To test a MongoDB client, impersonate a MongoDB server!
I wrote the MockupDB package in Python, and then I overhauled Christian's mock server in C. As I developed and used this idea over the last year, I generalized it beyond the problem of testing MongoDB drivers. What I call a "black pipe test" applies to any networked application whose API behavior and network protocol must be validated simultaneously.
