Born Sleepy

May 18, 2013

“Nigel Farage has a point”… isn’t a phrase that I thought I’d ever find myself thinking, let alone uttering in public.

There’s no doubt that there’s a delicious irony in Mr UKIP finding himself the victim of some - what shall we call them - intolerent views, robustly delivered.

There’s also no doubt that he’s milking it for everything that it’s worth.

As for UKIP themselves, and their views, I have no time for them*.

I’ve also no idea whether he was really barracked by genuine students, or a bunch of anti-English idiots (see here for my views on that sort of thing).

However, I do think that we mustn’t lose sight of the fact that Scottish nationalism is, well, nationalism.

more...

In a blog post last month I introduced Mock Server, an Objective-C toolkit to help with the unit testing of networking code.

Basically, Mock Server works by running on a local network port, pretending to be some sort of server (ftp, http, you name it). You point your networking code at it in your unit tests, thus avoiding the need to be connected to the network, and ensuring that you always have an available, predictable server to test against.

Since then I’ve been using MockServer a fair bit, and have expanded it’s capabilities somewhat, with hopefully more to come.

To help everyone else use it, in this post, I thought I’d go into a bit more detail about how exactly to set up a unit test to talk to Mock Server.

Using KMSTestCase

MockServer can be used in a number of different ways, but to simlpify things I’ve tried to make it easy to use it in what I think is the most likely scenario: as part of a suite of unit tests using the standard SenTestKit framework that ships with Xcode.

To do this, you need to make just two things:

  • a responses file
  • a unit test class that inherits from KMSTestCase.

Response file examples for ftp, http and webdav can be found in the MockServer project.

In a future post I’ll talk about how to set up your own response file to mimic a custom protocol, or to mimic something like an FTP server behaving in a very specific way (great for testing those obscure bugs that only show up on really weird FTP servers).

For now though, I’ll just use the example FTP response file, which provides responses for most of the common commands than an FTP client is likely to throw at an FTP server.

Imports

After making a new source file, you first need to import the MockServer headers that you need. You’ll definitely need KMSTestCase.h, and probably also KMSServer.h:

#import "KMSTestCase.h"
#import "KMSServer.h"

A Class And A Test

Next, you want to declare your unit test class, inheriting from KMSTestCase:

@interface KMSCollectionTests : KMSTestCase

@end

Set Up Server

As a simple example, we’ll just implement a single test, which performs an FTP request.

@implementation KMSCollectionTests

- (void)testFTP
{

First, we need to set up a new mock server instance, and start it running.

Luckily KMSTestCase has a method that does all the hard work for us: [KMSTestCase setupServerWithResponseFileNamed:].

This method takes the name of a response file, and sets up a MockServer instance, running on a dynamically allocated port, and using the “default” set of responses from the response file, which should be added as a resource to your unit test bundle.

We have to check the result of this call (which is a BOOL), in case anything goes wrong with the setup.

	// setup a server object, using the ftp: scheme and taking
	// the "default" set of responses from the "ftp.json" file.
	if ([self setupServerWithResponseFileNamed:@"ftp"])
	{

Because we’re going to fake a download, the next thing we have to do is to give the server some data to return to us when it pretends to respond to the download request.

This is the way MockServer works generally. It doesn’t actually perform as a server at all in any real sense; instead, you give it the thing you want it to return, then run the real client code that sends a request to it, then check that the client code got back the thing that you asked MockServer to return.

So, to set some data to be returned, we use the [KMSServer data] property:

        // set up the data that the server will return
        NSString* testData = @"This is some test data";
        self.server.data = [testData dataUsingEncoding:NSUTF8StringEncoding];

Make A Request

Next, we need to make an NSURLRequest object that we’re going to use in our client code to do our actual FTP request.

It’s important to note here that we don’t have to use NSURLRequest, or NSURLConnection, or any particular client-side technology. We can formulate the network request in whatever way we like, and can use any sort of third-party library (or even low level socket code) to perform the client part of the transaction (which is the part that we’re trying to test).

However, I’m using NSURLConnection here for simplicity, so I need an NSURLRequest, which means that I first need an NSURL object, including the address of the server and the path of the file we want to download.

We know that the server is running locally (as that’s the point of MockServer). However, because the server is using a dynamically allocated port, we can’t just hard-code the URL into the test; we have to figure it out on the fly. We do this by grabbing the port number back from the server, grabbing the scheme from the responses collection, and building up a URL from that information.

Luckily KMSTestCase provides a helper method to simplify this.

In this case we’re going to pretend to download a file called “test.txt” from the root of the FTP server:

		// setup an ftp request
		NSURL* url = [self URLForPath:@"test.txt"];
		NSURLRequest* request = [NSURLRequest requestWithURL:url];

Perform The Download

Next, we want to actually peform the download.

Because this is a unit test, the first instinct might be to do this synchronously. After all, a unit test is just one method, and we can’t perform a test on what we got back until we’ve actually got it. For an asynchronous case we’d have to give back control to the main loop for a while until we somehow know that the request is done, and that all sounds a bit complicated. There’s a danger that the unit test would just finish before anything had happened.

To test in real-world conditions though, we really do want to do things asynchronously. A synchronous test at this point really isn’t a good idea, since (hopefully) we aren’t going to be writing synchronous downloads in our apps.

Luckily, MockServer and KMSTestCase have this covered. KMSTestCase has a two methods: runUntilPaused, and pause.

The first of these hands back control to the main run loop, and pumps it until something calls pause. If we arrange to call this in our completion handler, we can happily set up an asynchronous request, wait for it to do it’s thing, and then return control to our unit test so that we can check the results.

Here’s the code:

	__block NSString* string = nil;
	
	[NSURLConnection sendAsynchronousRequest:request 
		queue:[NSOperationQueue currentQueue] 
		completionHandler:
			^(NSURLResponse* response, NSData* data, NSError* error)
			 {
			     if (error)
			     {
			         NSLog(@"got error %@", error);
			     }
			     else
			     {
			         string = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
			     }
		 
			     [self pause];
			 }];
	 
	[self runUntilPaused];

Test The Results

Finally, we can check that we got back whatever it is that we were expecting to get back.

In this case we should receive the test string that we asked MockServer to return to us:

    STAssertEqualObjects(string, testData, @"got the wrong response: %@", string);

And that, as they say, is that.

By using KMSTestCase, most of the setup work and all of the cleanup work is done for us, and we can just concentrate on the code that performs whatever network operation it is that we’re trying to test, and then checks the results to verify that they are ok.

Clearly there’s more going on under the hood of KMSTestCase, but most of it is just housekeeping and for a lot of situations it should be sufficient for your needs.

You can obviously use KMSServer directly if you need to do something more complicated - examining the source code of KMSTestCase should give you everything you need.

For More Info

You can find full source code and documentation for MockServer on github.

more...

Once or twice I’ve had the situation where I ran some unit tests in Xcode, and they seem to run incredibly fast. Too fast.

Spoiler: they didn’t

more...

Unit testing is great, but when you’re working on networking code, it can be a bit of a pain in the, erm, transport layer.

I’ve been doing quite a bit of network-related code for Karelia recently, and I wanted to be able to unit test, so this became an issue for me.

If there are two things that you definitely want from a suite of unit tests, it’s that they run fast, and that they are consistent. You want to be able to fire off the tests regularly, you want to be able to trust the results, and you want to be able to run them anywhere.

Throw a network server or two into your testing mix, and these bets are generally off. If the server is internal, the tests may not run when you find yourself on the wrong side of a firewall. If the server is public, you probably have no control over it. In any case, you have no control over the bit of soggy string between the test machine and the server, and any number of problems could cause your tests to fail spuriously.

This problem gets even worse when you consider that one of the best uses of unit tests is to check that your code copes when something goes wrong. Networking code is very good at going wrong in exciting and unexpected ways. Making real servers go wrong on demand in exactly the same way each time so that you can check that you handle it is challenging.

Typically the solution to this problem is some sort of “mocking” - i.e. finding some API or object that you can swap out or inject into the system in such a way that your tests can work as if they are talking to a real server, whilst actually talking to something else.

One approach is to somehow intercept the network request and, instead of passing it on to a real server, fake up the relevant response and return it directly. In the Cocoa world, a custom NSURLProtocol is a good way to achieve this, and it’s the approach taken by things like Nocilla. If you can work at such a high level, this is a nice solution as no actual networking need ever take place. You just intercept the URL request and deliver back an NSData or NSError object depending on what you’re trying to simulate.

The only problem for me was that this approach relies on the networking going via an NSURLConnection, and not all of the code that I wanted to test did this - I was testing code using libcurl, for example.

So I needed another approach, which would work with any networking code without me modifying it.

The second solution is essentially to implement a real (albeit very cut down) server. Bind a local port, listen on it for a connection, and respond to it as if you were a server.

Providing that you’re happy to require that your tests all connect to a custom port on localhost, rather than to real addresses, this solution is infinitely flexible.

There’s only one problem… the bit where I said “respond to it as if you were a server”. Isn’t writing servers a bit tricky?

Well, yes, if you want to write an actual server. But if you think about it, most interactions with servers are just a series of query, response, query, response. In a testing scenario you know what network operation you’re going to do, so you should be able to figure out what query the underlying code will actually send to the server. All you need to do then is to arrange for the right responses to come back.

Even better, in a testing scenario you generally only have to deal with one request happening at once (not always quite true, but near enough a lot of the time).

Subsequently I’ve discovered that there are a few solutions out there that do this sort of thing, but often they seem to just pretend to be a particular kind of server, most commonly a web server. I needed something that could pretend to be an FTP server, a web server, do WebDAV, maybe do SFTP, etc.

More to the point, being a reckless sort, I thought: how hard can it be?

I decided to find out, and thus MockServer was born.

MockServer

In a nutshell, the way MockServer works is this:

  • at the start of your unit test, you make a KMSServer object, and give it a responder (more on these later)
  • you then fire off the network operation that you want to test, but instead of pointing it at a known port like 80 on a real server you point it at localhost, at the port that the KMSServer object told you to use
  • you now tell the KMSServer object to run until your operation is complete - this pauses your test
  • once your operation has completed (or failed), your normal completion or error routines will be called
  • in your completion routine you tell the KMSServer object to pause
  • pausing the KMSServer object returns control to your test, and you can check for the right responses.

Internally, MockServer just opens up a port and listens for connections.

When it gets a connection, it opens up a socket and waits for input.

When it gets input, it passes it off to a responder, which matches it against things that it knows about, and sends back some commands. Mostly of these commands are just “send xyz back to the client”, but they can also include other things like “wait for x seconds”, or “close the connection”.

There is a bit more cleverness going on (for example, MockServer can fake the second connection that an FTP transfer requires), but essentially that’s it.

Responders

As I mentioned above, for the purposes of testing, most server interactions are just a series of query, response, query, response.

MockServer models this query/response behaviour using an abstract KMSResponder class - an instance of which you give to the server object at startup. So depending on the actual responder class, you can pretend to be any kind of server you like.

In real life some servers rely on a lot of state in order to give the correct response to a given query. In theory a responder could keep any state it liked, to deal with this problem. You could even genuinely implement an actual server protocol as a responder class if you were mad enough.

However, for testing this often isn’t necessary. MockServer supplies the KMSRegExResponder class, which is based on performing regular expression pattern matching on the input that it receives, and choosing from a series of stock responses.

It turns out that you can cover an awful lot of ground with this approach alone. Especially when you add in the ability to perform some limited substitution into the stock responses - things like inserting the current time in the correct format, or including part of the query you matched in your reply.

Responses

So now the meat of testing a particular network interaction comes down to identifying the queries that are going to arrive at the server, and arranging the right responses to send back.

This is where some low level knowledge of the underlying network protocol becomes useful, but even here it turns out that a lot of work can be done by trial and error.

You can configure MockServer so that it will log out everything it receives, so when it comes to figuring out the query, you can literally watch to see what arrives, and then write a pattern to match it.

Working out what to send back is slightly harder, as you can’t just watch MockServer, you have to actually do something yourself. The trick here though is to watch a real server, either by sniffing a network connection with it, or by using a tool such as Transmit and examining the log, or even by opening up a telnet session with the real server and pretending to be the client.

The good news is that a lot of this work only needs to be done once or twice for each kind of server.

You ask yourself “how does an FTP client send the password, and what does the server do?”.

Once you know that the client sends PASS thisismypass, and the server responds with 230 User user logged in. or 530 Login incorrect., you can just set up these a couple of alternate responses to match against a pattern like PASS (\w+). Normally you’ll want the first response, but if you want to test your error handling for the wrong password, you use the second one instead.

To support this sort of thing, MockServer provides the KMSResponseCollection class, which lets you define a series of patterns and responses in a JSON file, and then group them into higher level sets of responses.

So far in the course of my own tests, I’ve developed a couple of the JSON files - one for FTP and one for WebDAV, with quite a few useful responses in them. These files are included with MockServer, and can be used as the basis for your own tests. As time goes by I expect that I’ll add more response files for other protocols.

An Example Test

This is a simple example of how you might set up an FTP test, using the FTP responses supplied with MockServer.

#import "KMSTestCase.h"
#import "KMSServer.h"

@interface ExampleTest : KMSTestCase
@end

@implementation ExampleTest

- (void)testFTP
{
    // setup a KMSServer object, using the ftp: scheme and taking the "default" set of responses from the "ftp.json" file.
    if ([self setupServerWithScheme:@"ftp" responses:@"ftp"])
    {
        // set up the data that the server will return
        NSString* testData = @"This is some test data";
        self.server.data = [testData dataUsingEncoding:NSUTF8StringEncoding];

        // setup an ftp request for an imaginary file called "test.txt", on the local port that
		// our mock server is running on
        NSURL* url = [NSURL URLWithString:[NSString stringWithFormat:@"ftp://user:pass@127.0.0.1:%ld/test.txt", (long)self.server.port]];
        NSURLRequest* request = [NSURLRequest requestWithURL:url];

        // perform the request using NSURLConnection
		__block NSString* string = nil;
		[NSURLConnection sendAsynchronousRequest:request queue:[NSOperationQueue currentQueue] completionHandler:^(NSURLResponse* response, NSData* data, NSError* error)
		 {
		     if (error)
		     {
		         STFail(@"got unexpected error %@", error);
		     }
		     else
		     {
		         string = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
		     }

		     [self pause]; // this call causes runUntilPaused to return
		 }];

		[self runUntilPaused]; // we hang here until the completion block above has executed

        // check that we got back what we were expecting
        STAssertEqualObjects(string, testData, @"got the wrong response: %@", string);
    }
}

Obviously not all tests will use FTP, and not all tests will use NSURLConnection. Some networking APIs may be callback based, or require delegates. As long as you call [self pause] in your callback or delegate method, the test should behave correctly - waiting at the runUntilPaused call until the callback or delegate method has fired.

For simplicity in this test we encoded the user name and password into the request URL. More complicated examples would probably rely on some sort of NSURLCredential callback or delegate method (such as implementing connection:willSendRequestForAuthenticationChallenge:), as a way of obtaining the details to send to the server. The KMSTestCase class has handy user and password properties that you can use to store the details that your authentication method can pass back. This lets you set this information on a per-test or per-request basis, as a way of exercising the different code paths for authentication success and failure.

For More Information

This post is just a quick overview, but there is also a fair bit of documentation which you can find over on github.

Also, of course, you can look at the code.

I wrote this code for Karelia as part of some other work. Many thanks (and much kudos) to them for allowing me to open source it.

more...

In my Mac Pro Update - Think Different post a few weeks ago, I said that I thought we don’t really need a great big new Mac Pro from Apple.

What we need instead (and I admit that I don’t have an exact technical solution for this problem, but nevertheless) is a unit or units that can seemlessly enhance the processing power of a laptop when it’s on our home network.

You could argue that there’s still a need for a big box that can host all sorts of large expansion cards, but I’m not really convinced. Thunderbolt should deal with most of that.

Something else has occurred to me recently.

People have been talking about a potential switch away from Intel to ARM for the Mac range, and assuming that it would happen first with a Mac Mini or an iMac, since they are seen as “consumer”, and there’s a perception that consumer machines will be easier to switch over.

I actually think that the opposite will happen. I think that the first ARM Mac is likely to come in the guise of a new “Pro” desktop box. I also think that it’s likely to be loaded to the gills with cores, to the extent that it blows away anything out there in terms of raw processing power.

With GCD / blocks / NSOperation, Cocoa has a really nice clean model for making use of multiple cores. Better than anything I’ve used before (although admittedly I’m now a bit out of date with regard to the competition).

With the expansion of privelege separation and things like XPC, we’re also moving closer to a world where even a single app could be run seamlessly across multiple machines.

It seems to me that switching to a relatively low power, low cost, risc architecture makes perfect sense in this world.

Before you ask - I’ve absolutely no evidence for this. Just thinking aloud…

more...