Testing Tools with Commands Using Mockable

Published: September 6, 2024
Written by:

Natan Rolnik

Sponsored by

Get the best apps for your Mac

Setapp is a platform that provides a wide range of quality software for Mac users. The membership gives you access to a large library of curated apps and tools for Mac, web and iOS. My favorite ones are: Tableplus, Cleanshot X, Proxyman, and Bartender.

Check out Setapp

In a previous post, we covered how to build a tool that runs system commands, by taking advantage of the Command package instead of using the Process API.

When building reliable, solid and predictable software, adding a comprehensive tests suite is essential. In this case, tests should cover how your tool should behave when system commands fail. The more prepared your tool is for facing potential errors, the better the developer experience will be.

In this post, you will learn three things:

1. How to design a tool that exit by throwing errors and still test it
2. How write a “regular”, protocol-based mock runner
3. How the Mockable package and its @Mockable macro can facilitate writing mocks.

Designing Code Flows for Testing

In a new executable, one might be tempted to write the following code in the main function:

static func main() async throws {
    try await Self().longRunningTask()
}

func longRunningTask() async throws {
    // run your task that might fail
}

This is not ideal from two reasons. First, the tool will not be aware that there was a failure. If longRunningTask throws an error, the output of your CLI will be something along these lines:

... Swift/ErrorType.swift:253: Fatal error: Error raised at top level: ...

That’s far from a great developer experience, as the error message might not be clear to the user, without any instructions on what to do next.

Secondly, not only this code might be hard to scale when adding multiple commands, but it’s also hard to test. Having code performing a task in the same place where a command is ran doesn’t allow to easily test the outcomes.

Compare the example above with the following one:

static func main() async throws {
    do {
        let logs = try await FetchService().fetch()
        try await RunPersistenceService().persistLogs(logs)
    } catch {
        // Now, print the error message with more context and
        // exit with an error code
    }
}

// In other files

struct FetchService {
    func fetchLogs() async throws -> [RunLog] {
        // Fetch logs...
    }
}

struct RunPersistenceService {
    func persistLogs(_ logs: [RunLog]) async throws {
        // Persist logs...
    }
}

This is much better than the previous. The executable won’t crash in runtime, and each operation is isolated and more organized a different service. And the best part, the logical parts can be tested separately!

What to Test

This post will continue using the sample project from the previous post about command: a CLI tool that pulls a Swift docker image for the currently selected Swift version.

Download the sample project

While in the previous post the sample project contained the starter and with-command-runner directories, in this one we added a new one to start from: testing-starter. Additionally, following what’s explained in the previous section, the pullDockerImage method was extracted into a new struct, DockerPuller, to help writing the tests.

If you open the Package.swift file in that directory, you’ll notice that there’s a new tests target configured. Included in it, there’s a single method in DockerPullerTests, and it tests the expected behavior when docker isn’t available in the user’s machine.

In the DockerPuller.pullImage(named:) method, you can see that there’s a special error pattern matching in the do-catch statement:

func pullImage(named name: String) async throws {
    do {
        // Call `docker pull name`
    } catch CommandError.executableNotFound {
        throw Error.dockerUnavailable
    } catch {
        throw error
    }
}

This is what we’ll focus on testing in this post: that when docker is unavailable, the tool transforms the error from the command (CommandError) and throws another error that it owns.

The Standard Mock Approach

The DockerPuller service requires a runner, that conforms to the CommandRunning protocol. In the real usage, it will be an instance of the concrete CommandRunner implementation. Declaring the property as a protocol type allows us writing a test that simulates not having Docker installed, with a mock runner:

//1
private struct MockCommandRunner: CommandRunning {
    func run(
        arguments: [String],
        environment: [String : String],
        workingDirectory: AbsolutePath?
    ) -> AsyncThrowingStream<Command.CommandEvent, any Error> {
        //2
        .init { continuation in

            //3
            if arguments.first == "docker" {
                continuation.finish(throwing: CommandError.executableNotFound("docker"))
            }

            //4
            continuation.finish()
        }
    }
}

Explaining this code step by step:

1. Create a mock runner, that implements the CommandRunning protocol and its single method: run(...)
2. Initialize an AsyncThrowingStream compatible with the return type. Leverage the compiler and use the shorthand .init
3. If the first argument (i.e. the executable) equals to docker, finish the stream with by throwing the CommandError.executableNotFound error
4. Otherwise, just finish the stream.

Once the mock runner is ready, you can implement the function using it:

func test_no_docker() async throws {
    //1
    let puller = DockerPuller(runner: MockCommandRunner())

    //2
    await XCTAssertThrowsErrorAsync(try await puller.pullImage(named: "any-image")) { error in
        //3
        guard let error = error as? DockerPuller.Error else {
            XCTFail("DockerPuller should have thrown a DockerPuller.Error")
            return
        }

        //4
        XCTAssertEqual(error, .dockerUnavailable)
    }
}

This test might be scary, especially if you’re less proficient with tests. But breaking it down it becomes easier to understand:

1. Initialize a DockerPuller using the mock runner you just created
2. Use the XCTAssertThrowsErrorAsync helper method to assert that pullImage indeed throws an error
3. Assert that the error is of type DockerPuller.Error, and not CommandError or any other
4. Finally, assert that the error is the .dockerUnavailable case.

Running the Test

Now, open Terminal in the package directory. As this is the only test target, you can run the swift test command without any arguments:

swift test

You’ll see the output confirms the method threw an error.

Test Suite 'docker-swift-pullPackageTests.xctest' passed

To check that the test would fail (otherwise it wouldn’t be a good test), you can comment throwing the error in the mock, and just calling continuation.finish(). In that case, running the test target would output the following:

error: -[docker_swift_pull_tests.DockerPullerTests test_no_docker] : failed

Testing with the @Mockable Macro

Although the protocol-based mock works, there’s an even better way.

Imagine if you want to test different contexts and scenarios: you would need a mock for each case, or alternatively you would need to write a mock that can have different behaviors, by using a closure and modifying it in every test case. But even so, imagine having to create a mock implementation for each different protocol, again and again!

Fortunately, someone thought about that before and created a package that serves exactly this purpose! Making use of the powerful capabilities that Swift Macro and the code generation feature it enables, Kolos Foltányi developed a very useful package: Mockable.

How Mockable Works

For any given protocol that you want to generate a mock for, annotate it with the @Mockable macro. Imagine you are creating a protocol for fetching remote logs:

@Mockable protocol LogFetching {
    func fetch(logLevel: LogLevel) async throws -> [String]
}

Once your package depends on Mockable, and defines the MOCKING compilation condition (only in debug, so the mocks aren’t compiled for release), you’ll have this:

As you can see in the video, the macro creates automatically a concrete class, MockLogFetching that implements the protocol LogFetching.

From there, you can start using it in tests.

Transitioning from the Mock Runner

Back to the sample project, now you can get rid of the manual MockCommandRunner implementation. After deleting it, you can remove the first line of the test_no_docker method (where DockerPuller is initialized), and replace it with the following code:

//1
let mockRunner = MockCommandRunning()

//2
given(mockRunner)
    //3
    .run(
        arguments: .matching { args in args.first == "docker" },
        environment: .any,
        workingDirectory: .any
    )
    //4
    .willReturn(AsyncThrowingStream<CommandEvent, Error> {
        throw CommandError.executableNotFound("docker")
    })
//5
let puller = DockerPuller(runner: mockRunner)

The code above is a bit more complex than the previous one, but it’s more powerful and flexible. Breaking it down:

1. Create a mock runner using the generated MockCommandRunning class. It can be shared across multiple tests.
2. Use the given method to create a builder of the mock, which allows defining the behavior of a function call given certain arguments
3. As DockerPuller calls the run() function in its runner, define the behavior for this function, when it matches the conditions we’re intersted in. In this case, when the first argument is docker
4. In this case, when the first argument is docker, the mock runner will throw the CommandError.executableNotFound error
5. Finally, initialize the DockerPuller with the new mock runner.

For every function, you pass a Parameter enum case, which can be either (1) a specific value, (2) any value, or (3) a closure that you can use to define a custom condition. The former is what we’ll use here, as we want to define the behavior for the call only when docker is the first argument. This allows you to define different behaviors based on the arguments a method call receives, for any or more specific arguments.

If you run the test again, with swift test, you’ll see that the tests succeed! In the same way as the earlier example, you can play around with the values in the mock to see if the test fails.

Although CommandRunning has only one method, Mockable can be used with protocol that defines properties as well. Besides that, you can also use other methods on the generated mock, such as verifying that invocations were called as expected.