Testing

When thinking about testing serverless functions, it’s useful to think in terms of unit tests that are performed against a function in isolation, and integration tests that test the system as a whole. Unit tests can be performed locally because they just require the code for the function, but integration tests involving SaaS can really only be performed on the deployed system (some platforms may offer local executions as part of a deployed system, which mitigates this somewhat).

Unit Testing

Lambda functions are ideally small—a few hundred lines of code at the most—taken up mostly by error handling; the happy path should be very short, or at least relatively straightforward. Thus, introducing abstractions can create a lot of code bloat. So what should serverless function unit tests look like? A serverless function, by definition, can only have side effects by using other services. Unlike traditional (read: serverfull) systems, it’s less necessary to abstract out the service invocations. There are two reasons for this:

Create an abstraction (Boto3Wrapper class) that provides factory methods for sessions, clients, and resources will enable caching, which will then reduce the overhead across successive function invocations. For example:

# in package boto3wrapper
import boto3


class Boto3Wrapper(object):
    _SESSION_CACHE = {}
    SESSION_CREATION_HOOK = None
    @classmethod
    def get_session(cls, **kwargs):
        key = tuple(sorted(kwargs.items()))
        if key in cls._SESSION_CACHE:
            return cls._SESSION_CACHE[key]
        session = boto3.Session(**kwargs)
        if cls.SESSION_CREATION_HOOK:
            session = cls.SESSION_CREATION_HOOK(session)
        cls._SESSION_CACHE[key] = session
        return session
    # similar for client and resource, using get_session to obtain
    # a session, and also caching the objects

In a function, you use it in place of directly creating Session, Client, and Resource objects:

from boto3wrapper import Boto3Wrapper


def handler(event, context):
    # replacing dynamodb = boto3.resource('dynamodb')
    dynamodb = Boto3Wrapper.get_resource('dynamodb')
    # use as normal
    table = dynamodb.Table('MyTable')

Note that since the caching is done at the class level, it persists inside a given function container between invocations.

Unit tests can then use this functionality:

import unittest2, os.path
from boto3wrapper import Boto3Wrapper


class MyTest(unittest2.TestCase):
    def setUp(self):
        def attach_placebo(session):
            path = os.path.join(
                os.path.dirname(__file__),
                'placebo')
            pill = placebo.attach(session, data_path=path)
            return session
        Boto3Wrapper.SESSION_CREATE_HOOK = attach_placebo

    def test_function_requirement_1(self):
        # perform test, Lambda function will automatically get
        # placebo injected on its sessions

This approach allows for functions to be written as concisely as possible, focusing on business logic, and letting abstraction take place at the architecture level, in the separation of code and APIs between functions.

Integration Testing

In serverless architectures, control over many—or even most—components is given up. This is generally true of using SaaS products, but with a fully serverless system, the number of points where the developer has full control is further reduced. On AWS, user code is limited to Lambda functions, API Gateway mappings, and IoT rules, which gives no ability to, for example, induce a premature shutdown of the underlying EC2 instance handling an API Gateway connection, or cause SNS to fail when invoked by an event on S3. While the compute components of serverless systems are generally stateless (a good practice), this doesn’t mean that, in a degraded system, they will meet performance requirements (e.g., latency, data loss, management of distributed transactions, etc.).

While unit testing of serverless function code is fairly straightforward, as we’ve seen above, this does not suffice for verifying that a full system is production-ready; integration testing is required. However, integration testing for serverless architectures presents a problem. For the purpose of this section, we will assume the system uses solely AWS services. How can we test the situation where DynamoDB has less-than-perfect reliability? Does our system degrade gracefully? Does our logging and monitoring system adequately inform us of the problems?

In traditional architectures, a system like Netflix’s Chaos Monkey (and related pieces of the simian army) serves this purpose, by randomly shutting down VMs and interfering with network traffic. If a system has no SaaS components, nearly every error condition can be tested this way.

Using SaaS components, we have no way to induce those components to behave abnormally. In a fully serverless system, the only control we have is over the code we put in. Given that constraint, how can we do integration testing similar to Chaos Monkey? What would Monkeyless Chaos look like?

With the starting assumption that we are using only AWS services, and the further assumption that we are using Python (just to pick a particular SDK; the requirements work for all languages), we could establish some requirements for such a system:

Requirements for Monkeyless Chaos

It should be possible to deploy the system without any of this code included at all, so that it would be impossible to use it to cause system degradation by accidental or malicious means. This system would likely use a DynamoDB table, shared by all components of the system, to satisfy requirement #3. The table name would be provided to the Lambda function through environment variables. The error specifications themselves could be provided through environment variables, but are then not adjustable at runtime.

To extend this beyond the use of AWS services, the first logical step is HTTP calls. The system should allow similar specifications for HTTP errors, and a way to inject these errors into common HTTP libraries like requests.

Credits: Ben Kehoe (@benkehoe) ***