Property Based Testing With Hypothesis

Jan 25th, 2019 - written by Kimserey with .

Property-based testing is a testing method where a property of our system is tested against multiple datasets. Today we will see how we can create property tests using Hypothesis in Python.

Property-Based Testing
Example
Hypothesis
Strategies

Property-Based Testing

Property-based testing differs from unit testing in the way one thinks of testing.

For unit testing, we define input values, expected output values and test the actual result against the expected result. We would think of different scenarios which would cover the different case of our code.
For property-based testing, instead of defining values, we would define invariants, properties of our system which never change, and test whether they hold against varied datasets.

Property-based testing does not replace unit testing. Both methods can be used to test certain aspects of our application.

Example

Let’s consider the following example:

class Address:
    """An address composed of the address and a boolean indicating whether the address is a business or residential address"""

    def __init__(self, address, is_business):
        self.address = address
        self.is_business = is_business


def is_weekend(date):
    """Return a boolean indicating whether the date argument is a weekend day."""
    return date.isoweekday() == 6 or date.isoweekday() == 7


def shift_to_next_business_day(date):
    """Shift the provided date to the next business day."""
    date = date + timedelta(days=1)

    while is_weekend(date):
        date = date + timedelta(days=1)

    return date


def get_delivery_date(shipping_date, address, delivery_duration_days):
    """Calculate a delivery date.

    If shipping_date is a weekend, it is pushed to the next business day.

    If the calculated delivery date falls on the weekend and the address is a business address,the delivery is also pushed to the next business day.
    """
    if is_weekend(shipping_date):
        shipping_date = shift_to_next_business_day(shipping_date)

    delivery = shipping_date + timedelta(days=delivery_duration_days)

    if address.is_business and is_weekend(delivery):
        delivery = shift_to_next_business_day(delivery)

    return delivery

We have a class representing an Address which is composed of the address itself plus an attribute determining whether it is a business address or a residential address.

Then we have a utility function is_weekend defining whether the date provided is a weekend date or not by looking at the isoweekday() and determining whether it is 6 for Saturday or 7 for Sunday.

Next we have a utility function shift_to_next_business_day which shift a date to the next business day by adding a day and if the result is a weekend, shift until the next Monday.

Lastly the heart of this code is the function computing the delivery get_delivery_date taking an address and a number of delivery_duration_days to cater for a standard delivery duration.

The function will compute the delivery date by adding to the duration day to shipping date. If the shipping date is a weekend, it is shifted to the next business day, and if the computed delivery date falls on the weekend and the address is a business address, the estimation is also pushed to the next business day.

Testing the Example

If we were to unit test this code, we would define input and expected result. For example, we would have a unit test checking that the shift of a date to the next business day:

def test_shifting_resulting_in_weekend_should_shift_to_business_day(self):
    value_1 = date(2019, 1, 18)
    actual_1 = shift_to_next_business_day(value_1)
    
    value_2 = date(2019, 1, 19)
    actual_2 = shift_to_next_business_day(value_2)

    self.assertEqual(actual_1, date(2019, 1, 21))
    self.assertEqual(actual_2, date(2019, 1, 21))

We predefine two values, Friday 18 Jan 2019 and Saturday 19 Jan 2019 and for both values we expect the result to be Monday 21 Jan 2019.

We can also have another unit test testing the delivery date function:

def test_shipping_date_should_be_shifted_to_business_day_before_adding_delivery_duration(self):
    ship_date = date(2019, 1, 19)        
    result = get_delivery_date(ship_date, Address('10 street', False), 3)
    self.assertEqual(result, date(2019, 1, 24))

We predefine a ship_date of Friday 19 Jan 2019, we also predefine a delivery duration of 3 days therefore we expect the delivery date to be Thursday 24 Jan 2019 (Friday 19 being shifted to Monday 21 then adding 3 days of delivery duration resulting in Thursday 24).

Although those two unit tests pins down the behavior of our functions by setting up actual and expected values, there could be more scenarios which would not fall within the umbrella of those two unit tests. What we have done so far can be thought as:

When I provide the following values, this is how I expect my system to behave.

Another way of thinking about testing is by defining the invariants of our system. Te thought process then becomes:

Those are the invariants of my system, they are expected to hold against any circumstances

Hypothesis

Hypothesis provides the capabilities to write property-based tests. It provides a way to generate random data, called strategies, and runs our test for n iterations with random data on each iterations where n=100 by default.

pip install hypothesis

As we seen already, the key of testing properties is to define the properties themselves. In our example, shift_to_next_business_day exposes the following properties:

The next business day should never be a weekend day,
And obviously the the next business day should be a future date.

Therefore to test that, we start by importing the following from hypothesis:

from hypothesis import given
from hypothesis.strategies import dates

given is a decorator used on top of a test. It has as effect to generate the arguments of the function using hypothesis. For example here we can validate our property:

@given(dates())
def test_shift_to_next_business_day_always_fall_on_future_business_day(self, date):
    result = shift_to_next_business_day(date)
    result_weekday = result.isoweekday()

    self.assertNotEqual(result_weekday, 6)
    self.assertNotEqual(result_weekday, 7)
    self.assertGreater(result, date)

We use given as a decorator @given(...) and provide it dates() which is a date strategy. From hypothesis.strategies we imported dates which is a function constructing a DateStrategy. This function allows us to parameterize the strategy with a min/max value if we wanted to. For example we could constrain the date to be on a range from 1970 to 2100 as other years won’t make much sense. Therefore we could use the following decorator:

@given(dates(min_value=date(1970, 1, 1), max_value=date(2100, 1, 1)))

With this test in place, hypothesis will generate random dates and run the test. By default hypothesis runs each test 100 times. This value can be changed from the settings decorator by providing max_examples=1000 for example.

If we were to have a problem in our code, for example let’s say that we forgot to move forward of one day.

def shift_to_next_business_day(date):
    """Shift the provided date to the next business day."""
    # Let's say we forgot this line
    # date = date + timedelta(days=1)

    while is_weekend(date):
        date = date + timedelta(days=1)

    return date

Our test will fail and we would get the following message:

.Falsifying example: test_shift_to_next_business_day_always_fall_on_future_business_day(self=<test_app.DeliveryDateTests testMethod=test_shift_to_next_business_day_always_fall_on_future_business_day>, date=datetime.date(1999, 12, 31))

You can reproduce this example by temporarily adding @reproduce_failure('4.0.0', b'AAAAAQ==') as a decorator on your test case
FF.
======================================================================
FAIL: test_shift_to_next_business_day_always_fall_on_future_business_day (test_app.DeliveryDateTests)
----------------------------------------------------------------------

This provides us the name of the test that fails test_shift_to_next_business_day_always_fall_on_future_business_day and the argument provided to that test when it failed, self=<...> and date=date(1999, 12, 31). Because all example data are generated randomly, if we want to reproduce this exact failure, we can use the decorator reproduce_failure by copying the exact decorator provided in the error above.

@reproduce_failure('4.0.0', b'AAAAAQ==')

4.0.0 refer to hypothesis version and b'AAAAAQ==' refers to the data blob - but we don’t need to know that, we just need to use what the error provides.

Strategies

We used dates strategy. A strategy is a way of generating random data. In more details, a strategy is composed of a generator, which generates random data, and a shrinker which shrinks data to a smaller size data where smaller is subjective to the datatype it is shrinking.

There are many strategies already predefined by hypothesis for example we can import booleans to generate booleans, characters for random characters, integers and texts.

from hypothesis import given
from hypothesis.strategies import booleans, characters, composite, integers, text, dates

But there will a time where we want to compose objects so that they can be generated for the test. For example if we were the following properties:

Delivery date generated never fall on weekend when the address is a business address
Expected delivery date is always greater than shipping date

@given(dates(), address(), integers(min_value=1, max_value=30))  # pylint: disable=no-value-for-parameter
def test_delivery_date_should_not_fall_on_weekend_for_business_address(self, shipping_date, address, duration_days):

    result = get_delivery_date(shipping_date, address, duration_days)

    if address.is_business:
        self.assertNotEqual(result.isoweekday(), 6)
        self.assertNotEqual(result.isoweekday(), 7)

    self.assertGreater(result, shipping_date)

For this test to work, we need to have a way to generate address. To create the strategies that we need, we can compose existing strategies together using the @composite decorator. A composite strategy takes a draw function as first argument which can be used to draw values from strategies.

@composite
def address(draw):
    character_strategy = characters(
        max_codepoint=1000, blacklist_categories=('Cc', 'Cs'))
    text_strategy = text(character_strategy, min_size=1).map(
        lambda s: s.strip()).filter(lambda s: len(s) > 0)

    address = draw(text_strategy)
    is_business = draw(booleans())
    return Address(address, is_business)

Here we created an address composite strategy which uses characters, texts, and booleans to build an Address. We could also specify arguments to configure the strategies used within the composite and pass the argument as we would for other strategies address(addr_min_length=10).

Lastly we can see that if we need to execute simple operation, we can adapt strategies using map and filter or flatMap. Here we transformed the texts strategy by removing whitespace with .map(lambda s: s.strip()) and then filtering out text with no characters with .filter(lambda s: len(s) > 0).

What we endup with are two property tests which covers a larger ground than our unit tests as random values are generated and tested against them 100 times on each test.

hypothesis being a powerful tool, we need to make sure that we leverage its power in the best way possible. There are two common mistakes to make when writing property-based test:

Constructing strategies which are restricting the umbrella of values until the point that it becomes a unit test. If this is actually the need, writing a unit test would be less complex,
Writing a test so wide that the properties that needs to be tested never get proper input data, resulting in the property never being tested.

Remembering those two problems when writing the tests and building the strategies will ensure that we have useful tests!

Conclusion

Today we saw what is property-based testing by looking into a concrete example and comparing our approach in regards to unit tests and property-based tests. We then moved on to implement the example with hypothesis, a python library containing all the necessary tools. Lastly we saw how we could use hypothesis strategies to generate sample data and how we could compose our own strategies out of built-in strategies. I hope you liked this post! See you on the next one!