Jan 25th, 2019 - written by Kimserey with .
Property-based testing is a testing method where a property of our system is tested against multiple datasets. Today we will see how we can create property tests using Hypothesis in Python.
Property-based testing differs from unit testing in the way one thinks of testing.
Property-based testing does not replace unit testing. Both methods can be used to test certain aspects of our application.
Let’s consider the following example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 class Address: """An address composed of the address and a boolean indicating whether the address is a business or residential address""" def __init__(self, address, is_business): self.address = address self.is_business = is_business def is_weekend(date): """Return a boolean indicating whether the date argument is a weekend day.""" return date.isoweekday() == 6 or date.isoweekday() == 7 def shift_to_next_business_day(date): """Shift the provided date to the next business day.""" date = date + timedelta(days=1) while is_weekend(date): date = date + timedelta(days=1) return date def get_delivery_date(shipping_date, address, delivery_duration_days): """Calculate a delivery date. If shipping_date is a weekend, it is pushed to the next business day. If the calculated delivery date falls on the weekend and the address is a business address,the delivery is also pushed to the next business day. """ if is_weekend(shipping_date): shipping_date = shift_to_next_business_day(shipping_date) delivery = shipping_date + timedelta(days=delivery_duration_days) if address.is_business and is_weekend(delivery): delivery = shift_to_next_business_day(delivery) return delivery
We have a class representing an
Address which is composed of the address itself plus an attribute determining whether it is a business address or a residential address.
Then we have a utility function
is_weekend defining whether the date provided is a weekend date or not by looking at the
isoweekday() and determining whether it is
6 for Saturday or
7 for Sunday.
Next we have a utility function
shift_to_next_business_day which shift a date to the next business day by adding a day and if the result is a weekend, shift until the next Monday.
Lastly the heart of this code is the function computing the delivery
get_delivery_date taking an
address and a number of
delivery_duration_days to cater for a standard delivery duration.
The function will compute the delivery date by adding to the duration day to shipping date. If the shipping date is a weekend, it is shifted to the next business day, and if the computed delivery date falls on the weekend and the address is a business address, the estimation is also pushed to the next business day.
If we were to unit test this code, we would define input and expected result. For example, we would have a unit test checking that the shift of a date to the next business day:
1 2 3 4 5 6 7 8 9 def test_shifting_resulting_in_weekend_should_shift_to_business_day(self): value_1 = date(2019, 1, 18) actual_1 = shift_to_next_business_day(value_1) value_2 = date(2019, 1, 19) actual_2 = shift_to_next_business_day(value_2) self.assertEqual(actual_1, date(2019, 1, 21)) self.assertEqual(actual_2, date(2019, 1, 21))
We predefine two values,
Friday 18 Jan 2019 and
Saturday 19 Jan 2019 and for both values we expect the result to be
Monday 21 Jan 2019.
We can also have another unit test testing the delivery date function:
1 2 3 4 def test_shipping_date_should_be_shifted_to_business_day_before_adding_delivery_duration(self): ship_date = date(2019, 1, 19) result = get_delivery_date(ship_date, Address('10 street', False), 3) self.assertEqual(result, date(2019, 1, 24))
We predefine a
Friday 19 Jan 2019, we also predefine a delivery duration of
3 days therefore we expect the delivery date to be
Thursday 24 Jan 2019 (
Friday 19 being shifted to
Monday 21 then adding
3 days of delivery duration resulting in
Although those two unit tests pins down the behavior of our functions by setting up actual and expected values, there could be more scenarios which would not fall within the umbrella of those two unit tests. What we have done so far can be thought as:
1 When I provide the following values, this is how I expect my system to behave.
Another way of thinking about testing is by defining the invariants of our system. Te thought process then becomes:
1 Those are the invariants of my system, they are expected to hold against any circumstances
Hypothesis provides the capabilities to write property-based tests. It provides a way to generate random data, called
strategies, and runs our test for
n iterations with random data on each iterations where
n=100 by default.
1 pip install hypothesis
As we seen already, the key of testing properties is to define the properties themselves. In our example,
shift_to_next_business_day exposes the following properties:
Therefore to test that, we start by importing the following from
1 2 from hypothesis import given from hypothesis.strategies import dates
given is a decorator used on top of a test. It has as effect to generate the arguments of the function using
hypothesis. For example here we can validate our property:
1 2 3 4 5 6 7 8 @given(dates()) def test_shift_to_next_business_day_always_fall_on_future_business_day(self, date): result = shift_to_next_business_day(date) result_weekday = result.isoweekday() self.assertNotEqual(result_weekday, 6) self.assertNotEqual(result_weekday, 7) self.assertGreater(result, date)
given as a decorator
@given(...) and provide it
dates() which is a date strategy. From
hypothesis.strategies we imported
dates which is a function constructing a
DateStrategy. This function allows us to parameterize the strategy with a min/max value if we wanted to. For example we could constrain the date to be on a range from
1970 to 2100 as other years won’t make much sense. Therefore we could use the following decorator:
1 @given(dates(min_value=date(1970, 1, 1), max_value=date(2100, 1, 1)))
With this test in place,
hypothesis will generate random dates and run the test. By default
hypothesis runs each test
100 times. This value can be changed from the settings decorator by providing
max_examples=1000 for example.
If we were to have a problem in our code, for example let’s say that we forgot to move forward of one day.
1 2 3 4 5 6 7 8 9 def shift_to_next_business_day(date): """Shift the provided date to the next business day.""" # Let's say we forgot this line # date = date + timedelta(days=1) while is_weekend(date): date = date + timedelta(days=1) return date
Our test will fail and we would get the following message:
1 2 3 4 5 6 7 .Falsifying example: test_shift_to_next_business_day_always_fall_on_future_business_day(self=<test_app.DeliveryDateTests testMethod=test_shift_to_next_business_day_always_fall_on_future_business_day>, date=datetime.date(1999, 12, 31)) You can reproduce this example by temporarily adding @reproduce_failure('4.0.0', b'AAAAAQ==') as a decorator on your test case FF. ====================================================================== FAIL: test_shift_to_next_business_day_always_fall_on_future_business_day (test_app.DeliveryDateTests) ----------------------------------------------------------------------
This provides us the name of the test that fails
test_shift_to_next_business_day_always_fall_on_future_business_day and the argument provided to that test when it failed,
date=date(1999, 12, 31).
Because all example data are generated randomly, if we want to reproduce this exact failure, we can use the decorator
reproduce_failure by copying the exact decorator provided in the error above.
1 @reproduce_failure('4.0.0', b'AAAAAQ==')
4.0.0 refer to
hypothesis version and
b'AAAAAQ==' refers to the data blob - but we don’t need to know that, we just need to use what the error provides.
dates strategy. A
strategy is a way of generating random data. In more details, a
strategy is composed of a
generator, which generates random data, and a
shrinker which shrinks data to a smaller size data where smaller is subjective to the datatype it is shrinking.
There are many strategies already predefined by
hypothesis for example we can import
booleans to generate booleans,
characters for random characters,
1 2 from hypothesis import given from hypothesis.strategies import booleans, characters, composite, integers, text, dates
But there will a time where we want to
compose objects so that they can be generated for the test. For example if we were the following properties:
1 2 3 4 5 6 7 8 9 10 @given(dates(), address(), integers(min_value=1, max_value=30)) # pylint: disable=no-value-for-parameter def test_delivery_date_should_not_fall_on_weekend_for_business_address(self, shipping_date, address, duration_days): result = get_delivery_date(shipping_date, address, duration_days) if address.is_business: self.assertNotEqual(result.isoweekday(), 6) self.assertNotEqual(result.isoweekday(), 7) self.assertGreater(result, shipping_date)
For this test to work, we need to have a way to generate
address. To create the strategies that we need, we can compose existing strategies together using the
@composite decorator. A
composite strategy takes a
draw function as first argument which can be used to draw values from strategies.
1 2 3 4 5 6 7 8 9 10 @composite def address(draw): character_strategy = characters( max_codepoint=1000, blacklist_categories=('Cc', 'Cs')) text_strategy = text(character_strategy, min_size=1).map( lambda s: s.strip()).filter(lambda s: len(s) > 0) address = draw(text_strategy) is_business = draw(booleans()) return Address(address, is_business)
Here we created an
address composite strategy which uses
booleans to build an
Address. We could also specify arguments to configure the strategies used within the
composite and pass the argument as we would for other strategies
Lastly we can see that if we need to execute simple operation, we can adapt strategies using
flatMap. Here we transformed the
texts strategy by removing whitespace with
.map(lambda s: s.strip()) and then filtering out text with no characters with
.filter(lambda s: len(s) > 0).
What we endup with are two property tests which covers a larger ground than our unit tests as random values are generated and tested against them 100 times on each test.
hypothesis being a powerful tool, we need to make sure that we leverage its power in the best way possible. There are two common mistakes to make when writing property-based test:
Remembering those two problems when writing the tests and building the strategies will ensure that we have useful tests!
Today we saw what is property-based testing by looking into a concrete example and comparing our approach in regards to unit tests and property-based tests. We then moved on to implement the example with
hypothesis, a python library containing all the necessary tools. Lastly we saw how we could use
hypothesis strategies to generate sample data and how we could compose our own strategies out of built-in strategies. I hope you liked this post! See you on the next one!