Adding value to your TestOps with pytest

The pytest framework is one of the best opensource test frameworks that allows you to write test cases using Python. With pytest, you can test almost anything — databases, APIs, even a UI. However, pytest is mainly used to write tests for APIs and it is popular amongst the testers because of the simplicity, scalability, and pythonic nature. You can parallelly run specific tests or subset of test cases. Test cases are written as functions and test assertion failures are reported with actual values. Plugins can be used to add code coverage, pretty reports, and parallel execution. And the best part is, pytest can also be integrated with other frameworks like Django and Flask.

Since pytest is widely popular amongst the QA community, let’s see what all standard practices you should follow while writing any test case in pytest. I will elaborate on some of these practices using a few examples. Let’s start from the basics first — writing a test case.

The first test

Writing the first test case is always a big thing for most of us. So, how do you go about it? In pytest, it will search for test functions named test_* in modules named test_*.py. Interestingly, pytest doesn’t need an__init__.py file in any test directory.

Example –

Now, if you run pytest, you will get an output like this –

In the above command, -s -v parameters are used to display the test case name along with pass and fail status in detail.

Generate HTML report

Sometimes, when pytest runs at the command line, it prints a futile text. In such scenarios, visual reports are a much better way to understand the test result information. Adding the pytest-html plugin to your test project enables you to print better HTML reports with one simple command-line option.

The report will look like this –

Parameterized test case

Pytest has a decorator to run the same test procedure with multiple inputs. The @pytest.mark.parametrize decorator will substitute tuples of inputs for test function arguments and running the test function once per input tuple. If you are looking to perform data-driven testing, Parameterized testing is a great way.

Example –

The result will look like this –

Parallel test execution with pytest

It’s common in any development practice to have thousands of tests with an average of one test per minute. Consider an example where you have 100 tests that would take about 1 hour 40 minutes to run. By this rate, for 1000 tests, it would take around 16 hours to run. That is a lot of time if you are looking at it from a CI/CD pipeline’s perspective.

The only way to achieve truly continuous testing is to run them parallelly via the pytest plugin. In pytest, you can scale-up the test thread count with pytest-xdist and if you want to scale-out you can distribute the test execution to remote machines.

In the above output, 4 workers are created with names [gw0], [gw1], [gw2], [gw3]. It will create 4 threads on which test cases will run.
-n option is to run tests by using multiple workers.

The log file will look like this –

Although the time difference looks negligible when there are only a few tests to run, imagine the time taken when you have a large test suite.

Run multiple tests from a specific file and multiple files

Suppose you have multiple files, say test_example1.py, test_example2.py. To run all the tests from all the files in the folder and subfolders, you need to only run the pytest command py.test. This will run all filenames starting with test_ or ending with _test in that folder and subfolders under the particular folder.

Run tests by substring matching

Suppose you have test files where you have to run all the test cases having “calculate” substring in its test case name then you just need to run the following command –

py.test -k calculate

It will run all the test cases that contain the substring “calculate”.
Here -k is used to represent the substring to match.

Run tests by markers

Pytest allows you to set various tags to the tests using pytest markers @pytest.mark. To use markers in the test file, you need to import pytest library in the test files, and then you can apply different marker names to the test methods and run …read more