Selecting the Right Automated Tests

This is an article from DZone's 2023 Automated Testing Trend Report.

For more:

Read the Report

Modern software applications are complex and full of many dynamic components that generate, collect, and fetch data from other components simultaneously. If any of these components acts unexpectedly or, worse, fails, there can be a cascading effect on all other dependent components. 

Depending on the nature of the software, these errors or failures can result in system downtime, financial loss, infrastructure collapse, safety implications, or even loss of life. This is why we test and monitor software. Testing with the right techniques and test cases at the right stages in the software lifecycle increases the chances of catching problems early and before users do.

When and Where to Test

Generally, tests occur in the "testing" stage of the software development lifecycle (SDLC). However, for certain types of tests, this is not the case, and when you implement and run, each test type can vary. Before we get into selecting the right test, let's quickly review when and where to use different types of tests.

How To Choose Tests

As with many technical projects, reading a list of recommendations and best practices is only the beginning, and it can be difficult to decide which of those apply to your use case. The best way is to introduce an example and show the reasoning behind how to decide a strategy based on that use case. It won't match any other use case exactly but can help you understand the thought process.

Example Application

I have a side project I am slowly trying to build into a full application. It's a to-do aggregator that pulls tasks assigned to me from a variety of external services and combines them into one easier-to-view list. It uses the APIs of each of these services to fetch assigned task data. Users can sort and filter the list and click on list items to see more details about the task. The application is written in TypeScript and React and uses material UI. Additionally, there are mobile and desktop versions created with React native. 

Essential Tests

Unless you have a good reason not to include them, this section covers tests that are essential in an application test suite. 

Figure 1: Essential tests in the SDLC

Unit Tests

Essential for almost any application and possible to create as you build code, any application that has more than one functional component needs unit tests. The example application has one component that takes the data returned from the APIs and converts it to React objects ready for rendering in the UI. 

Some examples of unit tests in this example could be: 

• Determining whether there are objects to render
• Checking if the objects have essential data items to render (for example, the title)
• Determining if the UI is ready for objects to be rendered to it

As the application uses Typescript, there are many options available for writing unit tests, including Jest, Mocha, and Jasmine. They all have advantages and disadvantages to the ways they work, with no real "right answer" as to which is best. Jest is possibly the most popular at the moment and was created by Facebook to unit test React. The example application is based on React, so perfect!

API and Integration Tests

The example application relies heavily on multiple APIs that have multiple points of failure with the potential to render the application unusable if handled poorly. API and integration tests are not quite the same as each other. While API testing tests only API interactions, integration testing could test API tests but also other third-party integrations, such as external components. As the example application's only third-party integration are APIs, we can consider them the same. 

Some examples of API errors to test for could be: 

• Expired or changes to authentication methods
• A call that returns no data 
• A call that returns unexpected data
• Rate limiting on an API call

API tests typically happen externally to the application code, in an external tool, or in a CI pipeline. Open-source options include writing your own tests that call the API endpoints, SoapUI (from the same people that define the API spec standard), Pact, and Dredd. Personally, I tend to use Dredd for CI tests, but there is no obvious choice with API testing.

UI Tests

If an application has a visual front end, that front end needs automated tests. These tests typically simulate interactions with the interface to check that they work as intended. The UI for the example application is simple but essential for user experience, so some example tests could include checking whether: 

• Scrolling the list of aggregated tasks works
• Selecting a task from the list opens more details

Tools for automated UI testing are typically run manually or as part of a CI process. Fortunately, there are a lot of mature options available, some of which run independently from the programming language and others as part of it. If your application is web-based, then generally, these tools use a "headless browser" to run tests in an invisible browser. If the project is a native application of some flavor, then UI testing options will vary. 

The example project is primarily web-based, so I will only mention those options, though there are certainly more available: 

• Selenium – a long-running tool for UI testing and is well-supported  
• Puppeteer – a mature UI testing tool designed for Node.js-based projects

For the example application, I would select a tool that is well suited to TypeScript and React, and where tests are tightly coupled to the underlying UI components. 

Optional Tests

This section deals with test types to consider if and when you have resources available. They will help improve the stability and overall user experience of your applications. 

Figure 2: Optional tests in the SDLC

Security

Security is a more pressing issue for applications than ever. You need to check for potentially vulnerable code during development and also the increasing problem of introducing vulnerabilities through package dependencies. 

Aside from testing, generating and maintaining lists of external packages for software supply chain reasons is a rapidly growing need, with possible regulatory requirements coming soon. 

Some examples of vulnerability issues to test for could be: 

• Storing API credentials in plain text
• Sending API credentials unencrypted
• Using vulnerable packages

There are two groups of tools for testing these requirements. Some handle scanning for vulnerabilities in your code and external code, while others handle one of those roles. 

Vulnerability scanning is a new growth business for many SaaS companies, but some popular open-source and/or free options include, but are not limited to, GitHub, Falco, and Trivy. These tools are programming-language independent, and your decision should be based on the infrastructure you use behind the application. The example application runs on a user's device locally, so the best time to run a vulnerability checker would be in CI and CD during the build process. 

Performance Tests

There is little point in creating a finely crafted application without any kind of monitoring of how well it performs in the hands of users. Unlike most of the other test types on the list, which typically run at distinct phases in the SDLC, performance testing generally happens continuously. Some tools let you mock production usage with simulated load testing, and this section includes some of those, but they are still not the same as real users. 

Possible issues to monitor are: 

• Speed of populating task results
• Critical errors, for example, API changes in between builds
• Slow UI responses

As performance monitoring often needs a centralized service to collate and analyze application data, these tools tend to be commercial services. However, there are some open-source or free options, including k6 (for mocking), sending React <Profiler> data into something like Grafana, and Lighthouse CI. 

Smoke Tests

A lot of other testing methods test individual functionality or components but not paths through how these fit together and how users use an application. Smoke tests typically use a quality assurance (QA) environment to check that key functionality works in new builds before progressing to further tests. Smoke tests can be manually undertaken by a QA team or with automated tools. The tool options depend on what it is you want to test, so many of the other tools featured in this article can probably help. 

For the example application, a smoke test would check that the list of aggregated tasks is generated. 

Regression Tests

Regression testing isn't a set of tools but a best-practice way of grouping other tests to ensure that new features don't have an adverse effect on an application. For example, a new release adds the ability to change the status of tasks aggregated in the application, sending the status back to the source task. The other following tests would work together to ensure that introducing this new feature hasn't negatively affected the existing functionality, which was only to view aggregated tasks.

Some examples of regression test grouping are the following: 

• Unit tests for the new feature 
• API tests for updating items on the relevant service API 
• Security tests to ensure that calling the new APIs doesn't reveal any sensitive information
• Performance tests for the new feature and to check that the new feature doesn't affect reloading the task list

Conclusion 

This article covered the many different types of tests an application can implement and the kinds of issues they can prevent. All of these issues have the potential to hinder user experience, expose users to security issues, and cause users not to want to use your application or service. As you add new features or significantly change existing features, you should write relevant tests and run them as frequently as convenient. In many modern SDLC processes, tests typically run whenever developers check in code to version control, which you should also do frequently.

This is an article from DZone's 2023 Automated Testing Trend Report.

For more:

Read the Report