KOR Software Testing Methodology

1.0 Testing Approach

At KOR, our testing strategy is comprised of a range of testing approaches, with automated testing serving as the primary. These test methodologies can typically be broken down between functional and non-functional testing.

The goal of utilizing numerous testing methodologies in our development process is to make sure our software can successfully operate according to the specifications, in a secure, scalable, and performant way, while allowing us to make adjustments fast, forever. 

The key to releasing high quality software that can be easily adopted by our end users is to maintain a robust testing framework that implements both functional and non-functional software testing methodologies.

We do not employ dedicated testers, as we believe everyone within the company has to take responsibility for the code they write. When a problem arises in production, it’s the developers who have the responsibility to immediately address it. This direct line of responsibility and accountability guarantees developers focus their attention on delivering exceptional quality.

1.1 Functional Testing

Functional testing involves testing the application against the business requirements. It incorporates all test types designed to guarantee each part of our platform behaves as expected. These testing methods are conducted in order and include:

1. Unit testing
2. Integration testing
3. Contract testing
4. UI testing
5. System testing
6. Acceptance testing
7. Exploratory testing

1.2 Non-Functional Testing

Non-functional testing methods incorporate all test types focused on the operational aspects of the platform. These include:

1. Performance testing
2. Security testing
3. Compatibility testing
4. Chaos testing

1.3 Test Pyramid

The "Test Pyramid" is a metaphor that tells us to group software tests into buckets of different granularity. It also gives an idea of how many tests we should have in each of these groups.

Each of the tests outlined above is part of KOR’s Test Pyramid.

At the top of the pyramid we find tests that are more holistic and test functionality in integration. These tests are also slower to run and harder to maintain. Therefore they are at the top of the pyramid, as they represent only a small fraction of the entire test suite. Test types like exploratory testing and system testing are represented here.

At the bottom of the pyramid we find test types such as unit tests that are fast to run, easy to maintain, and focus on a specific piece of functionality. KOR tends to write a lot of these tests.

2.0 Test types

2.1 Unit Testing

Unit testing is the process of ensuring individual components of a piece of software at the code level are functional and work as they were designed to. They are the foundation of our testing suite. Unit tests are low-level, focusing on a small part of the software system, and are significantly faster than other kinds of tests. Therefore, we have a lot of fast unit tests, focusing on the core logic of the platform, and vastly outnumber any other type of test. Developers in our test-driven environment write and run the tests prior to the software or feature being written to allow the design to emerge based on the requirements.

2.2 Integration Testing

KOR applications do not work in isolation. They integrate with other applications, databases, file systems and so forth. Integration tests test the integration of an application with all the parts that live outside of that application. We test one integration point at a time by replacing separate services and databases with test doubles. Together with contract testing and running contract tests against test doubles as well as the real implementations we create integration tests that are faster, more independent and easier to reason about.

All of the integrations the applications part our platform make, are tested with integration tests to ensure interoperability.

2.3 Contract Testing

We want to scale our development efforts by spreading the development of the platform across different teams. Individual teams build individual, loosely coupled services with distinctly different functions which are integrated into a big, cohesive microservices system. Splitting our system into many small services means that these services need to communicate with each other via well-defined interfaces, most notably Kafka events and REST. 

Automated contract tests make sure that the implementations on both sides of the interface still stick to a defined contract. They serve as an effective regression test suite and make sure that deviations from the contract will be noticed early. Using Consumer-driven contract testing, consumers of an interface write tests that check the interface for all data they need from that interface. The consuming teams then publishes these tests so that the publishing team can fetch and execute these tests as part of their build pipeline. These approaches ensure that changes to interfaces are always compatible with all consuming parties, while keeping technical debt low. For each REST interaction, KOR employs consumer-driven contract tests. For our Kafka events, we use the Schema Registry to guarantee compatibility between the different parts of the platform.

2.4 UI Testing

User Interface (“UI”) tests test that the user interface of our platform works correctly. This includes the portal and the public ticker. User input should trigger the right actions, data should be presented to the user, the UI state should change as expected. We test behavior, layout, usability and adherence to compliancy specifications. Our tooling opens the web applications in different browsers and formats, takes screenshots and compares these to previously taken screenshots. If the old and new screenshots differ in an unexpected way, the tool will let us know. Testing usability in an automated fashion has its limits, so we combine our automated test suite with exploratory testing, as well as extensive user research to verify if users can effectively leverage all features without as intended.

2.5 System Testing

System testing, and to a greater extent end-to-end testing, attempts to validate the entire system as a whole. However, these tests can pose a number of challenges. Especially when there’s a UI involved, they are notoriously limited in utility and often fail for unexpected and unforeseeable reasons. Quite often their failure is a false positive. They often require a higher level of maintenance and have long run durations.

Due to their high maintenance cost, KOR intentionally limits the number of end-to-end tests to a minimum. This includes high-value interactions our users have with our platform. We define user journeys that make up the core value of our platform and translate the most important steps of these user journeys into automated end-to-end tests. We have a substantial number of lower levels in our test pyramid where a wide range of edge cases and integrations with other parts of the system are tested. There is largely diminished need to repeat these tests on a higher level. High maintenance effort and lots of false positives will slow us down and cause us to lose trust in our tests, sooner rather than later.

2.6 Acceptance Testing

Acceptance testing ensures that our software works correctly from a user's perspective, not just from a technical perspective. These tests come in different levels of granularity. Most of the time they are rather high-level and test our service through the UI. However, our application design and a lot of our scenarios permits us to write acceptance tests at a lower level. Having a low-level test is better than having a high-level test. The concept of acceptance tests - proving that our features work correctly for the user - is completely orthogonal to our test pyramid. This is why we our acceptance tests reside as part of the other test types.

2.7 Exploratory Testing

Even the most diligent test automation efforts are not perfect. This is why we still include exploratory testing in our testing portfolio. It is a manual testing approach that emphasizes the tester's freedom and creativity to spot quality issues in our running system. We take out time on a regular schedule to try to break our application. We use a destructive mindset and come up with ways to provoke issues and errors in our application. We look for bugs, design issues, slow response times, missing or misleading error messages and everything else that would annoy you as a user of your software.

We then automate most of the findings with automated tests. Writing automated tests for the bugs we spot makes sure there won't be any regressions of that bug in the future. Plus, it helps us narrowing down the root cause of that issue during bug fixing. We use the findings of exploratory testing as feedback on the maturity of our build pipeline. We make sure to act on that feedback so our pipeline and our entire software delivery will grow more mature the longer we go.

2.8 Performance Testing

Performance testing is the practice of evaluating how a system performs in terms of responsiveness and stability under a particular workload. We use performance tests to examine speed, robustness, and reliability. The process incorporates “performance” indicators such as browser, page, and network response times, server processing times, acceptable throughput volumes, processor and memory consumption, and the number and type of errors that might be encountered in our platform. All performance tests are executed against a replica of the production environment.

We use different types of performance tests, such as load tests, stress tests, spike tests and soak tests.

2.9 Security Testing

In order to harden our security posture, we shift left security testing as far as possible. By automating various inspections, we can fix vulnerabilities before they are exposed. Our pipeline includes vulnerability scanning for Java dependencies, operating system layers and containers, exposed secret detection, static application security testing (“SAST”) and dynamic application security testing. We also work with ethical hackers to independently assess our security posture.

2.10 Compatibility Testing

We apply two types of compatibility testing, backwards compatibility testing and forward compatibility testing.

Backward compatibility testing verifies whether consumers, be it external clients or internal components, will keep working with newer versions of components within our platform.

Forward compatibility testing verifies whether consumers, be it external clients or internal components, will keep working with older versions of components within our platform. This could happen if we have to roll back a release of a server component, even though the clients of said server have already migrated to the newest version.

We use the Kafka Schema Registry and contract tests to validate compatibility between versions of our contracts.

2.11 Chaos Testing

Chaos testing, or chaos engineering, is the highly disciplined approach to testing a system’s integrity by proactively simulating and identifying failures in a given environment before they lead to unplanned downtime or a negative user experience. We organize Game Days, events where we bring people together, along with their real systems, to practice a simulated-but-realistic incident in a replica of production. The Game Day could last for a day, a week, or a few hours; it all depends on the incident we’re practicing. Chaos tests are the automation of such Game Days, to reliably and safely test how we deal with non-functional issues, such as network delays, crashing instances, missing data segments, network partitioning, etc.

The automated nature of chaos tests ensures on the one hand that our teams are able to see real-life simulations of how their application or service responds to different pressures and stresses. On the other hand, because failure is guaranteed, our teams are forced to incorporate failure resilience by design.