Why should I pay you to test your own work?
This is a question I have heard a lot over the years when discussing testing budgets with clients. To the uninitiated, it sounds like a fair question. However, anyone involved in software development knows how complex and time-consuming testing can be. Testing is, in fact, one of the most important parts of any software development project.
A large e-commerce platform is an incredibly complex thing, with millions of lines of code, gigabytes of data, and many integration points. There are so many interlinked moving parts – so many links in the chain – that it is very easy for something to go wrong. The application will be used in millions of different ways through a multitude of browsers across numerous desktop and mobile devices. The development project will have lasted at least six months and involve many different people. The scenarios that can be tested are almost limitless. It’s a wonder anything works at all!
Testing can be split into a number of different areas, but each is important to consider. Every project is a little different; some clients like to take on much of the testing themselves; some like to outsource it, and some expect their developer to do it all. Testing is not a fixed entity; you can do a lot of testing, and you can do a little. The more you test, the more you will de-risk the project, but the more time it will take and the more it will cost.
A unit test is one which tests small “units” of code to ensure that they function as expected. For example, when a form is submitted, it should save the inputted details into a database table. It is a standalone test that specifically, and only, ensures that the unit functions as expected. Using a true test-driven development methodology, a developer will first write a test before actually creating any code; the code is considered completed only when the test is passed. In practice, unit testing is only used in some key areas of the application to ensure that core functions are working as expected. While unit testing can reduce the likelihood of functional issues occurring, it can also increase development time.
You will probably hear your development agency talk about smoke testing. A smoke test is a pragmatic subset of test cases covering the key user journeys and functions throughout your application. At the very least, your developer should be expected to carry out smoke tests before handing anything over to you for user acceptance testing.
User Interface (UI) testing can be very complex and time-consuming. The huge range of operating systems, browsers, and mobile, tablet, and desktop devices that will be used to access a website means that comprehensively testing every combination manually is almost impossible. Because of the vast number of different variations that need to be covered, UI testing is a perfect candidate for automated testing. Automated test tools are able to follow a scripted journey through your website and test whether the expected results are achieved. They can also record each journey so that each one can be played back. Although this method is not perfect, it can significantly reduce the number of major UI issues a website may face.
Some third-party testing services such as Bug Finders offer a crowd-sourced service where hundreds of freelance human testers from around the world are used to test a website and are paid when they find an issue. This approach can be a relatively cost-effective way of testing your application across hundreds of device/platform/browser combinations. It is normal for a test cycle to result in around 200 issues being raised. The challenge is often in categorizing and prioritizing the issues so that you focus your resources on dealing with the most important ones. Every website will have a constant backlog of low-level issues that are unlikely to ever be resolved.
User acceptance testing
User acceptance testing (UAT) is a critical part of any development project and involves full end-to-end testing of the platform before going live. UAT is the process that I most often see underestimated, and the first to suffer when timelines are tight. However, this is likely to lead to higher rate of failure. For any new website build, we advise planning at least two months of UAT. Your e-commerce website is only one part of your e-commerce business. The end-to-end process involving search, checkout, order management, payment, despatch, customer services, finance, and all of the other parts of the chain need to be tested.
UAT is often confused or merged with system integration testing (SIT), where you will be specifically testing the integration between multiple systems. SIT is part of the end-to-end testing that ensures that all parts of the chain are working correctly together.
Good UAT involves the creation of test cases and test plans. These generally take the form of a set of scripts (a set of tasks) that a manual tester will run through and either pass or fail the test according to the outcome. It is not unusual for an end-to-end UAT test plan to include more than 500 test cases.
The “A” in UAT is one of the reasons why it is so important. At the end of the UAT process, you will generally be deemed to have accepted the application. It is important that you have thoroughly tested it to ensure that it works in exactly the way you expected. This does not mean that undiscovered bugs will not be under warranty. But if there is functionality that does not work in the way that you expected, this needs to be picked up in UAT. It is also important because it is the final chance to pick up issues before it goes live. Any bugs and issues are likely to negatively impact the user experience.
UAT requires a lot of effort on behalf of the client and is often underestimated. Some clients use external testing agencies to support them during UAT, which can significantly de-risk a project when the client lacks in-house staff to carry out UAT effectively.
I am sometimes surprised to observe retailers that fail to take security testing seriously enough. It is not unusual to find that the retailer does not know when a penetration test was last performed on their Web platform. These are generally the ones who have not yet been hit with a cyber attack (or don’t yet know that they have been hit). In the current climate where cybercrime continues to grow in frequency and sophistication, and especially with GDPR on the horizon in Europe, security testing is increasingly important. All e-commerce Web platforms should be penetration-tested by a specialist third party at least annually, and ideally twice a year. It is also advisable that your application is scanned for vulnerabilities using specialist software such as Nessus on a regular basis. At Envoy, we tend to scan our clients’ Web platforms on a weekly basis to ensure that application vulnerabilities are picked up very quickly. At the very least, you should carry out application security scans before each release to production. It is no good waiting for six months until the next penetration test when you have introduced a new application vulnerability.
Performance testing is generally used to determine how much traffic, page requests, concurrent users, and order volume your website can handle. It is more difficult than you may imagine because accurate testing requires that you mimic real user behavior and, of course, real users do a lot of different things. The best you can do is mimic your key user journeys such as search, add to basket, and check out. You ideally want to carry out load testing on your production environment rather than a staging environment as it will give you a much truer picture. But this is also likely to take your platform offline at some point during the test.
Most retailers tend to carry out load tests once a year, normally before peak periods such as Black Friday or Christmas. However, since the last annual test, a large number of changes may have been made to the application with an incremental impact on performance. If an annual load test shows a drop in performance compared to the previous year, it is very hard to determine which change or changes are responsible. This also may not give you enough time to resolve the performance issues before peak trading starts.
To counter this, it is advisable to carry out performance benchmarks before each new code release. These do not need to be performed in a production environment as long as each test is carried out in the same environment. The aim is to determine whether performance has increased or decreased relative to the last release. This of course takes time and therefore will increase development time and costs
While the list above is not exhaustive, you can see that the scope of testing within software development can be large and complex. Each type of testing takes time and effort, and you should not assume that it is all done as standard with no additional charge. Companies with a strong focus on testing will allocate up to 40% of any project time to testing. Good testing can de-risk a project and pay for itself in the long run, as it will result in fewer bugs, better performance, and a better overall experience for your customers.
For more on this topic, see The Importance Of Cybersecurity In Modern E-Commerce.
This article originally appeared on The Future of Customer Engagement and Commerce and is republished by permission.