by David Pyle –
Randomly pick out any 10 people working in IT, and ask them this question: What is the purpose of QA & Testing?
It’s very likely that all 10 of the answers you receive will go something like this: “The purpose of QA & Testing is to test code to assure that it is working as intended, and without errors.”
That’s not a bad answer. And it’s certainly not an incorrect answer. But it’s an incomplete answer.
It’s Not Just About Testing the Code
While it’s certainly important to assure that the code itself is working as intended, that’s not enough. Doing just that is Testing but not QA. In fact, just testing code will leave your production environment exposed to considerable risk.
The true purpose of QA & Test is to answer one simple question: Will the system (not just the code) work properly, as designed, in production? And just testing the code won’t give you the answer to this question. Though the question is simple, answering the question is not so simple.
Providing the answer to that question does require that the code be tested. But it requires so much more. It also requires assuring the quality of the environment with which the code interacts.
And it requires QAing the build process along with the code for accuracy and completeness. That’s because all too often the code will work in the test environments only to fail in production. Missing elements, differences in environment variables, or data that does not match the new software are just a few examples of problems that commonly occur after the transition to production.
A Build Package – Not Just a Build
Every time development releases a new build, the code is tested but a QA review of the build package should also be performed.
The build package should include everything that will be used to implement the application in production, including scripts/instructions used to accurately build all components to the production environment. So in reality, QA will be testing far more than just the functionality of the code. They’ll also be monitoring and testing factors that impact the implementation of the code in production, things such as:
- Ensuring that the test environment matches production
- The configuration management of the code
- Testing data matches production data as close as possible, both user generated data and system required data.
The build package should provide instructions for everything that must happen to successfully implement the build in the production environment.
Here’s an example of the sort of detail that’s frequently overlooked in the test environment: server names. Servers used in system integration testing are not going to be the same as those used in production. So the build package instructions should specify changing the environment variable that points to the server for test. And that process should be tested during the QA process.
Another example: code that requires a certain amount of data to be populated in your database in a certain way. You set up that population data in development and you develop the code against it. Then you do the same thing in system integration testing. But if you don’t have a process by which you would set up that data inside of your production environment, as part of the build package, you’re exposing your organization to risk.
Consider, for instance, a table containing a list of all the valid service types that are available within the system, along with an indicator as to which functionality will be run for each service type. If the code was built to process the service type, but a corresponding record was not added to the table to define the service type, then the system will fail. So the build instructions should specify the inclusion of this record in the table.
QA & Testing should not test just the code. It should QA and test the entire process of implementing the build package – including testing the accuracy of the build package instructions.
So who has the responsibility of putting together the build package and build package instructions?
That would logically be development. Development, after all, is the only organizational entity that has full awareness of all components required for the system as designed.
A Single Snapshot Won’t Suffice
Keeping an eye on production is also critically important. Because the build package — and accompanying instructions — should be based upon the current production environment.
But production environments are dynamic. They change frequently. So a single production snapshot taken early in the development process won’t be enough.
Instead, it’s important to constantly monitor the production environment to ensure that both development and test environments mimic production as closely as possible. Failure to do so is a common cause of code sailing through QA, only to stumble when implemented into production.
In An Ideal World…
In an ideal world your development environment would be identical to your production environment. That way you would know that everything that happens or doesn’t happen in test would be the same in production.
But the ideal world isn’t the real world. For most enterprises, precisely duplicating the production environment in test is an outright impossibility. That’s why it’s so important that the entire build package be QAed, and not just the code, be tested during the QA & Test process.
It’s the best way to avoid the comments that all-too-often echo throughout an organization after an application is implemented in production: “How did we miss that?” “Why did we not catch that in test?” “Is the code wrong?” “Whose fault is this?”
That’s why it’s in the best interest of both development and QA & Test is to assure that the entire process is documented and tested. After all, the ultimate goal of testing is to assure that everything works as intended in the production environment. And attaining that goal requires much more than just simply testing code.
To read more, visit our page on RCG’s Quality Assurance and Testing Practice