Testing - The 2 Universal Truths

I am writing today about one of the main reasons ERP projects fail! Failure is due to improper testing, or, the lack of testing at all. The current metric illustrates that less than 10% of projects deliver the results agreed upon. I am going to highlight testing and what can be done to ensure that the proper resources, including time, are allocated in the beginning of the project plan.

During my 30 year global career in I.T., I have found that there are two 2 fundamental truths about testing. The first truth is, you should always test! That's right, you should test. The focus should be about creating the proper testing environment. Optimal testing helps to identify problem areas early in this phase. Certainly, the sooner the errors are identified, the faster and more cost effectively they can be resolved. If the code is migrated to Production without review, any issue found then needs to be tested in Production. When Production becomes the testing environment it becomes very costly to fix due to outages and downtime necessary. The expense of addressing a problem in Production rather than in Stage or Test environments costs more significantly. The second truth about testing is that you want to find errors. During any testing, finding issues is a good thing! As long as there are errors, testing should continue. In a recent project, the SI had the company stop testing because they were finding too many bugs in the code... That doesn't make any sense and the SI continued to bill the company while the SI fixed the bugs it created.

Planning the testing process can be difficult and require a variety of resources that understand the business processes that have been identified. Each area is represented in the various sections of the project plan. In the testing section, there are added complexities in the timing and the iterative remediation of the issues. Each fix needs to be tested again to ensure the fix didn't break another section of code. One of the more interesting things about ERP systems and projects are the inherent connections and interfaces to the other modules across the enterprise. The testing effort ranges from simple unit tests through the end user testing and acceptance.

There once were two Oracle projects I was asked to review that had very similar issues. Both sets of   management teams lacked the visibility of the project and was moving into System Integration Testing or (SIT) because the SI said any issue found would be discovered and remediation would be completed well before User Acceptance testing, or (UAT).  They also indicated they would do the testing for the company, that never works because they don't know your business. Make sure Business Process owners are in charge of the testing and quality of the product.  In one case, the project went live and was a severe crash and burn, the other was saved by an independent review that set the project on another course.

There once were two Oracle projects I was asked to review that had very similar issues. Both sets of management teams lacked the visibility of the project and was moving into System Integration Testing or (SIT) because the SI said any issue found would be discovered and remediated prior to the next phase of the project. They also indicated they would do the testing for the company, which never works because they don't know your business. Make sure Business Process owners are in charge of the testing and accept the accountability of the system operations including testing. In one case, the project went live and was a severe crash and burn, the other was saved by an independent review that set the project on another course. 

In the first example, the project team/SI lied on the status reports to show all issues had been identified and fixed when in fact, the product was still in development and some of the key functionality had not been tested. That's right, an ERP system that goes live without testing the critical functionality and features, has little chance of success. There are further examples of the same behavior where one SI suggested, and convinced the company that no testing was necessary because they were implementing a Commercial Off The Shelf or (COTS) system, Oracle. After the independent intervention, testing was reinserted into the plan and to no ones' surprise, found thousands of errors and issues. This in turn added many thousands of hours of coding and re-solutioning back in the project plan. I will use better judgment here and not disclose either SI, or companies name, but the result of the first project was to stop the project for 18 months and finally reduce the scope of the Oracle footprint, while the second company ended the Oracle deployment and move to another solution...after 28 months of effort and cost.

Ok, so here is the part that many of you will like to see. How do you use the right amount of testing within the plan without going too deep or too wide?  I have identified sixteen testing events that could have a place in the project plan. There is also a very strong integration to the SDLC process that supports the control of code being promoted into Production, but that will be a topic for another time.

Here are the different types of testing that should be considered during an ERP project or any IT project that introduces added risk to the business: 

                   ─    Unit Testing

─    Standard Functional testing 

─    Data Testing – Master Data Management

─   Custom Functional Module Testing – P2P – OTC- Finance

─    Month End Close, Quarter End Close, Year End Close

─    SOA Testing – Integration / Interfaces

─    SIT or End - 2 - End

─    3^rd Party tools and Bolt-on Testing 

─    Regression Testing

─    Day in the Life – Load test

─    Destructive Testing

─    UAT  - User Acceptance Testing

─    Fail-over Testing

─    Back out and Recover Testing – Contingency Testing

─    Backup and Recovery Testing - Operations 

─    Production Cut-over validation - In Production

─    Operations and Monitoring / Alerts  

I don't have the room to expand on each testing cycle, but the names of the tests should be self-explanatory. OK, there are a number of variables and issues that are part of the testing process around who can test, what to test, how often and other tangents such as Automated Testing and the integration of the offshore model. I cannot possibly address each of these areas here in this format, but they are important to understand before the SI and project team launches into the deployment. The business is responsible to identify the critical areas to be tested and it is the Quality team who architect the tests to truly represent the business processes being evaluated. The team as a whole including the SI, project team and management needs to come together to deliver the code, test it, and remit the errors...oh, and re-test and re-test and re-test. Testing is a cyclical process, not a one-time event, so include the testing process throughout the duration of the plan.

The last piece of the testing puzzle is the tracking and monitoring of the testing and results. In most projects I have seen, the complexities and immense volume of data that is created doing the testing must have a method of both understanding the data and a better way to present it. Senior Management might have a clear view of the business issues, but depend on the testing experts to explain the details about the project, so the results and status must be very clear. I recommend a graphical representation of metrics and details whenever possible. You can always share more detail if there are further questions. I have included just one of the charts I use to present metrics and testing details I have developed over a number of years.

Let me know what you think, drop me a note at vince.benz@jomaxconsuting.com.  I am also available to discuss further issues around Testing and ERP.