Testing - The 2 Universal Truths

I am writing today about one of the main reasons ERP projects fail! Failure is due to improper testing, or, the lack of testing at all. The current metric illustrates that less than 10% of projects deliver the results agreed upon. I am going to highlight testing and what can be done to ensure that the proper resources, including time, are allocated in the beginning of the project plan.

During my 30 year global career in I.T., I have found that there are two 2 fundamental truths about testing. The first truth is, you should always test! That's right, you should test. The focus should be about creating the proper testing environment. Optimal testing helps to identify problem areas early in this phase. Certainly, the sooner the errors are identified, the faster and more cost effectively they can be resolved. If the code is migrated to Production without review, any issue found then needs to be tested in Production. When Production becomes the testing environment it becomes very costly to fix due to outages and downtime necessary. The expense of addressing a problem in Production rather than in Stage or Test environments costs more significantly. The second truth about testing is that you want to find errors. During any testing, finding issues is a good thing! As long as there are errors, testing should continue. In a recent project, the SI had the company stop testing because they were finding too many bugs in the code... That doesn't make any sense and the SI continued to bill the company while the SI fixed the bugs it created.

Planning the testing process can be difficult and require a variety of resources that understand the business processes that have been identified. Each area is represented in the various sections of the project plan. In the testing section, there are added complexities in the timing and the iterative remediation of the issues. Each fix needs to be tested again to ensure the fix didn't break another section of code. One of the more interesting things about ERP systems and projects are the inherent connections and interfaces to the other modules across the enterprise. The testing effort ranges from simple unit tests through the end user testing and acceptance.

There once were two Oracle projects I was asked to review that had very similar issues. Both sets of   management teams lacked the visibility of the project and was moving into System Integration Testing or (SIT) because the SI said any issue found would be discovered and remediation would be completed well before User Acceptance testing, or (UAT).  They also indicated they would do the testing for the company, that never works because they don't know your business. Make sure Business Process owners are in charge of the testing and quality of the product.  In one case, the project went live and was a severe crash and burn, the other was saved by an independent review that set the project on another course.

There once were two Oracle projects I was asked to review that had very similar issues. Both sets of management teams lacked the visibility of the project and was moving into System Integration Testing or (SIT) because the SI said any issue found would be discovered and remediated prior to the next phase of the project. They also indicated they would do the testing for the company, which never works because they don't know your business. Make sure Business Process owners are in charge of the testing and accept the accountability of the system operations including testing. In one case, the project went live and was a severe crash and burn, the other was saved by an independent review that set the project on another course. 

In the first example, the project team/SI lied on the status reports to show all issues had been identified and fixed when in fact, the product was still in development and some of the key functionality had not been tested. That's right, an ERP system that goes live without testing the critical functionality and features, has little chance of success. There are further examples of the same behavior where one SI suggested, and convinced the company that no testing was necessary because they were implementing a Commercial Off The Shelf or (COTS) system, Oracle. After the independent intervention, testing was reinserted into the plan and to no ones' surprise, found thousands of errors and issues. This in turn added many thousands of hours of coding and re-solutioning back in the project plan. I will use better judgment here and not disclose either SI, or companies name, but the result of the first project was to stop the project for 18 months and finally reduce the scope of the Oracle footprint, while the second company ended the Oracle deployment and move to another solution...after 28 months of effort and cost.

Ok, so here is the part that many of you will like to see. How do you use the right amount of testing within the plan without going too deep or too wide?  I have identified sixteen testing events that could have a place in the project plan. There is also a very strong integration to the SDLC process that supports the control of code being promoted into Production, but that will be a topic for another time.

Here are the different types of testing that should be considered during an ERP project or any IT project that introduces added risk to the business: 

                   ─    Unit Testing

─    Standard Functional testing 

─    Data Testing – Master Data Management

─   Custom Functional Module Testing – P2P – OTC- Finance

─    Month End Close, Quarter End Close, Year End Close

─    SOA Testing – Integration / Interfaces

─    SIT or End - 2 - End

─    3^rd Party tools and Bolt-on Testing 

─    Regression Testing

─    Day in the Life – Load test

─    Destructive Testing

─    UAT  - User Acceptance Testing

─    Fail-over Testing

─    Back out and Recover Testing – Contingency Testing

─    Backup and Recovery Testing - Operations 

─    Production Cut-over validation - In Production

─    Operations and Monitoring / Alerts  

I don't have the room to expand on each testing cycle, but the names of the tests should be self-explanatory. OK, there are a number of variables and issues that are part of the testing process around who can test, what to test, how often and other tangents such as Automated Testing and the integration of the offshore model. I cannot possibly address each of these areas here in this format, but they are important to understand before the SI and project team launches into the deployment. The business is responsible to identify the critical areas to be tested and it is the Quality team who architect the tests to truly represent the business processes being evaluated. The team as a whole including the SI, project team and management needs to come together to deliver the code, test it, and remit the errors...oh, and re-test and re-test and re-test. Testing is a cyclical process, not a one-time event, so include the testing process throughout the duration of the plan.

The last piece of the testing puzzle is the tracking and monitoring of the testing and results. In most projects I have seen, the complexities and immense volume of data that is created doing the testing must have a method of both understanding the data and a better way to present it. Senior Management might have a clear view of the business issues, but depend on the testing experts to explain the details about the project, so the results and status must be very clear. I recommend a graphical representation of metrics and details whenever possible. You can always share more detail if there are further questions. I have included just one of the charts I use to present metrics and testing details I have developed over a number of years.

Let me know what you think, drop me a note at vince.benz@jomaxconsuting.com.  I am also available to discuss further issues around Testing and ERP.     

 

        

The Biggest problem with ERP implementations…Money

The Problem:

The ERP implementation field is covered with wrecked careers and businesses that did not survive.  In a recent Gartner report, 65% of all ERP projects fail and another 25% never meet the intended financial or functional goal of the company.  These failures are an accepted practice in the commercial software industry, keeping an arm’s length between the sale of the software and the numerous third parties that are available to implement the solution.  Not all System integrators are created equal.  I in no way include all System Integration Companies in painting the industry with one brush.   The project outcome is based on the quality of people and processes that are brought to the table,not the name of the SI. 

The cost to deploy this type of software is roughly 5-8 times the money spent to purchase it.  The SI creates the SOW that includes all the components to how the project will be run as well as details the cost of the project across such elements as milestones, resources, both on and off shore, hours, and expenses.  Many projects I have evaluated lack the clarity and visibility of the work product against payments to the SI.   More important is the payments are really structured as a payment schedule rather than payment for services.  If the money was distributed based on performance, many projects would be stopped midpoint for the code and configuration to catch up with the plan.  A stagegate process for checks and balances are an important process that includes code reviews and interum testing practices.   

It is the System Integrator who is responsible to bring the necessary subject matter experts or SME to the project in order to extract the value from the purchased software.  The tier 1 ERP vendors SAP and Oracle enjoy an estimated combined market share of 42% worldwide, there are over 350 other products that fill out the sector.  The major System Integrators have found a need and have developed “proprietary processes” that are sold to guarantee the success of the project.  These processes have been developed over a number of years, using collateral that seemed to be useful in prior projects. So, if the SI has so much experience, “WHY IS IT WE HAVE SO MANY ERP AND MEGA PROJECT FAILURES?” In 2 recent ERP projects I have been involved with, the SI systematically collected the monthly fee, and in the end, both companies lost multi-millions of dollars due to the poor performance and never completed a successful go lives. Remember, the System Integrator was hired because they knew how to execute this type of project and knew the issues that would be most significant to close.  What happened to the partnership and collaboration between the SI and the companies that are in need of help?    

When I say millions, I mean MILLIONS, like over nine digits.  In one case, over $100,000,000 was spent over 3 years on an ERP project that was driven over the cliff by the SI.  In the second case, the project was built completely in old technology and configured without the help or expertise of a solution architect.  The company never thought to bring in its own experts until it was too late and the money was spent.  This expertise is critical due to the level of integration within these types of systems; it is important to understand how the system integrates and impact other systems and uses the internal connection points. 

Neither project ever had a chance to make it to go-live.  The only saving point in the later project was the execution of the IV&V clause in the SOW and had an independent company review what was going on and was able to stop the project before it crippled their business.   Money aside, the only downside was the loss of money.  Not that money is ok to throw away, but we have read about other companies that have closed their doors after a catastrophic ERP or badly executed project event.   The cost of the project, remember millions, now must be written off because no value could ever be had from the project.

 

The Solution

Before I get into some of the corrective actions that I have used in putting these projects back on the rails, there are 4 things that happen during an ERP project that has gone badly.  Blame, Embarrassment, Accountability and Partnership / Teamwork breakdown.   Interestingly, blame is never a topic that companies or System Integrators what to talk about because it leads to litigation.  No one can admit fault because that will be used against them in a court of law.  The embarrassment is obvious in the company regardless of level of executive.  The senior executives and CEOs feel responsible for allowing this amount of money and time to be spent without clear accountability, and are very reluctant to share the failure with their board and shareholders. 

Most companies that jump into the ERP space usually are unskilled in the methods of what it takes for a successful implementation, but why should they?  Most have thriving businesses and are focused on their core competency and are looking for the next technology platform by which to grow.   Executives I have interviewed did as much research asthey could on the products and partners before engaging the ERP vendors.   Many hours of research are usually done to validate the capability of the product and the benefits are defined early in the process to balance the cost to the shareholders.  This is where it starts to breakdown.  Once the company selects the software, the System Integrators are asked to propose their solution.  Without having any knowledge of the business or what is necessary, each company that ventures into this space becomes dependent on the System Integrator for planning, resources, status, cost, and configuration of the system.   Remember, I put a disclaimer earlier in the article that not all System Integrators treat their customers badly. There are SIs that are very customer focused.  There are many components in putting together an effective team that includes both consultants as well as key business stakeholders that need to support the system after it is up and operational.    Select the partner carefully, interviewing all consultants on the project and getting each person to produce a resume to show they have the necessary experience and background to be as effective as possible.  Even get the resumes of offshore resources.  In most of the projects I was asked to evaluate, most lacked any visibility to who was actually on the project and even less of what each were doing to bring the project to fruition. 

If the customer company lacks the skills or understanding of the new product, it is important to hire resources to be the sounding board to the SI and is there to protect the interest of the company.  There will also be a point in the project that Knowledge Transfer will occur only if the people have been brought into the company as full time employees.  Without these people being available, the company will remain dependent on the SI for the foreseeable future.  Those are incremental costs that are seldom included in the original estimate.   The SI is a business the same as any other and can make decisions that are not in the best interest of the company.  Training here is the answer.  The more the company employees know about the new tool, the better result can be deployed.   

Finally, these projects can and occasionally do come out well.   The key is transparency and complete visibility of the status and risk of the project on a weekly basis.  That might seem like a tight schedule, but better to know everything is executing on schedule rather than find after a month or quarter, that issues or people or technology has caused a major challenge and cannot be recuperated.   Ensuring truthful communication up the chain of management as well as clear messaging across the organization will help alleviate rumors and keeping everyone on track.     Operation and Project metrics along with costs need to be highlighted on the status reporting to senior management.  References and checking on the resources within the industry can be a good way to show if the SI being considered is up to the work. 

The examples I have used here are true, the names have been changed to protect the ...well you get it, unfortunately, there are many other projects that are reminiscent of these examples.    I hope to give an update in a later blog.   Drop me a note on how you did executing an ERP project or mega project and the outcome, are you part of the top 10%?