PDF

Key takeaways:

• A resilient, stable PLM upgrade requires an effective quality plan and a solution to ensure that the plan is adhered to
• A key part of an effective quality plan includes automated testing technology to enable consistent and repeatable validation to be performed early, supporting timely corrective action and minimizing or eliminating surprises
• SQS’s PLM testing framework is used by Siemens PLM Software’s Teamcenter development and maintenance groups to test and validate Teamcenter
• Major manufacturing companies use SQS to ensure the quality of their PLM upgrades and implementations

What Could Possibly Go Wrong?

An enterprise PLM environment is a complex system of systems comprised of technology and processes to support the people who define and manage the product definition. Launching a new or updated PLM environment, or any enterprise solution in a manufacturing company, has many benefits, but also can have significant cost and potential risks. In most enterprises, a launch is a major event, requiring a lot of effort from the staff associated with the planning, development, testing, training, and data migration in support of the deployment. Ensuring a stable deployment helps reduce risk and is a key function of the deployment team and imperative to the enterprise’s success.

From a high level, the PLM implementation process is linear (plan, develop, test, launch), but development and testing typically require many iterations before production can begin. Testing helps to ensure that the design meets the requirements and that there are no technical surprises like programming errors or data corruption. Well planned and executed testing results in a stable launch with minimal issues; a launch with which everyone is happy. Unfortunately this process is rarely, if ever flawless.

Within manufacturing companies, proper management of the PLM solution definition is complex due to the variety and volume of data required to define products and the complexity of the processes used to control that data. Configuring and customizing the PLM environment to support the complexity often results in errors, leading to instability, a frequent occurrence in many software development projects. CIMdata regards customization as a last resort that should only be done when there is a strong business need. Testing is used to validate the configuration and any customizations to ensure the solution performs as planned.

Software testing is usually accomplished utilizing “use cases,” a predefined series of steps executed on a known set of data that produce a known output. Use cases for testing a manufacturing PLM upgrade or deployment are commonly modeled on well-defined business processes that are implemented within the PLM environment. Because PLM environments perform many different functions, ensuring that the use cases cover the entire scope is not a trivial effort.

While automated software testing is a common practice in technology companies, it is not as common within manufacturing, which relies more heavily on manual testing. This can induce significant variation in what is tested, how the tests are executed, and how the results are interpreted and reported. In addition, manual testing requires skilled people to do the testing, so ensuring the development staff and testing staff are available and aligned often results in negative schedule impacts, delaying the implementation or upgrade. Also, the number and execution time of manual testing iterations is difficult to forecast due to inconsistent execution and communication of results. All these issues create cost and schedule risks and can impact project momentum.

Domain Knowledge Required

Quality planning and testing are key parts of the software development lifecycle (SDLC) and cannot be treated as afterthoughts or grafted on at the end of a development process. They need to be incorporated from the beginning of the process to be effective. In a well-developed quality system, use cases cover all the requirements of a project, so when all the use cases have been executed successfully the product is ready for release. If a use case does not complete as expected then the software developer needs to identify and fix whatever issue caused the failure.

Developing use cases for PLM environments requires domain knowledge of PLM, the company’s processes, software quality and testing procedures, and testing tools and technology. The PLM domain knowledge and business process knowledge are the most difficult to obtain because the concepts are complicated and, in the case of the latter, particular to each company. An example of the complexity would be the integration of a CAD application. When a CAD assembly is changed, the files and assembly structure need to be captured and transformed into an EBOM. A key part of this process is to modify the data in the context of an engineering change process. Ensuring that the CAD data affected by the change and all possible paths in the change workflow have been tested is complex. The change could include the delegation of a vote in a majority-rules scenario and variants in the flow or approval roles based on the predicted cost of making the change. Configuring use case scripts to support this level of detail is difficult because the change process is complex. This level of complication is typically beyond the scope of those implementing PLM.

Automation is the Key to Stability

Ad hoc testing occurs when software quality is validated at the end of the development process, using human testers manually executing use cases (assuming the use cases even exist). This type of testing has several issues; including inconsistent execution and inconsistent evaluation of output, and it requires a lot of manual labor. The inconsistencies that ad hoc testing generates result in errors, mistakes, and even catastrophic failures, leading to last minute launch delays. Today’s modern testing technology can be integrated within the software development environment and processes. In virtually all commercial software companies this is common practice. Out of necessity, leading manufacturing companies are also starting to get on board with automated testing.

Automated testing enables testing to be done more often, in many cases almost continuously, so developers can catch errors in context while they are intimately involved with the offending code. As code modules are integrated with the full solution, tests are run to catch issues that manifest across modules. Since the use case scripts are repeated over and over again failures in code that was previously working are dramatically reduced.

Automated testing provides reporting tools that enable better test management. The tools manage the testing process and generate metadata that can be used for schedule planning and to assess progress. For example, as new software is developed the number of bugs grows, peaks, and approaches zero at the time of release. By understanding the bug identification and fix rates, the release date can be predicted and mitigations put into place if the release date is likely to slip.

Quantifiable Business Benefits

Automated PLM testing has quantifiable business benefits. While significant effort may be required to set up the test system and use cases, the return on investment is impressive. In most cases, a complete suite of automated tests can be run in a few hours, as opposed to being done by a room full of people manually executing steps for days on end. The automated use cases will be run the same way every time, and regressions will be caught. The use case library and testing process become assets that support major and minor upgrades, reducing risk. Automated reporting provides project, program, and executive management with quantitative data to support decisions. It also enables them to structure a complete quality planning program not just quality assurance. And finally, developers get better definition of bugs since the use case scripts are well defined and repeatable. This also helps reduce bug fixing time. A good testing program also reduces disruptions for those who are users of the PLM solution because fewer problems are likely to migrate into the operational solution.

SQS Testing Service

SQS, founded thirty years ago, and headquartered in Germany, is a leading global independent software quality management and testing company. They have a Quality Center of Excellence in Pune, India, dedicated to PLM and manufacturing software, with sales offices in over 27 countries. SQS has been testing Siemens PLM Software’s Teamcenter software for nine years, since Teamcenter Version 8.3. They are currently testing Teamcenter Version 11.1. CIMdata talked with Siemens executives and they spoke highly of SQS and contributions they have made to the Teamcenter product.

The testing framework used by SQS is flexible, providing many options that help ensure high quality. A key SQS differentiator is that they have thousands of use cases to test Teamcenter. These were developed in partnership with Siemens PLM Software. SQS is expanding their business to support Teamcenter based manufacturing companies. The use case libraries maintained by SQS are used as a baseline and extended to cover the configuration and customization done by industrial companies. SQS staff tailor the use case scripts to support their customers’ environments including MCAD and ECAD tools, other enterprise solutions, and proprietary tools. Customers that CIMdata has talked to stated that SQS is quickly able to adapt the testing solution to their environments.

Concluding Remarks

PLM environments are mission critical in many companies and downtime is not tolerated. Launching a minor or major update can be a high-risk project. While testing may be done, management can never be sure how well a solution was tested. Automated testing requires investment but provides repeatable, quantifiable results and helps to ensure that updates and deployments stay on schedule while keeping management informed. SQS has the confidence of Siemens PLM Software, as demonstrated by their Teamcenter testing partnership, as well as a growing number of happy industrial customers. CIMdata sees quality planning and assurance as a critical step within a PLM program and is impressed by the focus SQS has on PLM solution quality planning and testing. The results of the testing provided by SQS should help PLM project managers and those responsible for the PLM solution to sleep better at night.