CSV vs. CSA: Moving from Software Validation to Assurance
CSV vs. CSA: Moving from Software Validation to Assurance
John Todd, Sr. Business Consultant/Product Researcher, Total Resource Management (TRM), Inc.
Posted 2/29/2024
Oh, the exciting world of software validation! Hours and hours of white-knuckled test development, execution, and… as if those are not enough… resolution and documentation of the results! Then of course are the continued change management efforts needed to keep the system in a validated state over its lifecycle. It takes truly special people to function in this arena.
The FDA has recognized that the degree of effort and documentation required to maintain validated systems has been standing in the way of industry adopting software tools to automate processes and make moves into the cloud. Based on feedback from those impacted by validation methods and requirements, the FDA has made a course change.
The approach and guidelines provided by the FDA, last updated back in 2002, were under the term “Computer Software Validation (CSV).” General Principles of Software Validation is a guidance document that is deeply understood by those who operate as a regulated entity. This document refers to the original quality system regulation (Title 21 CFR, Part 820) and other applicable regulations. This document is also where the concepts of requirements definition, installation, operational, and performance qualification activities have been used to describe the validation process.
The new approach as outlined in the draftComputer Software Assurance for Production and Quality System Software (CSA), acknowledges that advances in manufacturing technologies (including software) enable reductions in sources of error, optimize resources, and reduce consumer and patient risk. By drafting this new guidance, the FDA wants manufacturers to take a more risk-based approach, assuring that the software systems being implemented are functioning as expected, focusing on those areas and functions of the tools that present the most risk.
From then to now…
In the past, testing and other verification activities have been the workhorses at each stage of the development lifecycle used to validate software. This kind of testing does not provide the confidence the FDA nor the regulated entity are looking for, if the goal is to determine the fitness of the software for its intended use in the context of the business and process operation.
CSA takes a risk-based approach intended to prevent the appearance of defects in the software development and deployment lifecycle. While still important and required by the validation requirements stated in 21 CFR 820, the hope is that the approach will enable manufacturers to pay more attention to product quality and adopt new technologies. Focus on the features/functions of the software that present the most risk in their use, rather than those that have little or no risk at all.
What does the CSA framework look like?
The CSA framework is easily digestible in that it includes only a few elements. While the details under each are considerable, such as determining the risk approach, the guidance leaves much up to the regulated entity to decide what applies and how to approach the tasks given their context.
The elements are:
Identify the intended use (of the software features/functions)
Determine the risk-based approach
Determine assurance activities
Establish appropriate records
The guidance begins with asking the question if the intended software even needs to be placed under the assurance umbrella. If the software is not used for production purposes, or a part of the quality system, assurance may not be necessary. If the software is used directly or in support of production, it will likely need to be under the assurance program.
Then comes understanding the intended use of the software as a whole and the features/functions it provides. A reasonable list of how each of the features/functions that are intended to be used provides a foundation for what the best risk-based approach would look like.
For example: If the software is simply used to document a reading in support of a quality record, this may be a low-risk situation. The initial assessment of the vendor, the installation, and any configuration may be sufficient to state the software is fit for use.
However, if customization has been performed to suit process needs, such as formulas and other potential sources of error, then the risk is increased and needs to be addressed.
Determining the risk approach begins with matching potential failures to the features/functions that are intended to be used. Given that a certain feature is going to be used in a certain way, the goal is to answer the questions: How might it fail? And, What are the risks that the failure would present? High risk features/function are those that impact the process and/or product quality that compromises safety. (The CSA document mentioned earlier provides several examples of high- and low-risk failures of software features/functions.)
Given a software feature/function has been determined to present a high process risk, assurance activities must be identified. Any assurance activities need to be in line with the risk level. High-risk items will have more rigorous testing and documentation than those deemed low risk.
What do assurance activities look like? Mostly there are various approaches to testing the focused features/functions. Automated and scripted testing are at one end of the spectrum, while unscripted or exploratory are at the other. No matter the activity, the results and resolution of the issues discovered remain important and required outcomes that must be documented.
The final piece to the framework is the definition of appropriate records that capture the evidence that after assurance activities, the software can be deemed ‘fit for use’ or ‘performs as intended.’ These records would include elements of the framework such as the intended use of the feature/function, the risks its use could present, the results of the testing, who did the testing and when, and how issues were resolved. Electronic audit trails, logs, and automated test result sets are preferred to manual or paper-based testing results.
Tools available for Maximo/MAS Manage Assurance
IBM has long provided test scripts (and results) of the out of box features/functions of Maximo via the Life Sciences add-on. With the release of MAS Manage, IBM has continued to release these foundational test scripts, now with a focus on the use of the well-known Selenium software testing automation toolset. This is known as the Maximo Testing Automation Framework (TAF). While there is a learning curve to setting up and running the tests within Selenium, they do produce clear and acceptable electronic records of the pass/fail results. Selenium also has built-in change management and data governance functions.
Further, TRM RulesManager Studio has a feature set called RampUp for developing test scripts. They can be developed traditionally with limited code required, or visually by recording and then playing back simulated users exercising the focused features/functions. RulesManager Studio is utilized on most Maximo/Manage instances we stand up for clients, so it is well known and easily adopted for your software assurance testing.
Wrap up – Software Validation
This change in guidance from the FDA for software validation has been greatly anticipated by regulated entities. (Remember, it is draft mode still – not yet implemented) By promoting and supporting a risk-based approach, the FDA is helping manufacturers take advantage of new technologies, move their computing systems to the cloud, and turn their focus more on product quality, patient safety, and innovation.
TRM has worked with many clients over the years in support of their validation efforts around Maximo, and now MAS implementations, on-premises and in the cloud. We are uniquely qualified to work with your Quality Assurance and Operations teams to not only maintain validation of your current systems, but to also assist in your CSV/CSA activities.
John Q. Todd
John Q. Todd has nearly 30 years of business and technical experience in the Project Management, Process development/improvement, Quality/ISO/CMMI Management, Technical Training, Reliability Engineering, Maintenance, Application development, Risk Management, & Enterprise Asset Management fields. His experience includes work as a Reliability Engineer & RCM implementer for NASA/JPL Deep Space Network, as well as numerous customer projects and consulting activities as a reliability and spares analysis expert. He is a Sr. Business Consultant and Product Researcher with Total Resource Management, an an IBM Gold Business Partner – focused on the market-leading EAM solution, Maximo, specializes in improving asset and operational performance by delivering strategic consulting services with world class functional and technical expertise.
ISO 12100 is a fundamental safety standard being developed that will apply to a very broad array of machinery. There are many machines that have specific industry consensus standards that pertain to the particular equipment (e.g., power presses, robots, packaging machinery). There are many more machines for which no industry specific standard exists. In this case, the ISO “A level” standard ISO 12100 Safety of Machinery applies. Complying with the requirements of ISO 12100 is a critical first step in achieving mobility of equipment and consistent operations.
ISO 12100 is a fundamental safety standard being developed that will apply to a very broad array of machinery. There are many machines that have specific industry consensus standards that pertain to the particular equipment (e.g., power presses, robots, packaging machinery). There are many more machines for which no industry specific standard exists. In this case, the ISO “A level” standard ISO 12100 Safety of Machinery applies. Complying with the requirements of ISO 12100 is a critical first step in achieving mobility of equipment and consistent operations.
Many smaller companies feel maintenance management software is overkill for their organizations. Nothing could be further from the truth. In reality, even a one-person maintenance department can reap the benefits of maintenance management software. The same benefits realized by the maintenance crew in larger companies are there for smaller maintenance departments also.
Many smaller companies feel maintenance management software is overkill for their organizations. Nothing could be further from the truth. In reality, even a one-person maintenance department can reap the benefits of maintenance management software. The same benefits realized by the maintenance crew in larger companies are there for smaller maintenance departments also.
Besides being the central hub for maintenance, the storeroom also provides functions that are absolutely critical to the maintenance operation. These functions are so important that when the storeroom is operating in a best practices mode, the rest of the maintenance operation can excel – the storeroom is the enabler. Put another way, if the storeroom is run improperly (such as poor inventory accuracy, parts unavailable when needed due to poor replenishment and procurement practices, etc), the rest of the maintenance operation has no chance of achieving high service levels of equipment availability and reliability.
Besides being the central hub for maintenance, the storeroom also provides functions that are absolutely critical to the maintenance operation. These functions are so important that when the storeroom is operating in a best practices mode, the rest of the maintenance operation can excel – the storeroom is the enabler. Put another way, if the storeroom is run improperly (such as poor inventory accuracy, parts unavailable when needed due to poor replenishment and procurement practices, etc), the rest of the maintenance operation has no chance of achieving high service levels of equipment availability and reliability.
We now have the ability to automate many of our standard maintenance processes, analyse in detail various parts of our businesses, and the performance of our equipment. We are able to plan shutdowns, technical change projects and operational maintenance procedures down to a very fine level of detail. As maintenance management generally makes up around 40 - 50 % of operational budgets, the savings made possible from increased efficiency and reduction of waste are staggering.
We now have the ability to automate many of our standard maintenance processes, analyse in detail various parts of our businesses, and the performance of our equipment. We are able to plan shutdowns, technical change projects and operational maintenance procedures down to a very fine level of detail. As maintenance management generally makes up around 40 - 50 % of operational budgets, the savings made possible from increased efficiency and reduction of waste are staggering.
Facility management personnel in all industries have waited a long time for computer technology to become applicable and affordable enough to implement in their respective maintenance efforts. In recent years, flexible, dependable and economical computerized maintenance management systems (CMMS) have become available to help fight the never-ending struggle to operate and maintain buildings of all shapes, sizes and functions.
Facility management personnel in all industries have waited a long time for computer technology to become applicable and affordable enough to implement in their respective maintenance efforts. In recent years, flexible, dependable and economical computerized maintenance management systems (CMMS) have become available to help fight the never-ending struggle to operate and maintain buildings of all shapes, sizes and functions.
Training for users of a computerized maintenance management system (CMMS) is a multiple-phase process. There are three areas of training needs: basic training, application (CMMS) training, and internal training.
Training for users of a computerized maintenance management system (CMMS) is a multiple-phase process. There are three areas of training needs: basic training, application (CMMS) training, and internal training.
The effectiveness of maintenance can make the difference between success and insolvency, between limping by financially and organizational excellence. Many companies and organizations consider their computerized maintenance management system (CMMS) and processes as catalysts for achieving a competitive advantage. Yet, good intentions are only part of the recipe for success.
The effectiveness of maintenance can make the difference between success and insolvency, between limping by financially and organizational excellence. Many companies and organizations consider their computerized maintenance management system (CMMS) and processes as catalysts for achieving a competitive advantage. Yet, good intentions are only part of the recipe for success.
The degree to which RCM analysis can contribute to profitability is directly variable with the quality and accuracy of the CMMS data on which the analysis is based. Because the technique is rigorous, its over-use will defeat the purpose and lead to cost overruns instead of savings. The way to optimize the RCM return is to apply the analysis strictly to the equipment and systems that will pay off from it, and to know this we must rely on the CMMS. This paper provides guidance for ensuring that the equipment data and history residing in a CMMS are complete and accurate; so that RCM analysis will be a success and positively impact a company’s bottom line, not hurt it.
The degree to which RCM analysis can contribute to profitability is directly variable with the quality and accuracy of the CMMS data on which the analysis is based. Because the technique is rigorous, its over-use will defeat the purpose and lead to cost overruns instead of savings. The way to optimize the RCM return is to apply the analysis strictly to the equipment and systems that will pay off from it, and to know this we must rely on the CMMS. This paper provides guidance for ensuring that the equipment data and history residing in a CMMS are complete and accurate; so that RCM analysis will be a success and positively impact a company’s bottom line, not hurt it.
Maintenance management is an around-the-clock challenge. Equipment failure can (and does) happen at inconvenient times–times when maintenance managers may be off-site and must rely on communication from technicians to convey problems and act efficiently. Mobile computerized maintenance management systems (CMMS) and applications provide live, synchronized status reports and other key features that can dramatically improve reaction time and efficiency.
Maintenance management is an around-the-clock challenge. Equipment failure can (and does) happen at inconvenient times–times when maintenance managers may be off-site and must rely on communication from technicians to convey problems and act efficiently. Mobile computerized maintenance management systems (CMMS) and applications provide live, synchronized status reports and other key features that can dramatically improve reaction time and efficiency.
The latest ARC Advisory Group study of the enterprise asset management (EAM/CMMS) software market profiles more than 80 maintenance application suppliers. This number of choices is enough to make any plant manager shudder. How can you pick the tool that is right for you? The tug-of-war between decision makers can make or break the success of an EAM/CMMS implementation. This article addresses the delicate balance of wants and needs, and how they apply in the application selection process.
The latest ARC Advisory Group study of the enterprise asset management (EAM/CMMS) software market profiles more than 80 maintenance application suppliers. This number of choices is enough to make any plant manager shudder. How can you pick the tool that is right for you? The tug-of-war between decision makers can make or break the success of an EAM/CMMS implementation. This article addresses the delicate balance of wants and needs, and how they apply in the application selection process.
Now that the best EAM/ CMMS application for your business has been selected, the deployment phase begins. But an EAM system is not a plug-and-play application. Will you miss out on some real operational savings? Will the data be clean enough to provide value in a production environment? Will processes be tailored to match the new system, or will they be optimized to improve business? Is the vendor knowledgeable about your specific industry or regulatory requirements?
Now that the best EAM/ CMMS application for your business has been selected, the deployment phase begins. But an EAM system is not a plug-and-play application. Will you miss out on some real operational savings? Will the data be clean enough to provide value in a production environment? Will processes be tailored to match the new system, or will they be optimized to improve business? Is the vendor knowledgeable about your specific industry or regulatory requirements?
A whopping 94.7 percent of plant maintenance managers feel they are not using their computerized maintenance management software system to its maximum capability, according to the results of a national CMMS survey conducted for Reliable Plant magazine by educator, consultant and author Kris Bagadia. “I knew that it was going to be a high percentage. I didn’t know it was going to be that high,” says Bagadia.
A whopping 94.7 percent of plant maintenance managers feel they are not using their computerized maintenance management software system to its maximum capability, according to the results of a national CMMS survey conducted for Reliable Plant magazine by educator, consultant and author Kris Bagadia. “I knew that it was going to be a high percentage. I didn’t know it was going to be that high,” says Bagadia.
What is a failure code? Quite simply, it is a code that illustrates why an asset failed or the reason that the asset failed. Codes can be a number which is cross referenced to a list of actual code descriptions or more conveniently a series of alphanumeric characters that are a logical abbreviation of their descriptions. However, with modern database technology and available disk space, the full descriptions are increasingly being used instead of alphanumeric codes.
What is a failure code? Quite simply, it is a code that illustrates why an asset failed or the reason that the asset failed. Codes can be a number which is cross referenced to a list of actual code descriptions or more conveniently a series of alphanumeric characters that are a logical abbreviation of their descriptions. However, with modern database technology and available disk space, the full descriptions are increasingly being used instead of alphanumeric codes.
An elephant is a large animal and it is doubtful anyone would want to eat one. But the old proverb, with a little twist, has a similar paradox to implementing a computerized maintenance management system (CMMS). Not developing the proper steps to implementation may lead a company to failure.
An elephant is a large animal and it is doubtful anyone would want to eat one. But the old proverb, with a little twist, has a similar paradox to implementing a computerized maintenance management system (CMMS). Not developing the proper steps to implementation may lead a company to failure.
Most, if not all, companies use CMMS systems to oversee their maintenance activities. From home-grown systems to complete ERP systems, leveraging technology allows companies to more efficiently and effectively manage their maintenance, repair and operations activities. So as a core maintenance function, surely routine, lubrication-related preventive and predictive activities such as regreasing motor bearings, taking oil samples, and executing oil top-offs and inspections belong in the CMMS system like any other maintenance task, right?
Most, if not all, companies use CMMS systems to oversee their maintenance activities. From home-grown systems to complete ERP systems, leveraging technology allows companies to more efficiently and effectively manage their maintenance, repair and operations activities. So as a core maintenance function, surely routine, lubrication-related preventive and predictive activities such as regreasing motor bearings, taking oil samples, and executing oil top-offs and inspections belong in the CMMS system like any other maintenance task, right?
