Scoping your improvement program based on the problems and goals of your organization allows you to make significant headway.
by Neil Potter and Mary Sakry
The capability maturity model (CMM), developed by the Software Engineering Institute (SEI), is a model to help software organizations improve their development capability. Acceptance and use of the CMM for process improvement varies greatly in the industry—some use it as an optional guideline, while others revere it as gospel. When used appropriately, CMM can help organizations develop software with predictable cost, schedule and quality. When used inappropriately, it can waste an entire organization's time.
Many organizations approach process improvement by simply documenting each and every process. This process-centric approach may be amplified when an organization attempts to adopt a more sweeping and systematic technique for improvement such as ISO9001 or the CMM. In light of a process-centric goal stating, "Be SEI CMM Level 3 by December," the approach of documenting all processes is reinforced and might even appear natural. A process-centric approach can work, but it has a high risk of failure because most people mistake documentation for progress. The improvement effort isn't integrated with the organization's product development goals, and this usually results in a large stack of unused process documents.
The capability maturity model (version 1.1) consists of five levels of engineering and management process maturity, each level building on the next. CMM Level 1 has no criteria and represents projects that typically have large amounts of rework, numerous technical surprises, frustrated customers, and significant cost and schedule overruns. Good software can come from a CMM Level 1 organization; however, this result isn't easy to achieve or sustain. CMM Level 2 comprises sound project management, negotiations with the customer, version control, vendor management, and simple process and product assurance. Organizations at CMM Level 2 can focus on product development rather than day-to-day crises. CMM Level 3 focuses on organization-wide engineering skills, basic systems engineering, advanced project management, and an infrastructure to support sustained improvement. Organizations at CMM Level 3 consistently produce reliable software on time. A complete copy of the CMM can be found at www.sei.cmu.edu/pub/documents/93.reports/pdf/tr24.93.pdf.
Goal-Problem Approach
An alternative to the process-centric approach of improvement is to scope your improvement program based on the problems and goals of your organization. Adopting this approach enables you to make significant progress on real issues and make headway using the CMM.
Examples of an organization's business goals include the delivery of a product, the completion of a software installation or the upgrade of a database. The goal could also be the desired outcome when a critical problem has been solved. For example, an organization may be unable to hit delivery deadlines, or it might spend 75 percent of its resources on rework. The related goals to these problems would be meeting deadlines 100 percent of the time or reducing rework to 25 percent.
To use the goal-problem approach, first determine your organization's business goals and problem areas, and then compare these to the elements of the SEI CMM. Select the appropriate elements of the CMM that address these problems and help you move towards your goals.
During one session to help a company plan an improvement program, we learned that the organization was about to form six teams to work on the six key process areas (KPA) of CMM Level 2, which are requirements management (RM), software project planning (SPP), software project tracking and oversight (SPTO), software configuration management (SCM) and software quality assurance (SQA). The key process areas of CMM Level 3 are training programs (TP), software product engineering (SPE), peer reviews (PR), intergroup coordination (IC) and integrated software management (ISM). We suggested that the developers and managers temporarily forget about reaching CMM Level 2 and instead state all the major product development problems they currently faced. Then we asked them to list the business goals they were trying to achieve over the next six to 18 months. These two independent lists are shown in Table 1.
We then compared their problems and goals with the practices in SEI CMM Levels 2 and 3 and placed the related KPA names and activities in parentheses after each item (Table 1). These activities are small solutions that address the problems and support the goals. (If the company had been using ISO9001 or the Malcolm Baldrige Award, we would have mapped the problems and goals to those criteria.)
What was the scope of the improvement program? To address the problems and the goals of the organization. As you can see, 21 out of the 23 items (91 percent) map to CMM Level 2. When all the problems and goals have been addressed, 46 percent of the CMM Level 2 activities will have been addressed.
The essential difference between this approach and addressing the six KPAs in parallel is that the problems and goals indicate which pieces of each KPA you should address first. Regardless of whether your organization is using SEI CMM ISO9001 or another model or standard, the problem-goal approach will help scope and sequence your improvement program.
Items that Don't Quite Fit
Not every problem in Table 1 exactly matches the key process areas of CMM Level 2. For example, there isn't much in the CMM to help this organization specifically address the fifth goal, improve performance of core software product. In this situation, you must determine which areas are the most important for the organization to fix now. Serious problems should be worked on first, regardless of whether they relate to the CMM. Solutions to this specific goal will likely come from other technical sources.
Five lessons can be learned from adopting the goal-problem approach:
All process improvement can be meaningful.
The problems and goals can help identify which part of the SEI CMM to work on first. The CMM shouldn't be seen as an all-or-nothing approach in which all parts should be attempted at once. It should be treated as a large toolbox of little actions, ideas and solutions, each of which is useful at different times.
Any process document developed to solve a problem will be meaningful and useful. A process improvement team will be less tempted to gold-plate the process if its scope is defined by a problem.
A group's motivation to work on improvement is increased when its problems and goals are the primary focus of the improvement program.
Focusing on goals and solutions to problems prevents an organization from creating academic process documents.
At the Project Level
Here is an example of how a single project used the goal-problem approach. We first asked the project manager for a significant project goal. From this goal, we derived areas that needed improvement by asking: "What is preventing you from achieving the goal?" and "What other problems do you have related to this goal?" The first question uncovered problem areas related to the goal, and the second helped elicit further problem areas. The goal and problem list formed the scope of the improvement program for this project.
What is your goal?
Reduce release cycle to six to nine months
What prevents you from achieving the goal?
Changing requirements.
Loss of resources. It's difficult to replace people with specialized skills who leave the project.
Too many features are required for a six to nine month development cycle.
The poor quality of the incoming code from other groups.
Inadequate availability of test equipment
What other problems do you have related to this goal?
Lack of visibility within any life-cycle phase-it's difficult to know whether we're ahead or behind schedule.
We don't always have the resources available to complete the planned work.
It's difficult to find defects early
We then worked through each of her answers and noted which KPA activity could significantly help address each problem area (see Table 2). We recommended the more advanced CMM Level 3 KPA components since her group had almost attained CMM Level 2. A selection of these practices are defined in Table 3.
In this example, five out of the eight problems (63 percent) mapped to SEI CMM Level 2, and 100 percent mapped to SEI CMM Level 3. The problems and goal became the scope of the improvement program, and by addressing these real issues, the project manager can also make significant progress toward completing CMM Level 2 and starting CMM Level 3.
What questions will help you scope your improvement effort?
What one goal will you be held accountable for over the next six to 18 months?
What prevents you from achieving this goal?
What other problems do you have related to this goal?
If you are using a process improvement model (such as the SEI CMM), which items help each of the problems listed
Covering All Your Bases
One of the primary concerns that people have with the goal-problem approach is that an organization won't achieve its initial goal of CMM Level 3 or ISO90001, since attention will be diverted to the business goals and problems lists. When the first set of problems and goals have been addressed, repeat the cycle to determine the next set of problems and goals. This new set can then be compared to the remaining elements in the CMM. Over a one-to-three year period, each section of the CMM can be matched with a problem or goal.
Figure 1. Typical Phase of a Goal-Problem Improvement Program
Each time a goal or a problem is addressed, solution(s) from the CMM can be applied.
Figure 1 shows the typical phases of an improvement program, using the goal-problem approach. Each time a goal or a problem is addressed, solution(s) from the CMM can be applied.
As an example, the company who owned the problems and goals listed in Table 1, conducted an informal process assessment after 11 months of improvement work to monitor progress. The assessment showed that 50 percent of CMM Level 2 practices had been adopted, and many of the initial problems had been fixed. At the end of the assessment, they revised their problem list for the next phase of their improvement program. The problems were:
Files delivered using CD-ROMs aren't verified.
Actual data isn't recorded from the initial project plan to determine how well we did.
We don't know the status of our testing activities and when we are ready to ship (for example, defect closure and fine rate).
We don't know the specific differences between one software release and the previous one.
We don't verify that the processes we have put in place are used correctly.
Each one of these items maps specifically to practices in the SEI CMM
There are situations where some of the elements of the CMM aren't used when solving a problem or achieving a goal. These elements should be left until the end of the improvement cycle. At that time, either the outstanding elements are put to good use, or considered not applicable.
Elements are put to good use by asking the question, "What problem do we have where this element could help?" For example, baseline audits are called for in the CMM under the topic of software configuration management (SCM). A baseline audit verifies that the files that go into a software product release are the correct ones. The audit can be a human-eye check to verify file version numbers, file sizes and file names. This practice is usually ignored since it's not obvious why it's needed. Asking, "What problem do we have where this element could help?" elicits project team experiences where the wrong files have been sent to the testing group or released to the customer. Baseline audits should be employed when these situations (or problems) occur.
The goal-problem approach also helps an organization avoid prematurely using practices from the CMM. Process auditing is an example of a practice that is sometimes adopted prematurely. Process audits verify that the correct process steps have been carried out when performing engineering activities such as schedule estimation, testing, peer reviews and SCM. Process audits identify and eliminate engineering mistakes before they cause large unnecessary downstream costs. In the beginning of an improvement program, there is usually little benefit in performing process audits. However, the need for it becomes apparent once the process has been defined, used and proven effective.
One company highlighted this with its software release management process. Before release management had been improved, performing an audit on the related SCM activities would have been futile. When SCM and release management were in place, one employee by-passed the process and incorrectly released a software patch by e-mail to a customer. The software didn't work and the customer was furious. The need for SCM auditing became apparent. After the audits were performed, the developers and managers realized that they now had a mechanism to verify the execution of this essential practice.
Scoping an improvement program can be difficult and frustrating. Moreover, the task becomes daunting when a process model, such as the CMM, is adopted wholesale. However, a simple, immediately available solution exists. The goals and problems of an organization can provide a timeless and effective scope for any improvement program. A model or standard can then be used as a source of ideas, solutions and actions to achieve this scope. The resulting goal-problem improvement program is compelling and practical.
Table 1. Problems and Goals List
Problems
Need better requirements. Requirements tracking not in place—changes to requirements aren't tracked; code doesn't match specifications at test time. [Level 2: Requirements Management (RM) activities 1, 2 and 3]
Management direction unclear for version 2.3. Goals change often. [Level 2: RM activities 1 and 3 and verification 1]
Hard to revise project plan—items drop off, new things are added, plan is out of date. [Level 2: Software Project Tracking and Oversight (SPTO) activity 2, 8 and 9]
Wrong files (for example, DLLs) get put on CD-ROM—don't know what the right ones should be. [Level 2: Software Configuration Management (SCM) activities 4, 7, 8, 9 and 10]
Defect repairs break essential product features. [Level 2: SCM activities 5, 6, 7, 9 and 10, abilities 1, 2, 4 and 5, and verification 3 and 4]
Customers are unhappy. There are approximately 300 outstanding defects that haven't been addressed. [Level 2: SCM verification 1, RM activity 3; Level 3: Intergroup Coordination (IC) activity 1]
Difficult to find time to do critical activities (product development) versus crisis activities. [Level 2: Software Project Planning (SPP) activities 4, 10 and 12]
Lack of resources and skills allocated to software design. [Level 2: SPP activity 10]
Quality department needs team training (product and test skills). [Level 2: Software Quality Assurance (SQA) abilities 2, 3 and 4]
Changes to specifications and documentation aren't communicated effectively to documentation and test groups. [Level 2: RM activities 1, 2 and 3; SCM activities 5, 6, 7 and 9 and ability 1]
Unreliable project schedule estimates. [Level 2: SPP activities 5, 9, 10, 12, 13 and 14 and ability 4]
Unclear status of software changes. [Level 2: SCM activities 8 and 9]
Testing doesn't necessarily comprehend things that matter to the customer. [Level 3: SPE activities 5, 6 and 7]
Goals
Orderly plans for development. [Level 2: Software Project Planning (SPP) activities 2, 5, 6, 7, 8, 13 and 14]
Understand what our capacity is—develop one list of all the work we have to do. [Level 2: SPP activity 7 and ability 1]
Improve schedule tracking and communication of changes to impacted groups. [Level 2: SPTO activities 3 and 4]
Successfully deliver product X. [Level 2: RM activities 1, 2 and 3; SPP activities 6, 10 and 13]
Improve performance of core software product. [Level 2: SPP activity 11; SPTO activity 7]
Identify needed skills for new designers and hire, promote and train accordingly. [Level 3: Software Product Engineering (SPE) activity 3 and ability 2]
Identify tools to support software developers. [Level 2: SPP activity 14; Level 3: SPE activity 1]
Keep making a profit. Keep customers happy. [Level 2: RM activities 1 and 2; SPP activities 10, 12 and 13; SPTO activities 4, 6, 8 and 10; SQA activity 5. Level 3: SPE activities 2 and 7; IC activity 1; Peer Review (PR) goal 2]
Identify tools to support software testers. [Level 2: SPP activity 14. Level 3: SPE activity 1]
Empower Quality Department to have final say on product shipment. [Level 2: SQA activities 6 and 7]
(The lists are not in any particular order. The KPA activities in brackets are small solutions that address the problems and goals.)
[Return to text]
Table 2. The Goal and Problems of One Project
Goal: Reduce release cycle to six to nine months
Problems KPA component that would help this problem
Changing requirements. Level 2: Requirements Management (RM) activity 3; Software Configuration Management (SCM) activity 5.
Level 3: Software Product Engineering (SPE) activity 2.
Loss of resources. It's difficult to replace people who leave the project due to specialized skills. Level 2: Software Project Tracking and Oversight (SPTO) activities 2 and 8.
Level 3: Training Program (TP) activity 1; SPE ability 2.
Too many features are required for a six- to nine-month development cycle. Level 2: Software Project Planning (SPP) activities 4, 12 and 13.Level 3: SPE activity 2.
Poor quality of incoming code from other groups. Level 3: Intergroup Coordination (IC) activities 2, 5 and 6; Peer Review (PR) activity 2.
Inadequate availability of test equipment. Level 2: SPP activities 13 and 14.
Level 3: SPE activities 6 and 7.
Lack of visibility within any life cycle phase—it's difficult to know whether we are ahead of or behind schedule. Level 3: Integrated Software Management (ISM) activities 4, 7 and 11 and verification 2.
We don't always have the resources available to complete planned work. Level 2: SPP activities 4, 12 and 13.
Level 3: ISM activities 3, 5, 10 and 11.
It's difficult to find defects early. Level 3: PR activities 1 and 2 and ability 1.
[Return to text]
Table 3. CMM Activities for the First Two Problems of Table 2.
Key Process Area Practice (from SEI CMM 1.1) Definition
Requirements Management (RM) activity 3 Changes to the allocated requirements are reviewed and incorporated into the software project.
Software Configuration Management (SCM) activity 5 Change requests and problem reports for all configuration items or units are initiated, recorded, reviewed, approved and tracked according to a documented process.
Software Product Engineering (SPE) activity 2 The software requirements are developed, maintained, documented and verified by systematically analyzing the allocated requirements according to the defined software process.
Software Project Tracking and Oversight (SPTO) activity 2 The project's software development plan is revised according to a documented procedure.
Software Project Tracking and Oversight (SPTO) activity 8 The project's software schedule is tracked, and corrective actions are taken as necessary.
Training Program (TP) activity 1 Each software project develops and maintains a training plan that specifies its training needs.
Software Product Engineering (SPE) ability 2 Members of the software engineering technical staff receive required training to perform their technical assignments