Adoption of People CMM® -01: Appraisal Results

The People CMM® has been available for more than 15 years. Version 1.0 of the model was released in 1995 and version 2.0 was released in 2001. So, maybe it is time to look at the extent of adoption by the industry and benefits accrued to organizations that have adopted the model.

There are many ways one can evaluate the extent of adoption; one way is to look at the volume of Class-A appraisals done (Class-A appraisals are the only recognized way of getting a maturity level in People CMM®). Earlier, the Class-A appraisals were done using People CMM® Assessment method. This method was replaced with the SCAMPI^SM-A appraisal method, since 2006-07.

Without further verbiage, here is the data, in the form of a trend chart:

Here is the data in a tabular format, with more details:

Some context on the data above:

1. Data for the years 2002-2007 is picked up from a presentation titled People Capability Maturity Model: Product Suite Maturity Profile (January 2008) by the People CMM® Team at the SEI.
2. For the years 2008-2009, there is no officially compiled data easily available – the numbers are extrapolated based on the appraisals done by the most active Lead Appraisers in those years.
3. The 2010-2011 data is picked up from the Published Appraisal Results website maintained by SEI. Some appraisals may be missing from the data, if the appraised entity did not wish to publish the data (some organizations do not like the data to be published, some decline permission because they are embarrassed by the maturity level rating that they have got ).

Going back to the graph, there seems to be an alternating trend, every 2-3 years. There is a peak of 10+ appraisals, followed by a dip to around 4-5 appraisals in the next year. Maybe, People CMM® is a seasonal flavour! . [Actually the data points are too few to reach any conclusion about trends].

The numbers are not flattering – given that the model has been in the market for so many years, just 4-14 appraisals per year (across the whole wide world) is very low. Not more than 2 Lead Appraisers are required to handle this volume!

Further analysis of the past 15 appraisals (in the last 2 years 3 months) listed in the Published Appraisal Results website (with Filter People CMM® v2.0) maintained by SEI shows the following:

• Geographic spread: India-8; China-3; Philippines-1; Oman-1; UK-1; Malaysia-1. It is interesting to note that there are no appraisals in the US.
• Industry spread: IT-8; BPO-3; Banking-1; Utilities-1; Engineering-1. So, not a model “just for software organizations”.
• Most of the appraisals are led either by Sankararaman Dhandapani or by Rajesh Naik of QAI India Ltd. The last fifteen appraisals are accounted for between four LAs (out of the 13 LAs listed for People CMM® in the SEI Partner Directory).

Hope the trend of low number of appraisals is broken in the coming years.

Other related posts uploaded on the same blog:

• Adoption of People CMM® -02: Benefits Experienced
• Adoption of People CMM® -03: Why is it Low?
• Interview with Prakash Hegde: Experiences with People CMM® Implementation
• Interview: Monty Bharali (Head-HR, DSTWS) shares his experience with the People CMM® in a BPO-IT context

^® CMMI and CMM are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
^SM-SCAMPI is a service mark of Carnegie Mellon University.

Hi,

If you like the posts on this blog and would like to be informed whenever a new entry is made, here is what you can do:

• Scroll back to the top of the page
• On the right hand side there is section called “FollowBlog via Email”
• In the space provided, type in your email id and click on the “Follow Blog” button (give your personal id, since companies often block wordpress)
• You will get an email at that email id
• In the email there is a clickthrough link – click on it and complete the formalities

CMMI® version 1.3 Released

CMMI® Version 1.3, originally scheduled for November 1, 2010, was released a few days before the planned release date. The release converts Development, Services, and Acquisition to version 1.3.

The new model documents can be downloaded as follows:

• CMMI® for Development: www.sei.cmu.edu/library/abstracts/reports/10tr033.cfm
• CMMI® for Services: www.sei.cmu.edu/library/abstracts/reports/10tr034.cfm
• CMMI® for Acquisition: www.sei.cmu.edu/library/abstracts/reports/10tr032.cfm

You can also see a quick summary of the changes at the blog post of August 23, 2010.

What to Expect in the new version of CMMI® for DEV Version 1.3

The first day (17th August 2010)  of SEPG Asia-Pacific 2010 conference covered the changes expected to the CMMI® models, as a part of  release of V1.3. The tutorial was conducted by Mike Phillips of the SEI and was attended by a large group of professionals (mostly from the IT industry).

Here are the key points that I have gathered and my reactions to some of the changes in the DEV model. The detailed presentation can be downloaded from here [Thanks Mike ]

A summary of the changes are available in a presentation titled CMMI v1.3 – What’s New on slideshare.

Changes to the Generic Goals and Generic Practices (DEV Model)

1)    Generic goals 4 and 5 have been removed from the models. So, generic goals stop at GG3 for all process areas. [Reaction: Good. The material in GG4 and GG5 was very scanty, and could not be used to implement CL4 and 5 practices]

2)    No significant changes in the intent of generic practices, other than a few changes in the verbiage for a few generic practices (GP 2.6, 2.9 and 3.2) [Reaction: It would have been nice if some generic practices were merged, for example GP2.8 and GP2.10, to reduce the number of generic practices. Maybe in version 1.4 or later ]

3)    In the model book (or technical report), the generic goals and generic practices are described just once at the start of the document and are not repeated for each process area  [Reaction: Slimmer book to carry, less trees to be chopped, nice touch]

Changes to the Maturity Level 2 Process Areas (DEV Model)

1)         Requirements Management (REQM) has been shifted to the Project Management category of PAs [Reaction: Makes no difference, except that there are no engineering PAs at maturity level 2. Imagine a maturity level 2 development company saying "we have great management and support practices, but our engineering practices may not be...."]

2)         Supplier Agreement Management (SAM) has been simplified. Two contentious practices of SG2 (erstwhile SP2.2 & 2.3), have been converted to sub-practices of other specific practices [Reaction: These two practices were often a source of grief to many organizations in their appraisal. Fantastic!]

Changes to the Maturity Level 3 Process Areas (DEV Model)

1)         The optional IIPD addition (one goal in OPD and one goal in IPM) has now been converted into specific practices in OPD and IPM (one additional practice each) [Reaction: This is pity, because IPPD has a great value. I would have liked to see more emphasis on IPPD with greater clarity, instead of IPPD becoming 2 practices in the whole of CMMI®]

2)         No other changes, other than changes in the language to bring in more clarity.

Changes to High Maturity Process Areas (DEV Model)

1)         OID has been renamed as Organizational Performance Management (OPM). A new goal has been added to align process improvements to business objectives and process performance data. [Reaction: Was always required. Though the change looks big, most high maturity organizations would already be implementing the requirement of this new goal. However, the new name "Organizational Performance Management" is an overkill and misleading]

2)         Quantitative Project Management (QPM) has been made tighter and the requirements are more explicit. No significant change in the intent of the process area.

3)         Causal Analysis and Resolution (CAR) and Organizational Process Performance (OPP) have undergone some changes in the verbiage, though nothing significant in intent.

Version 1.3 of DEV (along with SVC and ACQ) will be released in November 2010.

SCAMPI^SM-A appraisals using version 1.2 of the model can be conducted for period of 12 months after the release of version 1.3. Organizations aiming for an appraisal in the later part of 2011 should consider switching to version 1.3 right away.

The SCAMPI-A methodology is also undergoing an upgrade. The SCAMPI^SM methodology upgrade will be released slightly later. So, organizations could use the current SCAMPI^SM-A version 1.2 to appraise organizations for CMMI® version 1.3 for some time.

Will try and post about SVC and the expected changes to the appraisal methodology sometime soon.

What is (Project) Success in a High Maturity Organization?

Project success is measured by comparing the actual performance with what was budgeted, planned and committed – typically with respect to parameters of cost, schedule and quality. Projects that meet all parameters are considered completely successful, and those that meet some parameters are considered less successful. Projects that fail in most/ all parameters are labeled as failures. Of course, sophisticated systems may even use the extent to which they missed the objectives (near miss or missed by a mile/ kilometer) as a factor in determining the degree of success or failure.

Is this really how a high maturity (HM) organization (in terms of the CMMI® framework) should evaluate project success? I believe that the refinement in process and project management maturity should be used to fine-tune how we evaluate success.

A HM organization is “aware” that all processes have variations inherent in them. It “knows” that projects (that are composed of the processes) have a probability of achieving success in their objectives, but success is not guaranteed. The role of project management (esp. QPM) is to continually evaluate the probability of success and maximize the conditions to improve that probability.

When a single project goes through its life, those probabilities will play out. Which means that even if the probability of completing the project within its budget was 90%, a single project can overshoot the budget. Of course, if we run similar projects millions of times, only 10% of the projects will overshoot the budget; but we have only one project here. In such an “aware” organization, is the use of “actual budget compliance” a right way to measure success? If so, how is this organization different from a non-HM organization?

I believe that in a HM organization, project success should not be measured by after-the-fact results, but by the rigor and continual alignment of the project to maximize the probability of success. So, in a HM organization, a project is successful, if and only if:

*    The project, at start-up, consciously makes choices (composes the defined process, aligns plans) that maximize the probability of meeting its multiple objectives

*    The project continually evaluates the probability of meeting the objectives and revises its choices to maximize its probability of success

Now, in such an organization, the “best project” award may be given to a project which in the conventional sense has actually failed – such an organization would be truly acting on the belief- “if we implement the process, the results will eventually follow”.

Your comments?

What comes first – SPC or a stable process?

An interesting topic, which has been discussed very often. In every discussion, people agree on what is right and what needs to be implemented. But in actual implementation the principles are forgotten. Therefore it is good to re-align ourselves to the basics time and again.

What is often seen in actual implementation of SPC (ineffective and incorrect implementation):

1)    A process is documented and used

2)    Data related to the process is collected

3)    When we need to do sub-process control (because we are aiming for High Maturity rating), an SPC chart is prepared.

4)    Data which are outliers are thrown out (root cause analysis is not possible, because the outlier data belongs belongs to a distant past, and the causes are lost in the mists of time)

5)    Control limits are recalculated

6)    Steps 4) and 5) are repeated till all (remaining) points demonstrate process stability

7)    The SPC parameters (center line, UCL/ UNPL, LCL/ LNPL) are declared as baselines and used for sub-process control. The fact that the limits are too wide or that a lot of data points were thrown out (without changing anything in the process) is ignored.

What we have in the above scenario is a maturity level 2/ 3 organization using maturity level 4 tools. Usage of tools alone does not increase maturity. We cannot create a stable process through the use of SPC, we can only confirm the stability of the process through SPC and get signals when the process is out of control or shows changes in trends.

The More Effective Implementation of SPC:

1)    A process is documented and used. As the process is used, variations in the interpretation of the documented process are qualitatively analyzed. Actions are taken to augment the process definition, training and orientation till the interpretation and the qualitative understanding of the process is consistent.

2)    Process compliance audits (PPQA audits) on the implementation of the process identify more actions that need to be implemented to fine-tune the definition, training and orientation related to the process.

3)    Once the audits show consistent compliance, data related to the process performance are collected. Integrity of the data is checked and the data collection process is streamlined and consolidated- till the collected data demonstrates the required credibility

4)    Now we start looking at the data somewhat quantitatively (without using full SPC) – does the trend chart show stability? Is it showing too much dispersion/ variation? Based on the findings, the definition, training and orientation related to the process is refined further

5)    This is point we start using SPC charts to confirm process stability. Each inflection of instability is analyzed. Corrective and preventive actions are identified to further standardize the process, based on analysis of past instability. Once we are sure that causes of those inflections are removed, we can remove the points from the analysis.

6)    We are still left with points which show instability, and our CAR analysis tells us that some of the causes are truly extremely rare events. These are then removed from the data pool. Now all the remaining points are a part of the process. If the process still shows instability, then we can do further analysis – are these really part of a single process? Beneath the surface, are there two or more processes, and we need to separate out the data (e.g., the process may behave differently in the “performance appraisal season”? )

Having followed all the above steps, we now have a basis (and hence baseline) for an effective implementation of SPC.

Remember: We cannot create a stable process through the use of SPC, we can only confirm the stability of the process through SPC.

Size Does Matter! (for baselines and sub-process control) -Continued

Let us take the example of  examination/ test centers, that run an exam throughout the year, every day. Past one-year data shows – 30% of the candidates pass the exam and 70% fail the exam, all over India.

The Bangalore test center handles around 1000 candidates per month, whereas the Mysore center handles around 100 per month. Over the last one year, both centers have shown the same 30 pass: 70 fail ratio.

For the month of June 2010, one center has reported 38% pass and another has reported 29% pass. Which center (Bangalore or Mysore) is more likely (has a higher probability) to have reported 38%?

Well, Mysore is more likely to have the higher deviation from the average (+8%) than Bangalore (-1%), because Mysore, handling lesser candidates, has a lesser number of opportunities to “average out”. An easy way to figure this out is to take the case of a center that handles only 1 candidate. This center can have either 0% or 100%  pass percentage; a -30% to +70% deviation from the average.

Let us now get back to the process performance baselines that we create and the way we do sub-process control. Here are some things that we need to keep in mind while creating, publishing and using baselines:

1) Baseline (mean and standard deviation) for a sub-process parameter (like coding productivity) will be different depending on whether we consider each the coding phase of each project as a data point, or we consider each program coded in each project as a data point. The standard deviation in the first case (large base) is likely to be smaller than the second case (small base).

2) When we publish performance baseline data, we need to qualify it with the level of detail at which it applies.

3) When we use the baseline data to do sub-process control, it needs to be applied to the same level of detail. So, to do sub-process control on program level coding productivity, we need to use the baseline that was created using programs as data points (not each projects as a data points).

4) Baselines need to be created using similar situations of the base data. For example, we cannot combine the coding productivity on large programs with the productivity on small programs. Even if the average/ mean remains the same, the standard deviation will be higher when we take data from a smaller base as against a larger base.

The above points are not just “nits” but have an impact of the usefulness of baselines and sub-process control. Incorrect usage of baselines leads to incorrect displays process instability / stability.

Size Does Matter! (for baselines and sub-process control)

Here is a small brain-teaser.

Let us take the example of a examination/ test centers, that run an exam throughout the year, every day of the year. Analysis of the past one-year data shows that 30% of the candidates pass the exam and 70% fail the exam, all over India.

The Bangalore test center handles around 1000 candidates per month, whereas the Mysore center handles around 100 per month. Over the last one year, both centers have shown the same 30 pass: 70 fail ratio.

For the month of June 2010, one center has reported 38% pass and another has reported 29% pass. Which center (Bangalore or Mysore) is more likely (has a higher probability) to have reported 38%? Why do you think so?

See my post dated August 3, 2010 for the answer and implications.

Why Can’t Metrics be Used for Performance Appraisals?

While discussing collection and usage of metrics, one often hears an emphatic “We should not use metrics for individual performance management!”. The statement is made as if it is an unquestionable tenant of the religion called process management.

“And pray, why not?” Why should the performance management process be deprived of metrics? A process oriented organization would definitely not like to boast that their performance management system is completely subjective.

Here are some reasons why metrics should be used for individual performance management.

*    An individual performance management (including the appraisal part) needs to be SMART – the “M” stands for measurable.

*    Most individual performance parameters are the similar to and derived from the project, product and process objectives, they typically relate to cycle time, quality (defects), meeting commitments (schedule) and productivity (cost, effort and usage of resources).

*    A strong metrics system, that provides accurate, precise and valid data can support the project, process and individual performance management requirements.

*    Using the same sources of data, we can create a more aligned organization – the individual objectives are aligned to the project, product and process objectives. In this manner, individuals know that meeting their individual goals helps in meeting the other goals (and vice versa); conflict of interest is minimized.

The situations where we may not want to use process/ project metrics for managing individual performance are:

*    The metrics collection system is not stable, and there questions on the credibility of the data. In such a case, the use of the data for managing the project/ process is also diluted.

*    Usage of the data for individual performance management may make the individuals sabotage the process and the accuracy of the metrics. In which case, we need to strengthen the process and make it sabotage proof.

In the old SW-CMM® days, most metrics collection systems were unstable, and hence many experts of that time were pretty insistent on the metrics not being used for performance appraisals – some organizations even have policy level statements for the same!

We have now moved on from the SW-CMM® days for process management, so we need to move on in other aspects too.

Your comments?

Generating Lots of Data through Monte Carlo (a misuse?!?)

I have seen the metrics groups of organizations generating “enough” data for creating process performance baselines, from very few available data points, using Monte Carlo simulation.

Here is the method they use: Ten data points are available; using the pattern of the ten data points, they generate a thousand (or maybe a million) data points using Monte Carlo simulation. Now they feel that they have enough data points to generate a baseline.

But in reality the baseline has been generated using 10 data points. The 1000 data points only give a feeling of having lots of data and this is clearly a misuse of Monte Carlo simulation.

Normal Distribution is Actually Rare

When we often use statistical analysis tools and techniques, the underlying assumption is that process/ sub-process displays a “normal” behavior. Even if the limited data that we have shows non-normal behavior, we assume that the reason is the lack of data, and we approximate the distribution to normal.

This assumption and subsequent analysis, conclusions and decisions are therefore inaccurate, especially if we are combining “assumed” normal behavior across multiple processes, viz Process Performance Modeling.

“Normal” behavior is very rare in real life. For example, you travel from your home to office, let us say usually in 1 hour. The least time you have ever done the trip is in 30 mins. If the distribution was normal, the worst time should have been 1 hour 30 mins (symmetrical on both sides). You will find that on some days that you were delayed, the time could have been 2 or even 3 hours!

Another way of saying that real life does not behave in a “normal” way, is “there is a limit on how well you can do, but no limit on how badly you can screw up!”

There is more on this in the books “Fooled by Randomness” and “Black Swan” by Nassim Taleb — must-reads for anyone involved in high maturity CMMI® implementation.