Processes for Technical Performance Measures (TPM)

6.3.3 Define Performance and Design Constraint Requirements

Tasks 11, 13 and 14 shown in Figure 6-5, posted Wed. Dec. 29, 2010, are related to performance and design characteristics and are discussed together here. During the analysis of market data, customer documentation and user needs, and in constructing P-Diagrams and Quality Function Deployment (QFD) QT-1 tables the top level system performance parameters and key design characteristics such as size, mass, power and data rates are identified.  These parameters must be quantified and metrics must be defined that enable the team, management and customers to track progress toward meeting these important requirements. It is also necessary to judge which design characteristics are rigid constraints and which can be modified in design trade studies if necessary.

6.3.3.1 Define Tracking Metrics (KPPs & TPMs) Developing a consistent process for constructing and updating KPPs, TPMs and any other tracking metrics needed is essential to providing confidence to both the team and their managers and customers that work is progressing satisfactorily or if critical requirements are at risk. It is good practice, as defined by IEEE task 11, to begin the selection of tracking metrics and setting up tracking procedures during the requirements analysis task even though the design work is not yet dealing with physical design. One widely used process is presented here. Only TPMs are discussed but the process can be used for other metrics as well.  The benefits of TPMs include:

• Keeping focus on the most important requirements (often called Cardinal Requirements).
• Providing joint project team, management and customers reliable visibility on progress.
• Showing current and historical performance Vs. specified performance
• Providing early detection and prediction of problems via trends
• Being a key part of a design margin management process
• Monitoring identified Risk areas
• Documenting key technical assumptions and direction

The steps in generating TPMs include:

1. Identify critical parameters to trend (e.g., from a QT-1)
2. Identify specification & target values and the design margin for each parameter
3. Determine the frequency for tracking and reporting the TPMs (e.g. monthly)
4. Determine the current parameter performance values by modeling, analysis, estimation or measurement (or combination of these)
5. Plot the TPMs as a function of time throughout the development process
6. Emphasize trends over individual estimates
7. Analyze trends and take appropriate action  

Some systems have a very large number of TPMs and KPPs. Rather than report every metric in such cases develop a summary metric that reports the number or percent of metrics within control limits and having satisfactory trends. Metrics not within control limits or having unsatisfactory trends are presented and analyzed for appropriate actions. This approach works well if those reviewing the metrics trust the project team’s process.

It is recommended to track and report both margin and contingency with the current best estimate (CBE). Unfortunately there are no standard definitions for these parameters. Example definitions are shown in Figure 6-13.

Figure 6-13 Example definitions for margin and contingency where contingency is defined as the current best estimate plus/minus one sigma.

An example of a TPM tracked monthly for a characteristic for which smaller-is-better, e.g. mass or power, is shown in Figure 6-14 with the plotted points being the CBE plus contingency.

Figure 6-14 An example TPM chart for a characteristic for which smaller-is-better.

Included in the chart is the specification value, shown as a horizontal line, a target value selected by the project team and plotted as a point at the planned end of the development cycle, a margin depletion line, upper and lower control limits as well as the plotted values of CBE plus a contingency. Selecting the target value, slope of the margin depletion line and the control limits is based on the judgment of the project team.

Smaller-is-better characteristics like mass and power consumption tend to increase during the development so initial values should be well under the specification value. Thus a reasonable strategy for selecting the margin depletion line is to link the initial estimate with the final target value. Control limits are selected to provide guidance for when design action is needed. If the characteristic exceeds the upper control limit design modifications may be needed to bring the characteristic back within the control limits. If the characteristic is below the lower control limit then the team should look for ways that the design can be improved by allowing the characteristic to increase toward the margin depletion line value at that time. For example, in Figure 6-14 the first three estimates trend to a point above the upper control limit. The fourth point suggests that the project team has taken a design action to bring the characteristic back close to the margin depletion line.

Many requirements must be broken down into separate components, attributable to each contributing factor; e.g. mass of a system can be broken down into the mass for each subsystem. TPMs for such requirements are a rollup of the contributing components.  A variation on a TPM particularly useful for such requirements is to explicitly include the basis of estimate (BOE) methods of arriving at the CBE value each period. Typical BOE Methods are allocation, estimation, modeling and measurement. Allocation is the process of taking a top-level roll-up and budgeting a specified amount to each contributing component. Estimation, modeling and measurement are methods applied to each contributing component and then combining them to determine the result for the top level requirement. Figure 6-15 is an example of a smaller-is-better characteristic with the CBE shown as a stacked bar with the proportion of the CBE from each method explicitly shown.

Figure 6-15 An example of a TPM chart where the plotted values include the basis of estimate methods to provide more information about the confidence in reported values.

The proportions allocated and estimated drop as modeling and measurements are made, which increases the confidence in the reported CBE value. In the example shown the CBE is the sum of values for each contributing component with different uncertainties and the plotted values, i.e. the top of the stacked bar are the CBE plus the RMS of the uncertainties, i.e. the contingency in this case. For a larger-is-better characteristic the plotted value is the CBE minus the RMS of the uncertainties assuming the definitions in Figure 6-13.

One alternative method of tracking and reporting metrics is using spreadsheet matrices rather than charts. This approach allows the values for each contributing component to be tracked and reported. An example for a smaller-is-better characteristic attributable to three component subsystems is shown in Figure 6-16.

Figure 6-16 An example TPM for a smaller-is-better characteristic attributable to subsystems.

The total allocated value can be either the specification value or the specification value minus a top level margin, i.e. a target value. The BOE methods for each contributing component CBE can be included in additional columns if desired. The disadvantage of a matrix format compared to a time chart is that it isn’t as easy to show trends with time, margin depletion line value and control limits. Matrices are useful for requirements that involve dozens of contributing components; each important enough to be tracked. The detail can be presented in a matrix with color coding to highlight problem areas and either a summary line or summary chart used for presentations to management or customers.

A third alternative for tracking metrics is a tree chart. An example using the data from Figure 6-16 as mass in Kg is shown in Figure 6-17.

Figure 6-17 An example of a TPM in a tree chart format.

A tree chart has similar disadvantages as a matrix compared to a time chart but both trees and matrices have the advantage of showing the values for contributing components. Sometimes it is valuable to use both a time chart and a tree or matrix for very important TPSs or KPPs.

Common sense guidelines to follow in managing tracking metrics include:

• Keep metrics accurate and up to date
• Be flexible with metric categories (add, delete, modify as development progresses)
• Dovetail into customer process
• Plan ahead to achieve maturing Basis Of Estimates (BOE)
• Make performance prediction part of normal design process (flow-up, not run-down)
• Include in program status reports, e.g. monthly.

It should be part of each IPT’s normal work to report or post to a common database the estimates for metrics under their design responsibility. This makes it easy for one of the system engineers to compile up to date and accurate metrics for each reporting period. If this is not standard practice then someone has to run down each person responsible for part of a metric each period to obtain their input; a time consuming and distasteful practice.