Design Trade Matrices and Other Trade Study Methods

8.2.2 Design Trade Matrices - The Pugh Concept Selection process can be repeated until no new concepts are suggested that are better than the existing set. When a final set is agreed upon the next step is to conduct a weighted trade using a design trade matrix (decision matrix in NASA nomenclature). A design trade matrix might look like Figure 8-3 for three candidate concepts.

 Figure 8-3 An example design trade matrix for three concepts and three weighted criteria.

Again if QFD is used the weights for the criteria can be drawn from the QFD analysis. If not then engineering judgment can be used for the weights. In the example shown the weights are selected over a range from 1 to 5. An alternative is to use percentages that add to 100 percent for all criteria. Attribute scores can be 1, 2 or 3 or 1, 3 or 9 or the actual results of using an analytical tool. If different tools are used for different criteria the attribute scores can be normalized to a fixed range for all criteria to maintain the validity of the selected weights.

The step following determining the total scores is to perform a sensitivity check so ensure the results are significant. One method of performing a sensitivity check is to examine evaluation values for criteria with large weights. If a small change in one evaluation changes the total score to favor a different concept that the results aren’t reliable. If the results are not reliable then consider adjusting the weights, adding additional criteria or using a more sensitive method of determining attribute scores.

8.2.3 Pitfalls for Trade Studies – Common mistakes that can lead to ineffective trade study results include:

1.       Poor requirements definition can result in a trade result that may not be good for properly defined requirements.
2.     Valuable alternatives may be missing if alternatives are defined without brainstorming by several experienced people with a diversity of skills and experience.
3.     Allowing biased weightings or selection criteria often results in selecting alternatives that are driven by the biases and not the optimum alternative that would be selected with unbiased trades.
4.       The fatal error is having no winner. This results if the spread of the weighted score is less than the spread of estimates of errors. The sensitivity analysis step in Figure 7.1 is crucial to effective trades.
5.       Inappropriate models used for determining attribute scores. Models not only have to be relevant to the trade being performed they must have credibility in the eyes of the decision makers, they should lead to scores for the different alternatives that are spread more than the estimates of errors in the model results, the algorithms and internal mathematics must be transparent to the users and they must be sufficiently user friendly that the analysis can be conducted with confidence and in a timely manner.
6.       Conducting system and design related trade studies outside the control of systems and design engineering. Development program managers sometimes pull systems and design engineers off programs early to save money and then allow procurement, operations or product assurance personnel to conduct trades without the oversight of the appropriate systems or design engineers. This can lead to a multitude of difficulties and usually expensive difficulties. Suffice it to say that systems engineers and design engineers must retain control of systems and design trades throughout the life cycle.

8.2.4 Other Design Trade and Decision Methodologies - The design trade process defined in Figure 8-1 is a proven methodology but not the only useful tool or methodology available. The NASA Systems Engineering handbook describes several techniques useful for trade studies and more general decision making. These include:

·         Cost benefit analysis

·         Influence diagrams/decision trees – the NASA handbook and Wikipedia have poor descriptions of these tools. A more useful description can be found at http://www.agsm.edu.au/bobm/teaching/SGTM/id.pdf and for a more thorough and mathematical description see http://www.stanford.edu/dept/MSandE/cgi-bin/people/faculty/shachter/pdfs/TeamDA.pdf These tools have available software to facilitate developing the diagrams and converting influence diagrams to decision trees.

·         Multi-criteria decision analysis (MCDA) - useful for cases where subjective opinions are to be taken into account. Start with: http://en.wikipedia.org/wiki/Multi-criteria_decision_analysis See also http://www.epa.gov/cyano_habs_symposium/monograph/Ch35_AppA.pdf

o   Analytic hierarchy process (AHP) – a particular type of MCDA that employs pairwise comparison of alternatives by experts.

·         Utility Analysis (The DoD SE handbook calls this Utility Curve Analysis)

o   Multi-Attribute Utility Theory (MAUT) (A MCDA technique for Utility Analysis)

·         Risk-Informed Decision Analysis- for very complex or risky decisions that need to incorporate risk management into the decision process.