Who Leads System Design?

8.5 Design Oversight Responsibility

Although it is not the intent of this book to describe systems engineering management processes it is helpful to briefly describe how roles and responsibilities change during the system development phases. The first change in responsibility takes place when the baseline design is refined through trade studies to be the preferred design, e.g. a best value design, and the design requirements database is complete. The system development work is then at the end of the define requirements phase and ready to enter the design phase. This means the design requirements are complete to the level of responsibility of each of the lowest level IPTs, assuming the work is organized so that the lowest level IPT leaders are able to work in the “craftsman” model, i.e. the leader has the knowledge and experience to make all the design decisions for the work assigned to his/her IPT. At this point the individual IPTs take leadership responsibility from the SEIT or systems engineering and lead in determining how the system is designed and any prototypes built.

During the design phase systems engineering watches over the design to ensure requirements compliance, testability and producibility and monitors MOEs, progress on “ilities” and risk management. In addition, systems engineering is responsibility to manage any specification changes. If designers encounter difficulties in meeting an allocated requirement then systems engineers should take responsibility for determining if and how the requirement in question can be modified without jeopardizing overall requirements compliance. The systems engineering role during the design phase can be summarized as supporting the designers to ensure that a balanced and compliant design is achieved that is testable, producible and maintainable.

The transition from the system definition phase to the design phase is typically gated with a design review, e.g. a System Design Review (SDR). Program managers often wish to move systems engineers off a development project when the actions items from a SDR are complete. Although this is a time to increase the design specialty engineers and decrease the systems engineers on the IPTs removing too many systems engineers can leave the designers without adequate systems engineering support and cause other necessary tasks to be understaffed. It is better to assign the systems engineers to tasks like preparing system and subsystem integration and test plans and completing and maintaining the system design documentation.

When the design phase is compete and any prototypes are fabricated then systems resumes lead responsibility during test even though other specialty engineers may conduct the testing. Typically this change in leadership occurs sometime during integration and subsystem testing of an engineering model or prototype. These leadership responsibility transitions should occur naturally for IPTs with experienced systems and design engineers.

Some Useful Design Concept Diagrams

8.4 Diagrams Useful in Selecting the Preferred Design

As discussed previously much of systems engineering is determining relationships between a system and its environment and among the various subsystems. Ultimately these relationships are defined in detailed drawings but understanding the relationships in order to select the preferred design is aided by examining a system with different levels of abstraction. The modern tools used for electrical and optical design and the design practices of electrical and optical design engineers develop their respective design concepts with diagrams. The diagrams start with block diagrams with a high level of abstraction, perhaps just naming the subsystems, and proceed to greater and greater detail. This process makes it easy for systems engineers and other design engineers to readily understand the electrical and optical design concepts.

Although there are excellent modern design tools for mechanical and thermal design it isn’t as easy to present the mechanical and thermal designs with ever decreasing levels of abstraction so that system engineers and other design engineers can easily understand the mechanical and thermal designs. Experienced mechanical and thermal design engineers develop the desired diagrams and tailor their diagrams to the system being developed so that others can readily understand and assess their designs. A few examples are presented here to illustrate how experienced mechanical and thermal designers examine and communicate their design concepts. Note how easy it is to think of alternative design approaches when design concepts are presented in simple block diagram form with a high degree of abstraction. This enables engineers other than expert mechanical and thermal designers to assess design concepts and suggest design alternatives.

8.4.1 Simple Mechanical and Thermal Block Diagrams - Many times a simple block diagram is useful in describing a mechanical or a combined mechanical/thermal design concept. Figure 8-7 is an example showing a system that consists of five assemblies, a frame for mounting the assemblies and a mounting plate that supports the system with a three point mount.

Figure 8-7 A simple mechanical block diagram of a system illustrates how the assemblies interact with each other and the mounting frame.

It’s easy to see that the design concept is for each assembly to be coupled to the mounting frame and uncoupled from any other assembly. Four of the assemblies are temperature controlled whereas the fifth assembly is attached to the mounting frame but not within the temperature controlled region. By abstracting all of the size, shape, material and other characteristics of the system the basic mechanical and thermal relationships are easily understood and alternative concepts are obvious. The actual models that the mechanical designer uses to conduct trade studies of alternate mounting concepts are of course much more detailed and usually include the detailed characteristics of the various assemblies, frame and mounting plate; however, this detail is not necessary to explain the concepts and results of the trades to the system engineers and other designers.

Let’s suppose that three of the assemblies of the system shown in Figure 8-7 have critical alignment requirements. A perfectly good way to record and communicate the alignment requirements is with allocation trees. However, sometimes it makes the requirements clearer and possibly less prone to misunderstandings if a simple diagram is used in place of a tree. Such a diagram with the alignment requirements for each of the three assemblies and for the attachment points of each on the system mounting frame might look like Figure 8-8.

Figure 8-8 A simple block diagram illustrating the alignment requirements for three of assemblies and their respective interfaces with the mounting frame.

Similar simple block diagrams can be used to illustrate temperatures and heat flow paths. It sometimes makes design concepts easier to understand if mechanical and thermal diagrams are developed together. Examples are shown in Figures 8-9 and 8-10 that illustrate simple block diagrams of structural interfaces and thermal interfaces on similar block diagrams.

Figure 8-9 A block diagram illustrating structural interfaces within a system and between the system and its parent platform.

Figure 8-10 A block diagram illustrating thermal interfaces within a system and between the system and its environment using the same diagram approach as used for the structural interface diagram.

Even though it takes some time and care to develop diagrams such as shown in the examples above the benefits to the team in understanding and refining design concepts to reach the preferred design are well worth the effort. Diagrams like these and related simple diagrams for electrical, optical and other design concepts are invaluable in explaining a design concept to customers and management.

Modeling and Simulation Supports Entire Development Cycle

8.3 Modeling and Simulation

Modeling and simulation tools are used in all phases of system development from definition to end-of-life. Systems engineers are concerned with the models and simulations used in system definition, design selection and optimization, and performance verification. Systems engineers should identify the models and simulations needed for these tasks during the program planning phase so that any development of required models and simulations can be complete by the time they are needed. Examining the customer’s system requirements and the planned trade studies help identify the needed models and simulations. Parameter diagrams are often helpful in identifying the models and simulations needed.

Models constrained by requirements are typically adequate to be used in the system definition phase to define a baseline design concept. Models may be adequate to develop error budgets and allocations but performance simulations are often necessary to select and optimize designs.

System simulations and particularly performance simulations are especially useful in system performance verifications. Therefore it is necessary to include any necessary validation of system simulations in test plans and procedures. End-to-end system simulations are sometimes needed to verify final design compliance with requirements. Other uses include developing the requirements for data analysis tools needed during subsystem and system verification testing, reducing risk and time for developing test software and supporting troubleshooting during test and operational support.

Examples of how models and simulations might be used in system development are shown in Figure 8-4. In this figure the system under development is assumed to be a system that measures parameters by sampling the parameters that are related to a desired phenomenon that cannot be easily or economically measured directly. The measured samples are assumed to be processed first by Data Algorithms, which in this example produce calibrated data. The calibrated data are

Figure 8-4 Examples of ways models and simulations might be used in developing a sensor or measurement system.

then input to Product Algorithms which use the calibrated data to produce estimates of the desired phenomena.

It is assumed that a database of truth data is available. This truth data is used in two ways. It is used to predict the parameters that the system is designed to sample by using a model; called a Parameter Model in Figure 8-4. These predicted parameters are then the input to the System Model and System Simulation. The truth data is also used to assess the validity of the system model and the system simulation by comparing the results predicted by the Product Algorithms with the truth data. This example assumes that the System Model generates calibrated data and the System Simulation generates data that must be processed by the Data Algorithms to provide calibrated data. If truth data is available for the desired phenomenon during system operation then the truth data can be used to assess the performance of the system during operation as suggested by the figure.

It is assumed that Environmental Models are developed that can also generate the parameters to be measured. Information from the System Specification is used to generate the parameters in the desired range and with the desired statistics. If the database of truth measurements is representative of the specified range and statistics of the phenomena to be measured then the Parameter Model can be used to generate inputs for system design analysis and as comparisons for system test data analysis and comparisons. If no database of truth measurements is available then Environmental Models are used in place of the Parameter Model but it is not possible to assess results against truth data.

8.3.1 Performance Modeling and Simulation - System performance models and system performance simulations are used in trade studies to evaluate alternative designs and to iteratively optimize the selected design. Typically the system design objective is to develop the “best value” design solution. A “best value” design can be defined as:

·         Achieves performance above minimum thresholds

·         Has life cycle costs within customer’s or marketing’s defined cost limits

·         Meets requirements allocations (mass, power, etc.)

·         Assessed to be relatively low risk (so that cost targets are likely attainable)

A general approach to achieving a best value system design is to develop multiple design concepts, assess the cost and performance of each and iterate until the best value is achieved. This usually involves progressively lower level trade studies.

 Assessing the cost and performance of system design concepts requires analysis and state-of-the-art tools. Design tools for mechanical, thermal, electrical and optical analysis are well developed, widely available and indispensible for design of modern systems. The same cannot be said for the cost models and top level performance modeling and simulation tools for systems analysis. System performance modeling and simulation tools are too specialized for widespread utility. Thus most systems organizations must develop the modeling and simulation tools needed for defining their systems. Useful cost models are available for some systems for organizations developing systems for government agencies like the Defense Department and NASA. Examples of cost estimating models useful for several types of systems and cost estimating tasks include SEER (http://www.galorath.com/) and PRICE (http://www.pricesystems.com/).

The first two steps in seeking a best value design are shown in Figure 8-5. The cost model is used to identify a number of design parameters that drive the system cost and quantify how the cost, or the relative cost, depends on each design parameter. The system performance modeling and simulation tools are used to quantify the dependence of system performance on each of the same design parameters.

Figure 8-5 Cost models and system performance models and simulations are used to determine the relationship of cost and performance on design parameters.

Having the relationships of cost and performance on design parameters these data can be combined to reveal how the selected design parameters drive the relationship of performance to relative life cycle as shown in Figure 8-6. Assuming the cost and performance relationships are determined for n design parameters then the result is n trades of cost vs. performance as a function of each of the n design parameters. It is usually straightforward to select the value of each design parameter that offers the best value design according to the desired criteria. For example, in Figure 8-6 the best value for the design parameter shown is 8 cm because it’s near the maximum of the linear portion of the parametric curve and it offers the best performance within the constraints on this particular design parameter.

Figure 8-6 The best value design is determined by combining the data from the cost model and performance models and simulations that determine how design parameters drive cost and performance.

Design Trade Matrices and Other Trade Study Methods

8.2.2 Design Trade Matrices - The Pugh Concept Selection process can be repeated until no new concepts are suggested that are better than the existing set. When a final set is agreed upon the next step is to conduct a weighted trade using a design trade matrix (decision matrix in NASA nomenclature). A design trade matrix might look like Figure 8-3 for three candidate concepts.

 Figure 8-3 An example design trade matrix for three concepts and three weighted criteria.

Again if QFD is used the weights for the criteria can be drawn from the QFD analysis. If not then engineering judgment can be used for the weights. In the example shown the weights are selected over a range from 1 to 5. An alternative is to use percentages that add to 100 percent for all criteria. Attribute scores can be 1, 2 or 3 or 1, 3 or 9 or the actual results of using an analytical tool. If different tools are used for different criteria the attribute scores can be normalized to a fixed range for all criteria to maintain the validity of the selected weights.

The step following determining the total scores is to perform a sensitivity check so ensure the results are significant. One method of performing a sensitivity check is to examine evaluation values for criteria with large weights. If a small change in one evaluation changes the total score to favor a different concept that the results aren’t reliable. If the results are not reliable then consider adjusting the weights, adding additional criteria or using a more sensitive method of determining attribute scores.

8.2.3 Pitfalls for Trade Studies – Common mistakes that can lead to ineffective trade study results include:

1.       Poor requirements definition can result in a trade result that may not be good for properly defined requirements.
2.     Valuable alternatives may be missing if alternatives are defined without brainstorming by several experienced people with a diversity of skills and experience.
3.     Allowing biased weightings or selection criteria often results in selecting alternatives that are driven by the biases and not the optimum alternative that would be selected with unbiased trades.
4.       The fatal error is having no winner. This results if the spread of the weighted score is less than the spread of estimates of errors. The sensitivity analysis step in Figure 7.1 is crucial to effective trades.
5.       Inappropriate models used for determining attribute scores. Models not only have to be relevant to the trade being performed they must have credibility in the eyes of the decision makers, they should lead to scores for the different alternatives that are spread more than the estimates of errors in the model results, the algorithms and internal mathematics must be transparent to the users and they must be sufficiently user friendly that the analysis can be conducted with confidence and in a timely manner.
6.       Conducting system and design related trade studies outside the control of systems and design engineering. Development program managers sometimes pull systems and design engineers off programs early to save money and then allow procurement, operations or product assurance personnel to conduct trades without the oversight of the appropriate systems or design engineers. This can lead to a multitude of difficulties and usually expensive difficulties. Suffice it to say that systems engineers and design engineers must retain control of systems and design trades throughout the life cycle.

8.2.4 Other Design Trade and Decision Methodologies - The design trade process defined in Figure 8-1 is a proven methodology but not the only useful tool or methodology available. The NASA Systems Engineering handbook describes several techniques useful for trade studies and more general decision making. These include:

·         Cost benefit analysis

·         Influence diagrams/decision trees – the NASA handbook and Wikipedia have poor descriptions of these tools. A more useful description can be found at http://www.agsm.edu.au/bobm/teaching/SGTM/id.pdf and for a more thorough and mathematical description see http://www.stanford.edu/dept/MSandE/cgi-bin/people/faculty/shachter/pdfs/TeamDA.pdf These tools have available software to facilitate developing the diagrams and converting influence diagrams to decision trees.

·         Multi-criteria decision analysis (MCDA) - useful for cases where subjective opinions are to be taken into account. Start with: http://en.wikipedia.org/wiki/Multi-criteria_decision_analysis See also http://www.epa.gov/cyano_habs_symposium/monograph/Ch35_AppA.pdf

o   Analytic hierarchy process (AHP) – a particular type of MCDA that employs pairwise comparison of alternatives by experts.

·         Utility Analysis (The DoD SE handbook calls this Utility Curve Analysis)

o   Multi-Attribute Utility Theory (MAUT) (A MCDA technique for Utility Analysis)

·         Risk-Informed Decision Analysis- for very complex or risky decisions that need to incorporate risk management into the decision process.

Trade Study Methodology

8.2 Trade Study Methodology

It is a good practice for a systems engineering organization to adopt a standard methodology for conducting trade studies. A standard methodology makes control and management easier and the discipline of following a standard methodology usually results in faster and better trade results and gives management more confidence in the results. Independent of the methodology used trades must be defined and planned. Key top level trades should be identified in the SEMP but not all trades can be listed in the SEMP. Trade Tree diagrams are useful for defining trades if there are only a few alternatives and a few levels under consideration. Trade Tree diagrams are decision trees without the chance values applied to each node. Trade Trees and Decision Trees are useful tools well defined in the NASA Systems Engineering Handbook. If there are many trades to be conducted for a single level then N-squared diagrams are more useful than Trade Trees. The value of the N-squared diagram is that it provides an easy way to determine the best order to conduct trades and conducting the trades in the best order saves time and money. An N-squared diagram for trades is developed following the same process as for defining and ordering tasks using N-squared diagrams as described in Chapter 4.

One standard trade study methodology is present here and it is defined for physical design trades. Experience shows that this methodology is useful for design trades at all levels of a system hierarchy. This methodology combines Design Trade Matrices with Pugh Concept Selection and is best described by the diagram shown in Figure 8-1.

Figure 8-1 An excellent trade study process combines Pugh Concept Selection with a Design Trade Matrix.

Brainstorming is a good way to identify alternatives and many alternatives should be defined. It is important to select people with a diversity of skills and experience for effective brainstorming. Otherwise the alternatives may not have the variety and innovativeness desired. If a large number of alternatives is identified the number of alternatives can be thinned to narrow the trade space. This is accomplished by point design analysis or by using engineering judgment to select a subset of the “best” concepts. The next step is to define selection criteria. These criteria are derived from the requirements. If a process like QFD is used there are a set of requirements that are more important to customers than others. These are often called cardinal requirements or key requirements and they are the ones most likely to be the important criteria that should be used in trade studies.

8.2.1 Pugh Concept Selection - Having defined selection criteria the next step is to refine the concepts. Pugh Concept Selection is the critical step that leads to improved concepts. If large or complex system designs are being traded then first conduct Pugh Concept Selection at a sub system level. Then combine the best results into a top level system design concept.

Pugh Concept Selection is conducted by choosing one alternative concept as the benchmark. Each of the other alternative concepts is then compared to the benchmark. The objective is to use the results of the comparison to suggest new concepts or even new criteria. A matrix diagram is used to compare the alternative concepts to the benchmark as shown in Figure 8-2.

Figure 8-2 A Pugh Concept Selection matrix for three alternative concepts compared to a benchmark concept.

For each criteria each concept is compared to the benchmark concept and a decision is made as to whether the concept is better (+1), worse (-1) or the same (S) as the benchmark. Weighting should not be used at this point in the trades. The score for each concept is obtained by summing the number of pluses and minuses, each counting as a plus one or minus one and the S counting as zero. The next step is to examine the concept that scores the best, assuming that one scores better than the benchmark. For example, in Figure 8-2 concept a scores better than the benchmark and the other two concepts. Note however that for criteria 3 concepts b and c are superior to both the benchmark and concept a. Examine the reasons for concepts b and c being better for criteria 3 and see if changes can be made to either concept a or to the benchmark that will turn concept a’s minus score for criteria 3 to a plus or a same. The idea is to consider new alternatives that combine the best features of the traded concepts to arrive at a better concept. Thus the objective of Pugh Concept Selection is not to select the best of the alternatives but to define new concepts that are better than any of the initial concepts.

Managing to a Baseline System Design

We now turn to the next step for systems engineers after the first system design concept is defined. This next step is the subject of chapter 8.

8 Selecting the Preferred Design

8.0 Review and Introduction

Selecting the preferred design involves several types of systems analysis tasks including: trade studies, risk assessment, cost modeling, performance analysis/modeling and simulation. Before describing some of these processes it is helpful to review several points from chapters 5 and 6. Trade studies, assessment and analysis are involved in each of the three major steps in the systems engineering process. Referring back to the systems engineering process described in Figure 6-4 shows that iteration takes place not just between each of the major steps of requirements analysis, functional analysis and design synthesis but also between each of these steps and the systems analysis processes of trade studies, modeling and simulation included under technical management.  Figure 5-1 illustrates how these systems analysis tasks support the decisions involved in flowing down requirements. Chapter 6 discussed how functional analysis is involved in the flow down of requirements and how systems analysis supports allocating requirements to functions.  Chapter 6 also discussed the value in considering many design alternatives at each stage of design synthesis but particularly during concept design. The preferred process for considering design alternatives is to arrive at a baseline design and then conduct trade studies of alternative designs. It is systems analysis processes that aid in selecting the preferred design from the alternatives studied. This chapter describes methods for conducting trade studies, provides an overview of modeling and simulation and introduces some additional diagrams that are useful in describing design concepts.[J1] 

8.1 Baseline Concept Management

Project leaders must strive for a balance between forcing decisions too early so that promising alternatives are not considered and leaving too many decisions until just before major design reviews so that the design details are unclear. The standard process for achieving this balance is forcing a design baseline as soon as possible and then conducting trade studies of alternatives to the baseline. This facilitates having a controlled process for making decisions to adopt changes to the baseline and it enables all team members to have the same view of the baseline design at any time.

As soon as the top level function and physical diagrams, the system level specification document and the ICD are complete declare this documentation to be the baseline design. It’s ok that many assemblies or even subsystems are immature; just use the team’s best guesses, but do include as much detail as is available. The baseline design documentation should be maintained where it is available to all team members but no changes to the documentation are allowed without a formal decision process being executed.

As the design progresses and trade studies are performed use the baseline as the starting point and the reference for trades and design decisions. If analysis or trade studies suggest the baseline should change and the project leadership, including the lead systems engineer, agrees then the change is made and the lead systems engineer should notify all key engineering participants that a design change to the baseline has been made. Then the baseline documentation should be updated.

Following this simple baseline management process ensures the whole team is working from the same design data base at any time and it facilitates a controlled design decision process.

---

 [J1]