PAGE 1
PERSONALIZED DECISION ANALYSIS A! AN EXPERT ELICITATION TOOL: AN INSTRUCTIVE EXPERIENCE IN INFORMATION SECURITY POLICY February 25, 1985 Prepared for: Dr. Fred Wood Office of Technology Assessment United States Congress Prepared by: Decision Science Consortium, Inc. 7700 Leesburg Pike, Suite 421 Falls Church, Virginia 22043 (703)790-0510 Under Cont1act No. 433-0315.l hese contractor documents were prepared by an outside contracto a~ inputs to an ongoing OTA assessment. They do not necessarily reflect t.he analytical findings of OTA, the Advisory Panel, or th~ Technology Assessment Board.
PAGE 2
PERSONALIZED DECISION ANALYSIS AS AN EXPERT ELICITATION TOOL: AN INSTRUCTIVE EXPERIENCE IN INFORMATION SECURITY POLICY Rex V. Brown Decision Science ConsortiWII, Inc. 7700 Leesburg Pike, Suite 421 Falls Church, Virginia 22043 ABSTRACT Personalized decision analysis, or PDA, can often help an executive use expert opinion effectively in making up his mind on a policy issue. It largely failed in this case, for reasons which are instructive for future efforts. A group of experts in information security was presented with a PDA model which specified a list of policy options and criteria for choice together with illustrative numerical inputs and directions on how to quantify their own judgments about them. The effort was abandoned in the light of negative responses to this initial step. By hindsight, several lessons were learned. The options and criteria in the model were prematurely specified. The experts' first task should have been to help formulate the structure, before being asked to quantify it. Moreover, to avoid unrealistic expectations, it should have been made clearer that, within the resources available, PDA cannot do more than organize and present existing expert opinion. It cannot produce an analysis defensible on any firmer grounds--by contrast with other, more ambitious, decision analysis exercises. We recommend now working with a few experts on a limited issuts, 11~::11 as computer crime, to develop both structure and numbers for a PDA which captures their judgment to the satisfaction of OTA staff. -1-
PAGE 3
l. PERSONALIZED DECISION ANALYSIS (PDA) AND ITS THREE MODES Personalized decision analysis, or PDA, is a technique for quantifying the judgments that go into a decision, such that the logical implications for action can be inferred. The judgments relate to the options to be evaluated, their probable impact in areas of concern and the relative importance of the impact areas. It has been used increasingly in business and government in recent years, in many different forms, roles and levels of effort (Ulvila & Brown, 1983). There are three main modes of using PDA which involve increasing levels of effort and are often carried out in sequence, as interest and resources dictate. lJ.ill .f.126. The simplest form is where the judgment of the decision maker himself is directly quantified. A first pass is often the first step in a policy evaluation and can be completed in a few hours of interaction between the decision maker (or, if unavailable, a surrogate) and a decision analyst. Expert .f.126. In the second mode, expert judgment is channelled into the elements of a policy evaluation it bears on. In this mode, PDA is simply a catalyst and language of communicatiora. It does not add content to an inquiry; it does not require any more content than the decision maker would have drawn on without decision analysis; and it does not require the decision analyst, or whoever is managing the analytic process, to have mastered the subject area. For OTA purposes, it might be to take a given issue, say, computer crime, and ext~act what one or more specialists in the field know, in a form that allows the OTA decision maker to formulate his own perspective on the issue and communicate it effectively, say, in a report to Congress. This is approximately what was intended in the case we are reporting .. A merging of user and expert PDA is typified by "decision conferences" which are becoming increasingly popular with executives in business and government (Kraemer and King, 1984). They involve a decir.ion maker and his experts con ferring for two to three days at a stretch, under the guidance of a decision analyst using PDA software, and emerging with a decision. Such an exercise -2-
PAGE 4
will typically cost on the order of $10K. Study fQ.6. In the "study framework" mode of PDA, the analyst, rather than the decision maker or expert, takes responsibility for developing and justifying the judgmental inputs. In so doing, he may, of course, use experts as well as study reports and any enquiry of his own. Such an exercise involves appreciably more analyst resources and is typified by a recent PDA study for OTA to evaluate postal service options related to the "ZIP+ 4" ~ssue (Ulvila, 1984). These three modes of PDA are not exclusive, and can be (and often are) used sequentially on the same problem. For exacple, a decision maker's perceptio~ of a problem can first be modeled in a user PDA, then augmented by an exper: PDA, then by a study PDA and finally by a revised user PDA, where he takes into account all the predecessors. The focus of this paper, however, is on the expert PDA. BEST cePY AYAllAilt. -3-
PAGE 5
2. AN OTA EXPERIMENT WITH EXPERT PDA 2.1 m Backcround OTA had the task of producing a report for Congress describing and evaluating promising options within the general field of computer and telecommunications reliability and security. A significant part of the input of this report was to be the judgment ~nd knowledge of some of the most prominent experts in the field, whose cooperation had been secured for the project. This participation included a written review of the literature in the field (Sherizen, 1984) and a one-day workshop at OTA, attended by some thirty experts, and a number of observers. Following an earlier, favorable, expe~ience with PDA in the study mode (an evaluation of postal addressing systems related to ZIP+ 4), OTA decided to see how PDA might enhance the accomplishment of this task, and arranged for Decision Science Consortium, Inc., to lead that exercise. OTA's original intention was to analyze some eight issues areas using PDA and to devote the bulk of the one-day workshop to developing input to the point where the participating experts could generate their own input offsite. This would provide material for OTA's report to Congress on information security policy. 2.2 lJ2A Exercise The first step was to develop a PDA model structure, i.e., a preset format to take judgmental inputs from the experts. This format, and accompanying instructions, were distributed to participants in advance of the workshop (see Appendix A) In preparation for thf! meeting, DSC, in consultation with OTA staff and one expert, developed a structurally simple, generic analytic structure for each issue, using the "multi-attribute utility" variant of PDA, illustrated in Table 1. It involved specifying a set of attributes (impact areas) of general concern for informaLion security, weighting them for importance and scoring each option under consideration on each. A critical element was the defini--4Bl~1 CGPY AV AILAiLL
PAGE 6
'l'ah L, l : Numerlcnl PUA IWAl,UATION OF Ol''l'IOHS UNl>lm ltffOIUIATlOH S~:CIJRITY lSStrnn: 11 T1l1c11m11111nfrnt 11111n ~1rlly fur Civil Inn Nom:ln11strlecl tntvrmat Ion ----. ----------------RIAkA AdJressc,I ---~ion Costs Other Attrlbuth -.. -----r -----,Dur l>emoc f ATTRIDUTF.Sb INnt. Sec. 7-F.r.on. Lose Go Direct Ind. De. Clv. Gov. Dus. Wl-:IGIITSc 10 2 I 2 .2}tl 1011!1 cl 0. no 11utht11r. I ll 1JO I -20 I -IO I I. Encryption I. l>l:S I 0 I +20 I +5 I 0 I I 2. Stronger 0 +25 +10 0 v, I 2. lt#-ll on Encrypl () I +20 I I IO I +20 I 3, llrop I +101 Snt,l l l le It 0 +20 -10 I Microwave I, llPd I l' a I 1 () ... 20 I +IO I -IO I I I, I IWS il ...c: > c.,;. ~-.-,. ,' Notes n. See r,uldellnee on completing form. b. l>lmcnelone o( concern to government. c. Re I at lve lmport:nnce or etnkes under cnch nttrlbutc, or of .$10 swing for v. Pub. l'riv. Serv. Co\', Due. Pub. s 2 2 l 2 c; 0 I -'>0 I -JO I I 0 I 0 I 0 0 I +20 I 0 I I -20 I -IO I 0 0 +20 0 -50 -20 0 () I -120 I () I I -10 I -IO I 0 0 I +20 I 0 I I -4o I -20 I 0 0 I +20 I 0 I I _,,o I -20 I () d. Scor~e nre % of potentlnl deterlor ntlon from preeent. e. Welr,htecl sum of scores. -" --"tnx" Upset Value ._ I s I I 0 I 0 I 0 I I I -30 1-~o z I 0 I I -35 0 I -1 s I 0 I 0 I I I -10 I -10 I 0 I I I -10 I _, o I 0 I I Comments (Continue overleaf) z. NSA upset. I I I I I NE.T t:VAl.lJAT I ON e -2.ld +. 33 -.JO +. ,, 7 -.27 +.09
PAGE 7
tion of scales for each attribute and the interpretation of ~he importance weights defined in such a wey as to permit each option impact to be convetted into equivalent dollars. The definitions, as distributed to participants, are given and illustrated in Table 2. A member of OTA staff, familiar witn the problem area, developed a complete set of judgmental inputs (shown in Table l) in order both to test the face plausibility of the model and to facilitate understanding by the experts. The plan for the work session was to devote half the morning (after an introductory review of preset issues and options) to laying the ground rules for the PDA exercise, and to spend the afternoon developing PDA input material for each of eight issues in information securi:y policy. The participants were theu to be asked to fill in the blank foms on their own and return them to OTA within two weeks. These expert PDAs would then be used by OTA in preparing it's report ~o Congress (possibly incorporating them into OTA's own PDA). However, developments during the workshop persuaded the OTA staff to alter these plans. In particular, after the first morning session of general discussion of issues, they concluded, with the concurrence of DSC staff present, that the bulk of the day should be spent establishing what the key issues and options were, rather than taking OTA's starting position on these for granted, (which the previous PDA-oriented approach had assumed). As a result, only a very brief presentation of the PDA approach was made, focusing largely on an illustrative analysis (the one i1-i Table 2). In the ensuing discussion, several work session participant!, gave their opinion that the options and attributes put forward were not well enough developed to justify much effort in quantifying either option scores or importance weights. No action was taken at that time (or later) to take the PDA treatment any further -6BESl CftPY AVA1L~UL~
PAGE 8
TABLE 2 ATTRIBUTES FOR VALUATION OF CONSEQUENCES WITH PRELIMINARY IMPORTANCE WEIGHTS RISKS AND VULNERABILITIES WEIGHTS* NATIONAL SECURITY (vs. FOREIGN THREAT) DEFENSE SOURCE OF DATA (100) CIVILIAN SOURCE (100) ECONOMIC LOSS RISK (E,G,J VIA COMP, CRIME) GOVERNMENT ($1B) BUS I NESS ( SlB) PUBLIC ($1B) OTHER RISKS PUBLIC PRIVACY (100) GOV'T SERVICES (E,G,J RELIABILITY) (100) $ COSTS OF OPTIONS DIRECT$ COST (OUT-OF-POCKET) GOVE:1NMENT ($1B) BUS I NESS ( $lB) PUBLIC ($1B) INDIRECT COST (E,G,J IMPAIRED SERVICE) ($1B) OTHER IMPACTS 10 2 l .2 .. 2 2 1 .2 .5 l BUREAUCRATIC UPSET (100) ., DEMOCRATIC VALUES (CIVIL LIBERTIESJ OPEN GOVERNMENTJ ETC,) (100) 1 EQUIVALENCE IN BILLIONS OF FEDERAL DOLLARS A YEAR FOR THE VALUE SWING IN PARENTHESES (EITHER ~1B A YEAR OR 100, DEPENDING ON WHETHER SCALE IS MONETARY OR QUALITATIVE), THE 100 PT, SCALES ARE DEFINED AS FOLLOWS: 0 = STATUS QUO FROZEN INDEFINITELY (NOT THE SAME AS THE "DO NOTHING OPTION, WHICH CAN GET WORSE), 100 = MAXIMUM PLAUSIBLE DETERIORATION (DEFINE ARBITRARILY, BUT BE CONS I STENT) +100 = AS MUCH BETTER AS -100 IS WORSE, -7BEST COPY AVAILABLL
PAGE 9
3. CRITIQUE 3.1 Ihl lIRa &1151 k2.Da 2l Expert m The primary motivation of expert PDA is that it can focus the contribution of the experts on the critical issues (i.e., those that drive the decision), hlp the user to interpret and second-guess the experts' arguments (or separable pieces of them), and provide a communication vehicle for a clearly defined rationale behind the user's ultimate conclusions. It may also pinpoint gaps in the experts' knowledge and guide further effort in filling them. There are downside risks, however. The quantified analysis may not capture the expert argument any more accurately than listening to a conventional discussion or written exposition. PDA can be used in addition to (rather than instead of) the conventional exposition, but then it comes at an incremental cost in expert and user time. It takes time to educate the expert and the user of PDA in understanding the quantitative questions and answers of the analysis. (E.g., the scales on which impacts are measured.) The user may misinterpret the output of the analysis, for example, by giving more credence to the output than the firmness of the judgmental inputs justifies. To make sure that the net balance of pros and cons of using decision analysis comes out positive, great care must be exercised in its design and conduct. There are many examples of its successful application in cases substantially identical to the present one. However, this experience did not successfully clear this test. Ye believe the basic conception and modelling approach was sound, i.e., a simple one-tier multi-attribute model with no explicit probabilistic modelling. Appendix C discusses some possible modifications. However, the strategy and tactics of execution were flawed, as discussed below, but could be readily corrected. 3.2 2l.lI Interpretation~ .Ihll. Experience The experience described above prompts a number of reflections on what went wrong, and why, and whet the implications for any further use of expert PDA by OTA might be. -8-
PAGE 10
The appropriateness of the original plan (to channel the participants' exper tise into a pre-structured decision analysis) depended on a critical assump tion that wa1 apparently (and in our opinion properly) re-evaluated in the light of early expert input. Thia was the working assumption that OTA's preparatory analysis wa1 sufficiently far along that the analysis was ready for numerical judgment quantification. Specifically, this meant that the issues, options within them, and impact areas were firmly enough established that attention could largely focus on supplying the best numbers within that strcture. Our interpretation is that decision analysis was prematurely formalized in the expert elicitation process, i.e., we had moved too soon to a trial decision strucure, which attracted unfavorable comments on the grounds that the structure did not reflect a well-informed appreciation of the problem. This generated enough resistance among the experts (and derivatively at OTA) to the use of decision analysis on this problem, that support for proceeding further disappeared. The chance was thereby lost for DSC to act on its best judgment on how decision analysis is most effectively used in a case like this one. Our opinion is, and has been, that in a case like this, a one-day workshop of experts should be devoted entirely to generating basic material from which one or more decision analyses can be developed, and not to divert any of that time from this highly productive task to formal decision analysis. On the one hand, the time of workshop members is most valuable in getting ideas and material out in the open in as rich and unconstrained a form as possible. On the other hand, the premature attempt to converge on a decision in a decision analytic mold may discredit the tool and discourage its use in more appropriate contexts (as appears to have happened here). A generally successful model for expert PDA, on which future OTA efforts might be patterned, is the "decision conference." In the past few years, there have been dozens of successful (and some not so successful) decision conferences on which we base our confidence that it could successfully work here. The way it typically works (see Kraemer & King, 1984) is that a group of five to ten experts and decision makers meet for about three days to resolve a specific issue, using computerized decision analysis techniques. -9or.ST COPY AVAlLA8LL
PAGE 11
Typically, the first day will be spent clarifying the dimension of the problem, to the point where a decision analyst can conatruct the framework of a PDA model (aay, in the form of Table l). The second cay would be devoted to confirming that the analyst' framework acceptably represent the gtoup's con cenau.a (mediated if necessary by the ranking decision maker present); and then developing a coherent set of numerical inputs for that framework. Th rationale behind those inputs would be recorded (in computer memory) and the implication of the inputs displayed. The third day would be devoted to exploring the implicatiom of the model as quantified, including testing the im pact of alternative assessment or aaaumptiona and adjusting for any perceived mismatch between the model and the decision maker' perception. By the end of the third day, the senior decision maker will have come to at least a tentative decision and will take away with him documentation, which includes a quantified PDA and supporting verbal rationale. In situations like the present one, where the value of the expert PDA approach is, or may b, in question, care must be taken in presenting it so that a min ill\lll level of credibility is maintained for the exercise to be allowed to proceed. In particular, it requires a cautious, low-key, proven introductior, of the approach with low risk of early failure. More ambitious attempts can then be based on early modest successes which develop confidence. The opposite was attempted in this case. The original plan of forcing all expert discussion and analysis of all issues in a complex policy area immediately into a preset PDA mold has no successful precedent we were aware of. Given a supportive, pre-sold group of participants, with low cost of failure, the gamble of a ground-breaking experiment could be justified, but such was not the case here. The test of actual (as opposed to perceived) success of the experiment is, presumably, whether the resulting OTA policy evaluation is more firmly or more defensibly based than it would have been using a more conventional expert elicitation approach (using comparable resources, including expert time and effort). For this to be the case, we believe that the expertise should be in corporated at all stages in the PDA, including the definition of options and attributes (which was not done here). -10-
PAGE 12
3.3 Bsommeodttion .t2 QIA DSC' recoamaendation in the present case would be to proceed with expert PDA on a very limited ba1i1; both in order to test its value in this mode to OTA, and, if successful, to make a real contribution to OTA' current task. Basically, we are suggesting that OTA walk before it runs in using expert PDA, and start off with an approach that has worked successfully in comparable past 1ituat~ons. Specifically, we suggest focusing on a single issue (say, computer crime), and, working with a handful of experts in that i11ue, to identify a few op tiona of interest and come up with a quantified representation of their judg ments in decision analytic form. Th analyses, reflecting whatever range of judgments there uy be and the rationale behind them, would then be available to OTA staff in developing their own report. OTA would then have the option of either responding directly to these "expert models in coming to its own position; or of developing its own decision analysis (in the user PDA mode discussed above). In this latter case, OTA would be free to use its own beat judgment in coming up with its ovn quan tified inputs, using the expert inputs or not, as they saw fit. In preparing its report to Congress, OTA would also have the option of referring apecifi cally to the decision analysis or not. The decision analytic exercise could, for example, be described in an appendix which cites expert inputs anonymously. 3.4 caution J.n Implemcntina Experc mA There are tvo difficulties potentially impairing the success of an expert PDA exercise, and, in fact, they contributed to this one being aborted. The first has to do with the very principle of quantifying a tentative judgment that may not be defensible (or at least not stoutly defended). Someone looking at weakly based numbers may infer more precision than is justified, and expect more justification than can be provided. Evon if he understands that the nwnbers are no firmer than the judgments that went into them, he may question the value of putting numbers in such cases. This may be largely a problem of communication. For example, it may be resolved by giving ranges rather than single numbers. -11-
PAGE 13
The second difficulty has to do with the adequacy of the quantitative model, and the amount of effort that it takes to produce a minimally acceptable model. (The ~est is whether it captures at least as much of the expert judg ment available as a more conventional elicitation process.) If the experts being elicited are unfamiliar with the type of decision analysis being used, it takes time to get the elicitation right. In particular, the options must be seen as meaningul and of interest; the attributes must capture what is important to the expert; and the numbers should deserve the interpretation that is being put on them. It probably requires at least three hours of the expert and the analyst working together to extract a meaningful representation of the pros and com of two or three options. An important general caution is that all participants in a PDA exercise be clear on its role and scope, which can vary substantially and, indeed, evolve in a given situation. For example, the intended role in the present case was to provide a vehicle for eliciting expert opinion, and possibly also to help OTA staff organize its own evaluation of promising information security policy option.. Any more ambitious expectations, for example, involving validation and refinement of the expert opinion, could reasonably be realized as a second round of enquiry, but with, of course, additional effort. A potential development would be to then adapt the PDA as a framework for using th~ results of later, more intensive, efforts to refine judgments (at least those which had the most influence on the conclusions and were susceptible to improvement). When those more intensive studies have been undertaken, PDA can then serve as a vehicle for communicating their implica tions for action. PDA has, in fact, often been used in all these modes by decision makers and their staffs in the public and private sectors, including, very occasioru1lly, Congress. The scales and modes of use have been very varied, going from one or two days up to sever,) ,,ears of staff effort. They have also gone from zero decision maker involvement (wher the implications of the PDA were translated into purely qualitative positio~ before being presented to the deci1ion maker), to intensive decision maker involvement (occupying weeks of their time). -12-
PAGE 14
The intended scope of the present exercise was at the lower end of both scales: minimal effort in structuring judgment and a low ~rofile role in communicating to congressiunal decision makers. -13-
PAGE 15
4. AN EXAMPLE OF USER PDA Although its primary function was to set the stage for an expert PDA, ~he sample analysis summarized in Table l can be viewed as an initial atterrr1t at a user PDA, and evaluated a1 such. Its function in that case was to explore the perspective of a user (in this case an OTA staff member) on one issue (telecommunications security). The test would be whether the analysis helped the user to make up his mind, on the basis only of his current k!1owledge. The scope of the exercise as implemented was to capture the user's views on promising policy options, and to evaluate them relative to each other. The structure of the analysis, i.e., the labels and format of Tables 1 and 2, were developed by a decision analyst (the author) in discussion with the user, using about two hours of the user's time and a day of the decision analyst's. The user then developed his own numerical inputs for scores and weights, spending another couple of hours and conferring briefly with the analyst from time to time for clarification on what was needed. This produced numbers appearing at the top of Table 1 and in the main body of the table. The scores are expressed as percentages of arbitrary reference "swing" for each attribute and the weights corresponded to the federal dollar equivalent of that swing. Simple arithmetic produced, as output of the PDA evaluation, scores for each option appearing in the right-most column (the result of multiplying scores by importance weight for each attribute and adding). Each product of score and weight can be interpreted as the value of the attribute impact in question, measured in federal dollars. Taking the second attribut-column in Table 1, the swing used corresponded to an arbitrarily defined large deterioration in the civilian component of national security, valued at $2 billion federal budget dollars, hence the weight of 2. Then, the 30% deterioration in the civilian component of nat.onal security assessed to result from doing nothing to telecommunication security would be judged equivalent to loss of .3 x $2 billion, or $600 million from the federal budget each year. When the other impacts of the "do nothing" option are added together, we have, in the right hand colum.,, total loss evaluated at $2.~l billion per year, for the "do nothing" option. Completing the exercise for the ether rows (options) indicated that t.,e best -14-
PAGE 16
otion (based on these inputs) was to increase R&D funding for encryption, which is valued at an improvement over the current state of affairs of $470 million per year. A key step in this, as in any user PDA, was for the user to review and interpret the output. Any implausibility in the output would be traced to its source in the model's structure or inputs, which can be changed if necessary. In this case, -the user (OTA staffer) felt that the model and its output reasonacly reflected his current thinking, and it was allowed to stand, for the time being. Had the primary object of the exercise been to perform user PDA on a real decision maker (say, the chairman of an appropriate congressional subcommittee), this model would have been subject to greater scrutiny including, probably, several iterations of "build-test-build-test." Although the primary purpose of this user PDA was to prepare for an expert PDA, the procedure adopted was not significantly different from the firsc seep in any user PDA. Typically it would be followed up by several more passes, as the user reviews the output implications of the first pass and refines his analysis in the light of that evaluation. When he is satisfied that his own judgment has been reasonably captured by the analysis, he may then choose to extend the knowledge base reflected in the inputs, for example, by conferring with experts or carrying out additional studies on his own on parts of his analysis which offe= greatest room for improvement and/or have the greatest influence on the conclu~ions. In this case, the subject is only a surrogate for the real decision maker and one pass at modeling his judgment was probably sufficient for the purpose at hand. A promising new phase of a PDA sequence would be, at this point, to involve one or more real decision makers (say, the appropriate House Subcommittee chairman), but this would probably h,ve been impractical in terms of OTA's brief. Alternatively, the decision maker could be involved after completing expert PDA exercise. -15-
PAGE 17
APPENDIX A MATERIALS ON PDA EXERCISE DISTRIBUTED BEFORE WORK SESSION The following materials, prepared by DSC, were distributed to work session participants in advance of the meeting to help them take part in the planned PDA exercise both during and after the meeting. (Two of the items referenced, a list of attributes and a completed worksheet, are reproduced in Section 2 of the paper, rather than here.) Participants also n.ceived a general introduction to the w--rk session and descriptions of candidate options under each of eight infv.mation security issues, corresponding to options supplied in the PDA formats, prepared by OTA staff (see Appendix B). -16-
PAGE 18
OU 00...AFT -DO NOT CX>PY OR QCOTE Off1ce of Technology Assess~e~t loformac:on Security Policy ~ork Session STRUCTURED EVALUATION OF ISSUE OPTIONS USING DECISION ANALYSIS We intend to 12v:.Juate soce of the policy options io key issue areas us::.ng the structured, quantitative approach of personalized decision analysis. This technique represent:; fact and value judgments in a rigorous quantitative way, and explicitly displays their implications for actioc. le has often been found useful as a supplement to more traditional, inf or::ial ways of de cis ionmaking. We will devote the second half of the morn!ng to explaining this analytic procedure and illuscrating it with an example c.,n telecommunication technology options (see sample worksheet). In the afternoon we will begin the process of extracting from the group its inputs for this and other issues (see additional worksheets). During the first part of the morning we ..ill sort out the primary issues and options in each case, taking the attached "issue memo as a s carting point. By the time tte workshop ends, we expect to have proceedeG far enough along on some or all of the issues, that participants will be in a position to complete a worksheet for each issue on their own, based on their own besc judgmeot. We ask you co do so as a "homework assignment," to be returned to us Within the followiog two weeks. Please fill in all cells as best you can, indicating by parenthesis, if need be, your Wilder guesses (i.e., where you would defer to the judgmeot of other colleagues). le ~11 help us make progress ac the workshop if you have given scme chought to the exercise ahead of time, at least co the point of deciding whether each option for a glven issue does have an impact on the areas specified and ~hether it is positive or negative (as has ~en done on the attached worksheet for the "organizational" issue). To help you proceed, the follo,;,.ri~g are attached: o guidelines for cocpleting the work statement; o a list of proposed attributes (i.e., impact areas) and a tentative set of importance weights; o a sample worksheet completed (with preliminary numters) for one issue (telecommunications technology); o a second worksheet (on organizational issues) partially completed, i.e., it has options listed, but gives only the sign of impacts, o a set of worksheets for the remaining issues on the "Prospectus" with soce of the options supplied for you to work with. Attachments as noted -17-
PAGE 19
OT.1. DB.AF! -DO NOT COPY OR Q0OTE GUID:::!.IS!S FOR COMPLETING DECISION ANALYSIS ";.'ORKSHEETS This oote is i~teoded to gi~e gu:cance on each of the steps in evaluating a set of alternacive optioos, using the attached worksheet format. Step 1: List Options (First Column) t This is a fairly obvious step, -'1th the following more subtle caveats. There will almost certainly be more (and better) options than those initially (or ever) listed. That is fioe, provided that the ones evaluated are serious options anc worth comparing. New options -for exat1ple, combinations of old options can be added later or evaluated icformally (for example, by judgmental extrapolation). Each option also may have many variants which are not specified. For our purposes, we only need them defined at the level on which Congress might take a position. Jus~ how congressional directives will be il:lplemented is to be borne io !llind in scoring options, as a~ uncertain consequence (see step 3). Step 2: Define Attributes (First Three Ro~s) By "attributes" we mean the "bottom line" consequences of options that are of real or ulti::.ate concern to Congress (or whomever makes the ded.sion). The attached sheet, headed "Attributes gives.the structure of attributes we are ~Jrreotly using and we suggest sticYing with it, unless they really do no .. capture what is important. (There are three iempty columns on the worksheet to take aoy additional attributes.) The attribute scal~s oo which each option will be scorec (step 4) require careful interpretation. ln each case it is a percentage of a reference "swing" in :he at:ribute (which is therefore treated as a hundred points~ and it is cocpared wi:h the present state of the at:ribute. Tvo types of scales are used monetary and relative. Seven of the attributes are monetized, those dealing ~1th economic loss or cost. In these cases the slo7ing is taken co be Sl billion. Thus, a score of -50 on the attribute "option cost, direct, goverr.ment" means that the option will cost. the Government 50 percent of $1 billion, or $500 million more than at present each year. The other si.x att:ibutes have relative scales, and the s-.ing is taken to be the difference between where we are now and how much worse this attribute might plausibly get. If any option is judged to produce that (bad) result, it would get a score o! -100. A score of +50 means that the option is predicted to result in ao i::l~rovement (on that attribute) half as great in magnitude as the maximum poteo:ial deterioration. Step 3: Weight loportance of Attributes (Fourth Row) The cumbers here compare the stakes for each attribute, in terms of the seriousness of the reference "slo7ing", defined above. What is import.ant here is the relative size of the weights and an appreciation of what the "swings" are. -18-. .. f 1' _.::.11, ,1,, i, ..... 1 \(. r'1<. \ I J1 0. ,.i,.._, ..
PAGE 20
O'IA DR.AFT -DO NOT COPY OR. QOOTE The specific numerical weights, in fact, correspond to the value of the swing in billions of Federal dollars. Thus, a weight of 10 for "national security, defense source" means 1t is worth 10 billion Federal dollars to avoid the maximum deterioration. A ,.,.eight of .2 for "direct cost, l,usiness" means a loss of Sl billion to business is only as serious as $200 c.illion for the Government. A preliminary set of weight.s is supplied on the attached "att::-ibutes" sheet. Step 4: Predict Option Scores (Body of Table) The interpretacion, to be consistent wit.h Step 3, is: what percentage of the specified swing for each attribute (SlB or -100 to 0) would actually result from the option in question. If 1t is judged to be an improvement (rather than a deterioration) over the present, it would have a positive sign. Note that ale hough these interactions start to sound complicated, they are taken into account more or less automatically in the course of the decision analysis process. So, if you prefer, focus attention for the time being s!mply on the magnitude of impact (expressed as -100 worst possible, 0 same, HOO as much better than now as -100 is worse) of the options for each attribute. It would be useful to i:ake notes, in the "comments" section or on a separate sheet, of any important clarification of where the numbers came from. (For example, the negatj_ve score under "Opt.ion Cost -Indirect T3x" for DES in the telecommunicatio~s issue example is due to its slowing down performance.) This makes it easier for anyone revie~ing your number to second guess your results. Section 5: Calculate Evaluation Scores (Right Colucn) This is simply a weighted sum, the result of multiplying scores by weights for each attribute of a given option, and adding che products (chis additive rule is not always exactly appropriate, but is usually a good approximation). Step 6: Interpret Results Relate calculated evaluation scores to real decision problem. Does calculated ranking of options confon: to your intuition? If not, try to reconcile by adjusting one or both. Make "eyeball" adjustment for anything known to be left out of the analysis. Extrapolate evaluation for new options similar to those evaluated. -19-BEST C9PY AVAILA~Li..
PAGE 21
E\TAl,UATlON OF Ol'TlONS llNl>ER lNFOllMATlON SECURITY lSSUEa: /I] The Data Encryption Standard Risks Addressed Option Costs I Ot1he~ Attributes ATTR IOUffSh Nat. Sec. Econ. Loss Gov. Def.I Civ.l Gov.I Bus.I Pub.lPriv.lServ. Gov. Direct Ind. I Dur. I Democ Bus. Pub. "tax" Upsetl Value WEIGIITSc d Options 0. Do Nothing 1. Expand DES ke) 2. New encryptior strategy I N 0 I Notes a. See _guidelines on completing form. b. Dimensions of concern to government. .c. Relative importance of stakes under each attribute, or of SIB swing for costs & losses. BE'1 CIPY AYAILAilL d. Scores are% of potential deterioration from present. e. Weighted sum of scores. .2,./. Comments (Continue overleaf) NET EVAl,UATlONe
PAGE 22
EVALUATION OF Ol'TIONS UNDER INFORMATION SECURITY ISSUE8 : Oli Rellahlltty Risks Addressed Option Costs Other Attributes ATTRillUTESh WEIGIITSc d Options I 0. Do Nothing l. New reliability regs. 2. "Reliability Analysis"for major fed sy~ Iv ,_. I Notes Nat. Sec, I Econ. Loss Uef. I Civ. I Gov. I Bus. I a. See guidelines on completing form. b. Dimensions of concern to government. c. Relative importance of stakes under ench attribute, or of .$18 ewinr. for costs & losses. tit.ST CGPY AYAIWLL ----------~--1,---D i rec t Ind. Bur. Democ I 'Gov. Pub. I Priv. I Serv. 1 Dus.I Pub. "tax" Upset Volue Gov. d. Scores are% of potential deterior ation fro~ present. e. Weighted sum of scores. d-) Coinments (Continue overleaf) NET EVAJ.UATIONe
PAGE 23
EVALUATION OF OPTIONS UNDER INFORMATION SECURITY ISSliEn: 15 Internal Security Measures -----,----------------r------.,-------:----.----I 1 1 1 1 I Option Costs I Ot1h,er Attributes Risks Addressed AT'fRIDUTESb Nat. Sec. Econ. Loss Gov. Def.I Civ.l Gov.I Bus.I Pub:_)Prlv.jServ. Gov. Direct Ind., Bur. I l>emoc Bus. Pub. "tax" Upsetl Volue WEIGIITSc d Options O. Vo nothing. 1. Personnel screening 2. Audit I N N I Software Notes a. See guidelines on completing form. b. Dimensions of concern to government. c. Relative importance of stakes under each attribute, or of JIB swing for wn, .c.iA:t.ts & .. ,!sl!ses. iAlrr .. UAiWU. d. Scores are% of potential deterioration from present. e. Weighted sum of scores. ') ~: (~ -Comments (Continue ov~rleaf) NET EVALUATIONe
PAGE 25
EVALUATION OF OPTIONS UNl)ER lN..-ORMATlON SECURITY ISSUEa: 06 Hot I v11Uon:it Risks Addressed -~--t 01>tion Coots I Olhc~ Attributes I I -------ATTR I BUTESb Nat. Sec. I Econ. l.oss 'Gov. Direct I Ind. Bur. ll)emoc. Def.I Civ.l Gov.I Bus.l_fub,IPriv.lScrv. Gov. Rua.I Pub.l"tax"IUpsetl Value WEIGIITSc d Options 0. Do nothing l. More regulations 2. Fund Ing for 1 SPn1ri ty as I of system 1 cost 3. Secur ltyawarencss campaigns 4.Incrcase R&D funding Notes a. See guidelines on completing form. b, Dimenstons of concern to government. ~. Relntive importance of etnkee under ench nttribute, or of .$ID swing for costs & losses 'SEST CIPY AVAJLMll. I I d. Scores are% o( potential deterioration from present. e. Weighted sum of scores. i -:-:-:. ,_) Comments (Continue overleaf) NET EVALUATION'
PAGE 26
EVALUATION o~ OPTIONS UNDER INFORMATION SECURITY ISSUEa: (17 Computer Cr Jme Risks Addressed Option Costs Other Attributes ATTRIBUTES 0 Nat. Sec. Econ. Loss Gov. Direct I Ind. t-----,,-----r. --Bur. I Democ~ Def.I Civ.l Gov.I Bus.i Pub,IPriv.lServ. Gov. Bus.I Pub.l"tax"l!Jpeetl Value WElGIITSc d Options O. l)o Nothing 1. Increase crime penalties I l'--l l' I 2. Technological protection Notes a. See guidelines on completing form. b. Dlmensiops of concern to government. c. Relative'importance of stakes under ench attribute, or of sin swing for 1St~Ftt111 bilmu:i d, Scores are% of potential deterioration from present. e. Weighted sum of scores. .2 ( -.. Comments (Continue overleaf) NET EVALUATIONe
PAGE 27
EVALUATION OF OPTIONS UNDER INl:ORl1ATION SECURITY ISSUE8 : f/8 Divestitur-e Risks A w:, F' I I --Nat. Sec. I Econ. Loss 'Gov. Def. I Civ.l Gov. I Bua.I Pub. I Priv. lServ. Direct Ind. ~Democ Gov. Bus. Pub. "tax" Upset I Value __._ __ _____i _._ ____ _. ____ _._ ____ ____ .. Notes I Comments (Continue overleaf) a. See guidelines on completing form. b. Dimensions of concern to government c. Relative importance of stakes under each attribute, or of .$18 swing for costs & losses. d. Scores are% of potential deterior ntion from present. e. Weighted sum of scores. '7 c:.:.~-.) NET EVALUATIONe
PAGE 28
APPENDIX B Information Security Policy: A Prospectus for the OTA Work Session The purposes of OTA's works~op are to: 1) ~xplore risks and issues related to the security and reliability of Federal computer and telecommunications syst~s, and identify the most critical of these problem areas; and 2) weigh policy options for dealing with these critical issues, perhaps develop oew and creative options, and identify areas of agreement and disagreement on policy recommenciations. Some special cltaracteristics of this workshop: 1) we are addressing issues at the level of policy, primarily policy actionable by Congress, rather than issues at the technical level. However, appropriate technical issues will be raised where they are releva~t to policy. 2) we are interested in risks anc: issues related to .ion-military and nonnational security uses of Federal computers and telecommunications, as well as the national security-related uses which traditionally receive attention. 3) The mandate for OIA's study, as well as for this workshop, is exceptionally broad, covering the use of computers and telecommunications throughout the Federal Government. Hence, we will try wherever possible to: focus or. the most critical problems; realistically limit our level of detail; and address generic kinds of risks and proble:i.s rather than problems idiosyncratic to a particular system. However, specific examples will help wherever possible co make issues more concrete. A "Straw Man" List of Issues and Options The following prelimina:-y ar.alysis of issues ar:d op:ions is intenced to serve as a starting point for the group. 1) ISSUE: Civilian Ielecoro:aam1 cations Security An issue which may gee relatively little attention in the current climate is telecommunications security and reliability, versus computer security and reliability, speci:ically for civilian, nonclassified information. The vulnerability of telecoomunicacions signals to interception and the vulnerability of such systems to large-scale necwork failure are issues many major system operators do not coi::.sider. The defense and intelligence com..ounities have studied this issue closely for some time, with unclear payoffs for those outside of those communities. OPTIONS: 0) Do nothing 1) Mandate the use of encryption for all or a significant portion of civilian nonclassified information tra~smission. la) Use the Data Encryption Standard (DES) lb) Use a stronger forn of encryptior. (e.g., double the size of the DES key) -26-
PAGE 29
2) Boost ~&D funding to aim :o: a simple, technological ~eans to protect teleco=t:runications s::.gnals froc eavesdropping. (For ex.ample, double the ~BS effort in this area). 3) Eliminate vulnera~le :ticro-~ave and satellite paths fo, transt:J.is sion of sen.s i ti ve Fece .-:al infor=iation. ( Replace largely with fiber optics a~d copper lane-lines). 4) Construct a dedicated, federally-operated and protected network of land-lines and other channels for telecommunicatioo of sensitive federal i:i.fon:i.ation. 2) ISSUE: Organization The new executive branch arrangement for information security, with the National Security Agency play::.ng a prica:ry role as dictated by the President's National Security Decision Di:ective 1,s, presents a variety of potential problems. Among them are the conflict bet~een necessary openness and NSA's traditional secrecy; and be.tween military and civilian needs for secure and reliable systems. OPTIONS: 0) Do nothing. Oa) Cancel NSDD 145 and revert to previous status quo Ob) Continue to implement NSDD 145 1) Modify NSDD 145 through Congressional action to ir,crease the role of civilian needs 2) Create a separate organization for civilian-side issues. 2a) Expand the NBS function in this area 2b) Create a ne~ agency (which might also include privacy and other non-security information policy concerns) 3) Break off the co:puter security center and make it part of some kind of independent agency. 3a) The organization would have essentially the same scope a.: DOD's computer center currently has, for both military and civilian agencies. 3b) The organization woulc have a broader scope, including privacy and ot~er non-security information policy concerns. 3) ISSUE: The Data Encryption Standard There is still controversy about to what ex:ent the Data Encryption Standard meets the needs of a variety of users, inciuding those in the government. OPTIONS: 0) Do nothing 1) Expand the size of the DES key. 2) Adopt a ne~ encr;ption st~ategy as a standard, such as a public key crypto sys:em. 4) ISSUE: Reliability The current concern about sec~rity, which is at leas: in part motivated by "hacker" horror stories, may ':le masking other critical issues, particularly issues of system reliability --the ability of systems to operate on a day-today basis and deliver essential services. OPTIONS: 0) Do nothing~ 1) Promulgate new regulations regarding reliability. 2) Mandate a "reliability analysis" for each major federal system. -27-BEST COPY AVAILAiLL
PAGE 30
5) Issue: Internal Security Measures Anocher issue which may be somewhat neglecced is measures to protect systems from chose who are auchorized to use them. Much discussion seems to focus on protection from outside penetration, ~hereas current experience see=s co indicate that ::iacy (and ~any of the mosc serious) problems relate co protection from incernal fraud or abuse. OPTIONS: 0) Do nothing. 1) Mandate tougher personnel screening procedures for those ~ho work with information technology systems. 2) Mandate enhanced audit software to detect internal fraud. 6) ISSUE: Motivations A persistent problem ic the way of successful implementation of information security practices seems to be the motivations of many system managers and users. Security cos ts time and money which mea.lS a sacrifice in capabilities; hence most people who work With the systems on a day-co-day basis would rather spend money to buy performance (e.g., another disk drive) than to cake the system more secure. OPTIONS: 0) Do nothing. 1) Issue more regulations which attempt to force system manage rs and users to pay attectioc to security. 2) Allocate a certain percentage of system costs for security. 3) Institute large-scale security-awareness campaigns. 4) Boost funding for R&D to aim for a technological fix which could in large part relieve many system managers and users from worrying about security. 7) Issue: Computer Crime Many in Congress and in the public are worried about the vulnerabili cy of Federal information systems to fraud, abuse, or outside pen~tration. OPTIONS: 0) Do nothing. 1) Increase penalties for crimes involving Federal computers. 2) Take technological measures to protect federal systems (reduce dial-up access, improve and l.landate audit software, improve password tllc:.nagement). 8)ISSUE: Divestiture The AT&T divestiture has greatly complicated the situation for Federal agencies and others seeking to create secure telecommunications systems. OPTIONS: O) Do nothing. 1) Partially re-aggregate AT&T. 2) Continue to try to adopt federal strategy to compensate. (How? and How would this be different in the military and civilian sectors?) -28-
PAGE 31
APPENDIX C -SOME TECHNICAL SUGGESTIONS In a case, like here, where a given expert has at most a few hours to devote to the decision analytic excraction of his judgment, the questions asked of him must be few enough and easy enough to understand for him to answer them adequately. The format of Table 1 is probably as ambitious as one should try to get in this regard, and even this may be on the demanding side. Two promising simplifications suggest themselves. Instead of numerical impact scores and importance weights, coarser symbols can be used, such as one or more stars to indicate orders of magnitude, as is done partially in Table 3. It usua!.ly makes sense in any case, to do something like this on a first pass through the expert elicitation, but the suggestion here is that one go no further. This goes some way to avoiding the objection that numbers give an illusory impression of precision. Another promising simplification, suggested by this experience, is not to attempt to disentangle prediction of the impact of an option from the importance of that impact. This could be done by having the expert directly assess the dollar value of a given option impact. Under the present scheree, this can be inferred indirectly. In Table 1, for example, looking at the "do nothing" option on telecommunications security and the attribute "civilian component of national security risk," we see the option is judged to cause a deterioration of 30% of a potential swing valued at $2 million in federal dollars, i.e., $600 million. This could be assessed directly without reference to the "swing." This procedure avoids the expert having to understand and use the difficult concept of a "reference swing" in each impact area (which is then used in a scale for predicting impact and as a unit of comparison for assessing importance weights). On the other hand, we lose a-significant capability, which is to distinguish between value judgments and predictive judgments (which might come from different people). For example, OTA might want to take the lead in assigning importance weights between impact areas and rely primarily on the experts for predicting how each option will score on each impact area. Another important issue of analytic strategy is the sequencing of informal and -298SI CIJPY AVAILA8LL
PAGE 32
Table 3: Qualitative PDA EVALUATION OF OPTIONS UNlllm INFOltMATlON SECURITY lSSUEa: f/2 Org.111 I :;,:n t I onn l Risks A
PAGE 33
formal sessions with experts. It is our view, based on substantial experience in comparable situations, that a substantial fraction of the total exercise, at least half the total, should be devoted to analyzing the problem informally (before any attempt at quantified str\:cture), and to discussing the implications of such a structure after it has been complete. -31-
PAGE 34
RF.FERENC ES Kraemer & King, 1984 Sherizen 1984 Ulvila, J.W., & Brown, R.V. Deciaion analysis comes of age. Harvard Business Review, September-October 1982, 130-141. (Reprinted in Dickson, D.C. (Ed.), Using Logical Techniques for Haking Better Decisions. New York: John Wiley and Sona, 1983; and in Advanced Hanagement Report, 1982-1983, 4(4,5&6). Ulvila, J.W. Decision analysis of postal automation (ZIP+ 4) technology. Falls Church, VA: Decision Science Consortium, Inc., March 1984. -32-
PAGE 35
A BRIEF REVIEW OF EXECUTIVE AGENCY USES OF PERSONALIZED DECISION ANALYSIS AND SUPPORT March 14, 1985 Prepared for: Dr. Fred Wood Office of Technology Assessment United States Congress Prepared by: Rex V. Brown Decision Science Consortium, Inc. 7700 Leesburg Pike, Suite 421 Falls Church, Virginia 22043 (703)790-0510 Under Contract No. 433-0315.1 ----,, c.. .)
PAGE 36
TABLE OF CONTENTS Page Page OVERVIEW .............................................................. .. : 1-35 1. 0 INTRODUCTION ...................................................... ; . 3-37 1.1 Scope of Paper. . . . . . . . . . . . . . 3 1.2 Personalized Decision Analysis ..................................... 3 1.3 Comparison with Other Techniques ................................... 6-40 2. 0 IMPACT ON EXECUTIVE AGENCIES. . . . . . . . . . . g-43 2.1 2.2 2.3 2.4 Historicrl Perspective ............................................ Breadth of Executive Branch Use ................................... The Role of Computerized Decision Support ......................... Distinctive Features of Executive Branch Use ...................... 9 9 10-44 11-45 3. 0 MA.JOR USES W'ITH EXAM.PUS ................................................ 14-48 3.1 Defense Oriented Applications ...................................... 14 3.1.1 Resource allocation ......................................... 14 3 .1. 2 Procurement and systems acquisition ......................... 15 3.1.3 Contingent decision aids .................................... 15 3.1.4 Information strategy ........................................ 16-50 3.2 Civilian Oriented Applications ......... .......................... 17-51 3.2.1 Health and safety risk regulation ........................... 17 3 2 2 Environmental management .................................... l 7 3.:.3 National policy ............................................. 18-52 3. 2. 4 Negotiations ................................................ 19-53 3. 3 Agency Communicati 'ln with Congress ................................. 19 3. 3 .1 Program evaluation........ . . . . . . . ..... 19 3.3.2 Support for legislative proposals .......................... -20-54 4.0 EVALUATION AND FINDINGS ............................................... ,.21-55 4.1 The Case for Executive Branch Use of PDA .......................... ,21 4. 4. l Potential Advantages ........................................ 21 4. l. 2 Cautions .................................................... 2Z56 4. 2 Action Implications ................................................ 22 4.2.1 Desirable developments ...................................... 22 4.2.2 Poss!~le congressional initiatives .......................... 2,3-57 REFER.ENCES ................ 2.5-59 I I U ~-) BEST GOPY AVAILA6Lt.
PAGE 37
OVERVIEY Decision analysis--or personalized decision analysis (PDA), to distinguish it from other decision aiJing approaches--is a way of quantifying the judgments that go into a decision such that the action implications can be inferred. Its purpose is to help executives make sounder decisions which are more readily reviewed by others. It is being used increasingly in the executive branch, especially to make and support decisions with clear cut options which are controversial and have hig~, stakes. The applications have been very varied in level of effort (from days to years), application area (most government agencies), mode of use (to structure an individuals thinking or to organize a major study) or problem type (resource allocation vs. contingency planning vs. policy formulation). Two of the most common and successful uses have been in making major investments, for example, weapons acquisition, and in allocating budgets. A promising, emerging area of application is computerized contingent decision aids, currently developed largely for military tactics, but applicable to civilian emergency management. The extent of applicat5~n has, however, been uneven and generally unsystematic. Defense originally took the lead, but other agencies are catching up. For example, the Department of Energy is undertaking one of the more _ambitious plans to use PDA to support the decision making process for nuclear waste siting. PDA has the potential of becoming the discipline which pervades the everyday thinking of decision mdkers throughout government, as well as becoming the dominant principle underlying studies which they use. What is required to release this potential is expanded efforts to educate users, develop technology, train specialists and promote appropriate application. Because of its interdisciplinary nature, drawing on both social and quantitative sciences, PDA has no natural institutional home or sponsor which would assure that this takes place. Congress may wish to sponsor a fact finding study to establish more definitively what, if anything, is needed to stimulate development and adoption of PDA (or a broader class of decision aiding technique, including artificial in--1. J_ ) -; I 8[$1 cepy r-V/ILMll.l
PAGE 38
telligence and operation research), with a view to designating an agency to tak~ a more active role in furthering PDA. More directly, Congress might ask agencies, perhaps selectively, to justify major budget requests, for example, in support of the MX p~ogram, in a d~c1-sion analysis format. BlS1 COP~ A~A\U\iL\.
PAGE 39
1.0 INTRODUCTION 1.1 Scope of Paper This paper reviews current applications, within the Executive Branch, of a recently developed tool to support the decision making process--decision analysis, or more precisely, personalized decision analysis (PDA). PDA is an information technology which quantifies judgment on facts and values relevant to a decision and computes its implications for action .. Our main purpose is to provide a congressional oversight perspective on whether the executive branch is making the most effective use of this new technology, and what kind of congressional initiatives might enhance it. The primary source of material in this paper is the experience of the author and his colleagues at Decision Science Consortium, Inc., of applying and guiding the application of PDA for executive agencies over the past t~elve years. That experience covers more than half the major government departments and several hundred individual case studies presenting all the major variants and motivations for decision analysis. This experience is augmented by thorough familiarity with the technical literature in decision analysis and close on-going contact with most other leading practitioners and users in the field. Our treatment of the topic in this paper will be to give a brief overview of the executive branch use of PDA, to discuss particularly promising areas of application with illuminating case studies, and to evaluate what this experience means for congressional oversight purposes. Although there h~ve been several reviews of the application of certain quantitative methods in the federal government, notably by GAO (1982), to the best of our knowledge, none of them have specifically addressed PDA, even as a subset of the larger grouping of quantitative techniques to aid decision making. 1.2 Personalized Decision Analysis Although the s:mpler term ndecision analysis" is commonly used to describe the analytic approach covered in this paper, it is potentially confusing because -3--< ;--., BlST COPY AVAiL~~I.L
PAGE 40
there are other, perfectly legitimate, ways of analyzing decision, for example, the Kepner-Tregoe problem-solving approach and classical econometric models. By adding "personalized," we emphasize the distinctive feature of PDA, which is involving the quantification of personal judgment. The PDA technique is a discipline for systematic evaluation of alternative actions as a basis for choice among them. It entails setting up models of the problems to be analyzed, selecting inputs to the models that quantify the judgment (typically of those responsible for the decisions), and deriving the models' outputs from these inputs. Decision analysis models are highly flexible. They can be as simple as a one-line formula expressing overall value as a weighted sum of a few attributes, which can be done in an afternoon, or sufficiently complex to demand extensive use of a computer, and several years of effort. However, a fundamental body of concepts underlies all applications, primarily a cype of probability and utility theory recently developed at Harvard and other major universities (see Raiffa, 1968). Decision analysis models often involve decision diagrams or trees. Inputs to such models may include numerical probabilities which quantify judgments about uncertain future events, and numerical assessments that express the decision maker's attitudes, or the organization's policies, in relation to value tradeoffs and risk. Among the models' outputs may be a display of the probabilities of each possible outcome of every action alternative, or a specification of the single course of action to be preferred under the assumptions of the model--for example, the action with the highest average or "expected" benefit. The representative tools of PDA include decision trees, multi-attribute utility, decision conferences, Bayesian statistics and subjective probability. Preposterior analysis. (See Figure 1) For presentations of techniques and applications see Brown, et al. (1974); Barclay, et al. (1977); Howard & Matheson (1983). Bt:Sl COPY AVAILAW
PAGE 41
t;.i; !r1'' l-~.-I.'" O -u ..: :,, .. ... 4 ,,: > ,r.-;: ,,rI VI I FIGURE 1 TOOLS OF DECISION ANAL VSIS SINGLE ACTOR PROBABILITY ASSESSMENT DIRECT INDIRECT PSYCIIOLOGY STATISTICS HICITATION BAVE'ilAN UPDATING RESPONSE MODES DECOMPOSITION SCORING RULES CONDITIONING DISCRETE VS. MARKOV CONTINUOUS UTILITY EVALUATION RISK PREFERENCE MAUT TIME DISCOUNTING ECONOMICS --;::;,(1 --__., I MODEL ANALYSIS OPTIMIZATION SENSITIVITY SOFTWARE USER DISPLAY PREPROGRJ\MMING INFORMATION VALUATION RECONCILIATION MULTIPLE ACTORS NEGOTIATION MODELS (PARETO) ORGANIZATION TtlEORY GROUP ELICITATION MULTI-CONSTITUENCY ANALYSIS
PAGE 42
The main motivations for using PDA include: improving the quality of decisions; making the underlying rationale transparent; making it easier to second guess decisions; organizing information, expertise and research more efficiently; making decisions more defensibl:!; updating conclusions given new information; and highlighting gaps in information. 1.3 Comparison with Other Techniques Without attempting a water-tight definition, there is enough distinctive about what passes as decision analysis, or more specifically, personalized decision analysis, that it is worth trying to approximately to limit its scope for the purposes of this paper. It is to be contrasted with conventional decision modeling and simulation, characterizing much of what is known as operations research and managemen~ science (see Katzper, in preparation), which do not purport to relJ on judgment of the values of key relationships and parameters. The distinctive characteristic of this information technology is that it involves the quantification of judgment, whose implications are computed to help a decision maker make up his mind. In this respect, it has much in common with what has become known as nartificial intelligence," whose more recent development it in many ways parallels. It is not useful to push definitional distinctions too far. For present purposes, it is sufficient to characterize PDA in terms of the activities of a generally recognizable community of its practitioners. It comprises an intellectual tradition stemming from statistics and psychology, typified by the collaboration of Luce and Raiffa (1957), and an action principle of maximizing subjective, expected utility. The key elements of this principle are subjective probability (which measures personal uncertainty) and utility (which measures personal value judgments). The main academic tri~uta:ies of this movement are (or were) located at the universities of Harvard (Raiffa and Schlaifer), Michigan (Edwards and Tversky), Stanford (Howard), and London (Lindley). The applied wing of the -64-0
PAGE 43
field is represented largely by consulting groups led by graduates of these schools clustered in the San Francisco and Washington, D.C. areas. The network of theoreticians and practitioners is linked by an annual conference (now held at the University of Southern California), a bi-annual international conference (on subjective probability, utility and decision making), and two newsletters. One of the characte.ristics distinguishing decision analysis from other quantitative tools available to managers--such as linear programming, Bayesian updating, mathematical programming--is that these other techniques entail much more narrowly specific classes of model. Such operational research models are useful in, for example, selecting warehouse sites, in balancing assembly-line models, or forecasting energy demand. But none of these models will fit the great majority of large and small decisions faced by managers who have to plan and implement strategies. Decision analysis, on the contrary, can be applied to any problem meriting more than momentary consideration. Figure 2 shows how personalized decision.analysis relates to some of the other main branches of decision science, or the use of formal aids to aid decision making. -74I
PAGE 44
CT.I rn V) -~ ~) C2 --,:, -c:: > c::: > r= rrI (X) I FIGURE 2 PERSONALIZED DECISION ANALYSIS RELATED TO OTHER TECHNIQUES PERSONALIZED DECISION ANALYSIS GENERAL I PERSONALI ST DECISION TREES SUBJECT PROB/\B M.JLTI ATTRIBUTE UTILITY SPECIAL INFO VALUE BAYES UPDATE DECQ"v1P I r : QUrlTATIVE GENERAL DETERMIN OPTIMIZN CU\SS STATS SIMJLN MJLTI CONSTITUEMCY --PARETO I I 00N PERSONALIST r MATI-t PRcx:;RM I LINEAR NEOORK DYNAMIC / ,, -r-/ I ALL DECISION MAKlr TECHNIQUES r FOr r I ND IV I ll.lAL c:mrn SPECIAL l STOCHAST PROCESS QUEUING MARKOV I 1 INFORW\L ( INTUITIVE) QUALi TA Tl VE I GROUP _J SYNECTICS DELPHI l DATAl t-'GT GES
PAGE 45
2.0 IMPACT ON EXECUTIVE AGENCIES 2.1. Historical Perspective The intellectual basis for PDA was largely established in the 1950's and significant practical applications began in the early 60's. The first major field of application was business. Probably a third of the five hundred largest U.S. businesses now make some use of decision analysis, many of them at main board level. Decision analysis has also now beco~e an important tool of the Executive Branch of the U.S. government in almost all major agencies. (Our own organization has first hand knowledge of several hundred applications covering most of the cabinet-level departments.) The pace of PDA's application at upper levels of government, has perceptibly quickened during the last decade. 2.2 Breadth of Executive Branch Use In principle, PDA can be used to conprehensively evaluate any choices whatsoever, and the range of actual applications of this type in the Executive Branch of government is, in fact, very wide, as the following examples suggest. Decision analysis is routine in the defense world for comparing alternative weapon systems and for resource allocation. Pre-programmed decision aids are being developed to accelerate and rationalize responses to tactical emergencies. The White House has employed it in determining whether to embargo sales of a high technology product to the Soviet bloc. The National Security Council has evaluated alternative Middle East strategies, taking account of issues such as how much sacrifice of Allied goodwill is worth how much oil. In the diplomatic area, U.S. negotiators have used the technique to evaluate possible treaty agreements. In the energy field, alternative methods for safeguarding the security of nuclear materials and by-products have been appraised using similar methods. Decision analysis does not necessarily have to tackle the complete decision problem being faced (though that mode of use is the main focus of this paper). It is commonly used as a device to display the implication of just some of the -9-
PAGE 46
consideration.. In the technology embargo problem referred to above, decision modeler~ were specifically asked not to consider the impact of any embargo on the politics of detente. Government officials would be able to consider that aspect later, after seeing the other probable consequences of an embargo. Again, the model may focua exclu.ively on one of the uncer:ainty components of a decision problem. For example, in a study of alternative disarmament 1trategie1, attention wa1 restricted to predicting how long it would ~ake NATO to mobilize in the event of a Yar1aw Pact attack. In this case, in fact, the complete deci1ion analy1i1 procedure was employed, but as a descriptive not prescriptive device, It was assumed that NATO would make the mobilization deciaon aa a rational act--using decision analysis. The assumption was then relaxed to accomod&te the way in which bureaucratic processes really work. Indeed, helping de,ision makers make up their minds in the most logical fashion--the traditional justification for decision analysis--may not even be its primary function. In many cases, its main purpose is simply to lubricate communication in the corridors of power--po11ibly by making one decision more acceptable to, or ~efensible against, ochGrs. For example, the analyses used by procurement officers to evaluate competing weapon systems designs are often made available to contract bidders as a basis for appeal and subsequent improvement. In such cases, the proeurement choices themselves may not be markedly affected by the use of decision analysis. But decisions are made with much less effort chan hitherto, and have a much less stormy passage at the hands of unsuccessful bidders. The technique may be used, with only notional quantification, as a means of focusing the deliberations of, say, a key committee on a time-urgent decision. Decision analysts are appearing increasingly as expert wit~e1ses, for example on behalf of the Federal Trade Commission to establish that improper decisions were made by ensinesses. Agencies are using PDA to work with congressional staffs on supporting proposed new legislation. 2.3 The Role of Computerized Decision support The implementation of POA ha1 been accelerated and in some ways redirected as a result of the recent developments in complementary software capability. For example, the development of user friendly spreadsheet programs, such as Lotus 1-2-3, permits decision analysis in the user override" mode, where the user -10l/
PAGE 47
can second guess inputs, make immediate new assessments and see instantly the implication for the decision. Again, telecommunicati~n technology permits distributed inputs from experts and/or decision makers in teleconferencing and computerized voting systems. Potentially, the most important development may be computer support for decision conferences, r~oneered by Decisions and Designs, Inc. In these, the judgments of a group of decision makers ~nd experts on a given decision are progressively and dynamically modeled in a PDA format, to 'the point of making a decision with the help of an impressive array of interactive software and computerized display devices. Within the past two years, at least one agency (Commerce) has developed its own in-house computer facility and expertise to condict decision conferences. A review of this branch o~ FDA technology appears in Kraemer & King (1983). A number of propriety software packages (and some more generally available, such as through the Defense Advanced Desee.rch Projects Agency, Cybernetics Technology Office) have been developed to permit rapid turnaround interactive applications of decision analysis, for example, hierarchical multi-attribute models, resource allocation models, etc. Agencies are increasingly maintaining their own versions of these package~ from which special purpose prograrus can readily be constructed. The rapid development and increasingly wide .. spread adoption of micro-computers will certainly stimulate the direct use of PDA by decision makers (whereas their main use of PDA up till now has been through staff and consultants). 2.4 Pistinctivu~atures of Executive Bran~h Use Although no systematic survey of PDA use in government has been undertaken, it is clear that the preponderance of application has shifted over the past dtcade from business to government, especially the federal executive. The apparent reasons for the shift reflect differences between business and government. Whereas business has a largely monetary "bottom-line" criterion against which decision options are to be evaluated, in government the criteria are many, varied, and typically intangible, such as the quality of life and the -11'-/ ~) BE~1 COP~ A\AILAJl.1.
PAGE 48
environment. The early variants of decision analysis were ill-suited to handle such problems, being largely limited to univariate monetary scales. With the development of judgment based multi-attribute utility techniquLs in the early 70's, all this changed. Virtually any decision has become accessible to PDA techniques, often with very little expenditure of effort (using simple additive models). Moreover, in many executive branch cases, PDA is the only decision modeling approach available (whereas many business problems are amenable also to more conventional operations research and micro-economic approaches) A second distinction has to do with the political process. Business decision makers are held accountable for their rasults and are mot:i.vated to make those results as good as possible. To the extent that PDA serves to improve results (as it should, if it makes decisions consistent with available judgment), it should motivate businessmen to use PDA. It is generally held, on the other hand, that government d~cision makers, to the extent that they are held accountable at all, ~re held mora accountable for the process whereby decision are made. For example, courts and other ajudicatory proceedings need to be persuaded that regulatory decisions were not made in "an arbitrary and capricious manner." Congressional oversight committees need to be persuaded that agency decision are made in a responsible manner. PDA permits decisions to be presented with a transparent and defensible rationale and is often, therefore, appealing on that account. This is particularly true of issues subject to substantial controversary, for example, the siting of a high-level nuclear waste repository, where whatever decision is made is quite likely to be challenged up to the Supreme Court. A third distinctive feature of government, compared with business, has to do with institutiona1 constraints on the decision process itself. In business, major decisions are commonly made by executives who have a large amount of experience and knowledge of the issue in question. They may have spent twenty years working on substantially comparable problems. In government, on the other hand, senior decision makers (especially political appointees) are often new to the particular decision tasks they must address and are obligP.d to rely on the expertise of others. PDA lends itself naturally to this "knowledge division of labor." It permits the decision maker to break down one big decision problem into a number of smaller decision problems, each of which taps a distinctive source of expertise available to him. Unless he has some alterna-lZ-ijbT GOP} I\VAluusLL. L/-. ?~/
PAGE 49
tive way of tapping into and combining this expertise, he stands to gain substantially from a PDA framework. The businessman, on the other hand, can rely on his own informal reasoning processes, operating on the store of information which is already in his head.
PAGE 50
3.0 MAJOR USES WITH EXAMPLES The following are some of the uses (with illustrations) of PDA in the executive branch, which have been particularly successful for the agencies that have used them, and appear to offer the greatest promise for more widespread application in these and other agencies. 3.1 Defense Oriented Applications Among the public sector organizations, the Department of Defense originally took the lead in both the development and application of PDA technology. Many of the most significant theorei-.ical .,,nd methodological developments in PDA were sponsored by Defense agencies around the mid-70's, especially by Engineering Psychology Programs in the Office of Naval Research and the Cybernetics Technology Office of the Defense Advance Research Projects Agency. These led first to demonstration projects, and then to routine application in a few areas. This technical development has been paralleled by development of teaching materials and courses in decision analytic techniques for defense managers. See Barclay, et al., 1977, which also includes a review of defense applications of PDA. The following appear to be the most successful of these an~ ~he ones whose adoption in other agencies appears most promising. 3.1.1 Resource allocation. Resource allocation, especially in budgeting processes, is becoming an increasingly popular application of PDA in a number of defense agencies (such as, for example, the Defense Nuclear Agency). It typically involves rating competing projects (and different levels of expenditure within each) on each of a number of common attributes, whose relative importance is weighted. The quantified judgments involved would typically be supplied initially by staff and then finalized by the responsible committee or decision maker, using standardized software. Among the reasons for PDA's popularity in this use are: the standardized and repeating nature of t~e decisions (enabling the initial investment in analysis -14'-/ 8' titSl CfJPY AVAILA1il1.
PAGE 51
and training to be recovered ~ver many applications); multiple sources of ex pertise (on each competing project); the need to equitably balance conflicting interests; the need for the decision to be reviewed at several different levels within the Department of Defense, and sometimes above. There is no reason not to expect corresponding growth in other agencies, which almost always have comparable circumstances. 3.1.2 Procurement and systems acquisition. Hierarchical multi-attribute utility software packages are being increasingly used, both to evaluate alternative weapons systems and alternative contractors to supply a given system. Depending on the size of the stakes involved, evaluation structures may get very complex, with distinguishable attributes numbering in the hundreds, and occasionally thousands. The appeal of PDA in this use rests on its ability to decompose a complex problem into a multitude of smaller ones; its providing a framework for resolving controversy, and its ability to support "design-tocost" procurement strategies. This latter has the advantage of flexibility over the practice of seeking lowest bids for a preset design, and was previously the favored format because cost could be quantified and, before PDA, design rarely could. There is no obvious reason why non-defense applications of PDA for procurement should be any less successful, and they are beginning to re so used (for example, by DOE in contracting for a demonstration geo-thermal project). 3.1.3 Contingent decision aids. Resource allocation and system acquisition, discussed above, are both examples of "current" decision analysis, i.e., the analysis is carried out at the time the decision is current. This is contrasted with "contingent" decision analysis, where the analysis is performed in advance of a contingency which will make a decision necessary. This mode is typified by an analysis (highly classified, of course) of whether to press the nuclear button in the event of some future ominous contingency. Substantial development effort has been devoted to such contingency analysis for defense purposes, notably in the development of tactical decision aids, for example, for air combat and submarine combat engagement situations. The primary motivation is to economize on the amount of time needed to make a decision when it becomes current, where reaction time is exceedingly short. Illustrative situations include decisions of whether to fire at an uniden--15f-(
PAGE 52
tified approaching aircraft, when to fire a torpedo, and which of multiple targets to engage. As a result of complicating issues, typical of such problems (urgency, user overload, confidence, and the high cost of error), in order to be adopted here, PDA aids need to clear high thresholds of performance, with particular attention to problems of user interface. This leads to a need for great caution before such an aid replaces, or even competes for the attention of, a human decision maker. A critical feature of any contingency decision aid is the ability to incorporate on the spot information and judgment that may not have been anticiated at the time the aid was designed. This might be achieved not only by allowing the decision maker to override the output of an aid, but also to override selected inputs or intermediate variables. Although substantial resources are being devoted to developing the field, including a major ONR program on "operational decision aids," we are not aware yet of any such aids being adovted in an operating system, but a number of aids are approaching the status of an "initial operating capability." Efforts at other agencies are at a still earlier stage of development. They include a prototype developed for EPA to prompt safety measure decisions at laboratories handling hazardous materials. Promising potential applications include decision rules for responding to nuclear incidents such as TMI. The potential benefits of a successful contingency decision aid of this kind, if it could embody the best available technical and value judgments, wculd clearly be very large. 3.1.4 Information strategy. A relatively immature applied area of PDA, but one with substantial potential, is the evaluation of information gathering strategies, for example, the purchase of an intelligence satellite, ~r investment in research and development. It is immature because of the technical difficulty of formally taking into account the complex and uncertain uses to which information will be put, and the potential value of such uses. The conventional decision theory paradigm known as "preposterior analysis," has serious implementation flaws which have yet to be successfully overcome. Substantial effort is being devoted by the military for developing practically useful PDA methodology here (for example, by the Ad~anced Methodology Research -16-c;(',
PAGE 53
and Development group at CIA). However, to date it has had limited practical success (except for relatively crude applications of multi-attribute utility where information value is directly assessed instead of being explicitly ,....,deled). The potential value of PDA methodology developed in this area is enhanced by the same factors that limit its current success. I.e., the same difficulties that stand in the way of a PDA solution also make it difficult to do intuitively, which means there is great room for improvement if we can but realize it. 3.2 Civilian Oriented Applications 3.2.1 Health and safety risk regulation. Health and safety risk regulation has been a particularly fertile application area for PDA. This is partly due to the adversarial nature of the process, which establishes a need for a defensible argument (for or against a proposed regulatory action); and partly as a result of Executive Order 12291, which, in effect, calls for an explicit weighing of pros and cons for regulatory actions with major impact (more than half a million dollars in general). Should the Ritter Bill, currently under consider.ation, be passed, it would require the major regulatory agencies to conduct demonstration risk analyses for which PDA is one promising candidate approach and this would further stimulate the application of PDA in the regulatory process. An additional stimulus was provided by a report of the prestigious Committee on Risk Analysis and Decision Making of the National Academy of Sciences which recently advocated increased application of decision analysis to regulatory decisions. The lead agency is probably the Nuclear Regulatory Commission, which has used decision analysis in support of a number of regulatory actions, including the prioritization of regulatory effort, the evaluation of proposed new regulatory requirements, and the evaluation of safPguard designs. There have been a number of other applications of PDA to regulation, including explosives tagging and medical records standards. 3.2.2 Environmental management. Environmental decisions are a particularly promising area of application since they involve the balancing of conflicting -174)
PAGE 54
objectives like economics and ecology and are subject to intense public controversy. One case study involved working with two successive Secretaries of the Environment in the State of Massachusetts to help decide whether the Connecticut River should be partially diverted to ease water supply shortages in eastern Massachusetts. A decision model was constructed which incorporated quantitative predictions of the impact the diversion would have on factors such as cost, ecology, economic disruption, etc., and the relative importance of these factors. The main alternative was to rehabilitate, at substantial cost, existing ground water supplies in eastern Massachusetts. Not surprisingly, the conclusion depended critically on whose importance weights were used: representatives of eastern Massachusetts favored the Connecticut diversion (which was in western Massachusetts); ~hereas both western Massachusetts and the State of Connecticut were in favor of the main alternative which was to rehabilitate eastern Massachusetts ground water supplies (at substantial cost to eastern Massachusetts). 'What was interesting was that the interests of eastern Massachusetts would have to be given overwhelming predominance (some 80% of the total constituency weight) before diversion was favored overall. However difficult it might be to assign precise political weights to the various constituency, it was clear that such imbalance would not be indicated and therefore the diversion was solidly disfavored. Other representative examples include: whether to seed hurricanes, whether to invest in satellite power systems; whether to enlarge the Shasta Dam. Potentially, the most significant application of PDA may prove to be in conjunction with the implementation of the Nuclear Waste Policy Act of 1982, which over the nex.: few years, is intended to produce decisions on the siting of two repositories for high-level nuclear waste. In the process of developing recommended sites, the Department of Energy is explicitly calling for the use of PDA (or a close variance of it) on an ambitious scale in documenting various stages in the decision process. 3.2.3 National policy. On a number of occasions, alternative national economic and foreign policies have been evaluated using PDA. Examples include an evaluation of alternative mid-East oil sources (for the National Security Council), and alternative export control policies on computer technology (for -18s :)--
PAGE 55
the President's Council on International Economic Policy). In both cases, PDA was used to help senior staff in a short period of time to distill expert knowledge on separable aspects of the problem and synthesize them into a policy evaluation for the President. In both these cases, as in others, although staff's position was derived using PDA, ic was presented to the President or his National Security Advisor in traditional qualitative terms. The potential value of the PDA approach in such cases would no doubt be enhanced if the argument could be presented effectively in PDA terms, but this presupposes a level of training the ultimate decision maker which is not cur-rently realistic. 3.2.4 Negotiations. A distinctive application of PDA, particularly in for eign policy contexts, is negotiations, in this case treaty negotiations. In this case, PDA is used to characterize the purported interests of all parties, including the U.S., with a view to exploring probabilities for joint gain by all parties. This approach has been used in support of negotiations on the Panama Canal, the law of the sea, and tanker design. 3.3 Agency Communication with Congress Of particular relevance to this paper are uses by executive agencies of PDA to communicate with Congress. They are not yet on a large scale but two areas are significant. 3.3.1 Program evaluation. PDA is sometimes used by executive agencies to evaluate congressionally mandated programs. One such example was the community anti-crime program administered by the Law Enforcement Assistance Administration, where a multi-attribute utility framework was presented. The program was scored on a number of attributes such as crime and fear of crime, based on a combination of expert judgment and field surveys. In a related exercise, the implications of different levels of congressional funding for the program were evaluated using essentially the same multi-attribute utility model. A very simplified version was presented to the congressional staffs of appropriate subcommittees, whose response appeared generally favorable. -19-
PAGE 56
3.3.2 Support for legislative proposals. In another PDA study, indications of alternative legislation on tax incentives for residential solar heating were evaluated on behalf of Federal Energy Administration. Although, in the cases of which we are aware, the initiative to develop PDAbased conclusions came from the executive agency itself, there have been indications from congressional staffs that they would welcome more presentat~ons of this kind as a way to receive information in a compact, but reviewable form, whose congressional action implications can be readily extracted. BEST COPY AVAILM~b
PAGE 57
4.0 EVALUATION AND FINDINGS 4.1 The Case for Executive Branch UsP of PDA The pros and cons of PDA as a technique for itself weighing the pros and cons of decision options has been recently evaluated in contrast and comparison with cost-benefit analysis (Watson, 1980?). 4.1.1 Potential advantages. When successful, PDA has the following arguments in its favor: The quality of decision making may be enhanced, in the sense that the decisions follow more logically from the data and expertise available. (This is not to say that this is always achieved by PDA. It requires a skilled analyst, but not necessarily an elaborate analysis, to outperform a capable decision maker's intuition.) It makes the reasoning behind a decision transparent and available to scrutiny (this may be more valuable co the reviewer of a decision, for example, a congressional oversight committee, or someone higher up in the executive branch chain, than whoever is promoting the decision). It may encourage decisions to be made in accordance with required, defensible considerations, such as service to the public and cost savings to the tax payer, and make it more difficult to advance hidden agendas (such as empire building). In one study for GSA on whether certain computing services should be provided in-house or coritracted out (pursuant to the Brooks Act), a simple computerized PDA decision aid showed that certain government officials' recommendations in favor of keeping the work in-house could only be sustained if a higher importance weight was put on "administrative morale" (read empire building) than on service to the public and savings to the tax payer! Different parties can contribute more effectively to the decision making process, including different types of expertise and different levels of authority. Without such a structured framework, it is more difficult for any given contributor to limit his contribution to a properly circumscribed role. E.g., the technician may need to get into political issues, and the high-level decision maker may not b~ able to avoid getting into detail he does not care to address. It may pinpoint the soft spots in an argument and highlight were more definitive work or research needs to be done. It may make the total executive branch decision making process more orderly (which is not necessarily the same as better--the British empire was built, they say, on the art of muddling through!). It may be easier to defend a decision to others, for example, to a court or the general public. (This requires, of course, that the --.----<:'--:-(..__ ) _) .. -I
PAGE 58
intended audience understands what it is being told in a PDA. This requires a level of skill in the communicator or training in the audience which is by no means trivial, though PDA may be easier to understand than oany other "scientific" arguments.) 4.1.2 Cautions. To set against these potential advantages of having federal agencies make and report their decisions and positions with personalized decision analysis, are a number of cautions which, if not.adequately heeded, may jeopardize or negate that value. PDA is still an immature technology and it requires both analyst and user to make effective u~e of it. it requires a fusion of mastery of the methodology the subject matter, which is still rare. unusual skill by In particular, with mastery of It may give the illusion of being more definitive than it can justify, by virtue of being precisely quantified. It may shift the locus of power in an or~anizational hierarchy in unintended ways, notably in the direction of greater centralization, since it makes it easier for higher level authorities to intervene selectively and effectively. (This may, of course, be considered a desirable consequence, for example, in permitting the congressional oversight function to be more active.) Care must be taken to use PDA where it is most appropriate, for example, where the options are clear but, their impacts are difficult to evaluate; where the stakes are high; and where there is substantial room for improvement in the current decision process. 4.2 Action Implications 4.2.1 Desirable developments. It seems clear that decision analysis, as currently developeJ, has potentially valuable applications throughout federal government which are only beginning to be realized. This is demonstrated by inroads it has made in certain agencies, such as the Department of Defense, and the absence of its use in other agencies with comparable needs and cir.cumstances. Much could be done to make these selectively exploited benefits accessible and promoted throughout the executive agencies. It is no less clear that tha resources available to effect this implementation are seriously inadequate to the need. In particular, there is a severe shortage of suitably trained people to apply the technology and to use it effectively. This is, in large part, due to the rigidity of the educational establishment which is organized along lines that make coordinated PDA UL~-, f-OPY AVAILAiLL
PAGE 59
development difficult. Decision analysis is an expertise that cuts across traditional institutional divisions of learning, such as mathematics, psychology, sociology, law, economics. Institutions are not set up to encourage development of appropriately multi-disciplinary professionals. Although selected aspects of decision analysis are taught in traditional university departments, such as statistics, psychology, engineering, there is not a single major institution in this country or elsewhere that covers major aspects of PDA in a balanced way, and as a result, few well-rounded professionals are being produced. Advancement of r.he state-of-the-art itself is in somewhat better shape, in that significant funding has recently become available for research in the area (e.g., through NSF, ~A. ONR, ARI). Note that a large fraction of that funding comes through the research arms of the Department of Defense. However, only the Decision and Management Science Program of NSF is explicitly multi-disciplinary. The others are compartmentalized, for example, into mathematics and psychological sciences at ONR. At DARPA, there was a multi-disciplinary PDA program which has now been discontinued on the grounds that its function was only to seed an area. It was left to other institutions to develop it further, but this has only happened in a fragmented way. The full flowering of PDA in executive agencies will not doubt need to wait until there is a critical mass of both decision makers and PDA practitioners who have appropriate mastery both of the substantive area and PDA methods. This may have to vait until PDA is taught routinely in colleges and even high schools as an essential art of "making up one's mind." 4.2.2 Possible congressional initiatives. Related legislative initiatives include the Paperwork Reduction Act and the Ritter Bill. These might be augmented or amended by legislation to encourage or require more effective exploitation of the potentials of de~ision analysis The Risk Analysis Demonstration Act (Ritter Bill) can be seen as a special case of decision analysis. Such legislation might include the institutional responsibility for promoting decision analysis, such as OSTP, or an executive agency, such as 0MB. As a first step towards stimulating the more generalized effective use of decision analysis within the executive branch, Congress might sponsor a fact -23,.. .. --; ) I
PAGE 60
finding study on how PDA is currently being used and what opportunities there are for more extensive or more appropriate application. This could be in the form of~ sub-committee report oriented towards a particular agency, possibly accompanying an ppropriations or authorization bill. Another initiative, more diroctly useful to Congress, would be to ask for agencies to submit justifications for key budget requests in a decision analytic format. I.e., it would be part of the accompanying documentation for the budget. Congress might single out particular topics such as the MX Missile or the Strategic Defense Initiative, where the stakes are high enough and the controversy great enough that Congress might find it useful to have conpact, structured argument that lends itself readily to critique and secondguessing. A more ambitious variant of this suggestion (that has been tried on a pilot basis in support of the community anti-crime program before appropriate House and Senate sub-committees) would be to develop a simple, but comprehensive, computerized model of the options available and allow congressmen or their staffs to judgmentally override key assessments of fact or value.
PAGE 61
REFERENCES Barclay, S., Brown, R.V., Kelly, C.W., III, Peterson, C.R., Phillips, L.D., and Selvidge, J. Handbook for decision analysis. McLean, VA: Decisions and Designs, Inc., September 1977. Brown, R.V., Kahr, A.S., & Peterson, C. Decision analysis for che manager. New York: Holt, Rinehart and Winston, 1974. GAO/PAD-82-46. Sur~ey to identify models used by executive agencies in che policymaking process. Washington, DC: U.S. General Accounting Office, September 1982. Howard, R.A. & Matheson, J.E. (Eds.). The principles and applications of decision analysis (Vol. I & II). Strategic Decision Group, 1983. Katzper, M. Report for OTA on use of models and simulation (in preparation), 1984. Kraemer & King, 1983. Luce, R.D., and Raiffa, H. Games and decisions. New York: Wiley, 1957. Raiffa, H. Decision analysis. Reading, MA: Addison-Wesley, 1968. Watson, S.R. Deciaion analysis as a replace for cost-benefit analysis. European Journal of Operational Research, 1981, 7, 242-248. -25-
PAGE 62
SELECTED APPLICATIONS OF COMPUTER-AIDED DECISION ANALYSIS AND SUPPORT May 10, 1985 Prepared for: Dr. Fred Wood Office of Technology Assessment United States Congress Prepared by: Rex V. Brown Jacob w. Ulvila Decision Science Consortium, Inc. 7700 Leesburg Pike, Suite 421 Falls Church, Virginia 22043 (703)790-0510 Under Contract No. 433-0315.1
PAGE 63
TABLE OF CONTENTS Page 1. SCOPE OF TASK 4. . . . . . . . . . . . . . 1-62 2. SYSTEMS ACQUISITION: A MILITARY HELICOPTER EXAMPLE ................... 2-63 2.1 2.2 2.3 Multiattribute Utility Analysis ................................. Evalu&ting the Advanced Scout Helicopter ........................ Steps in the Analysis .......................................... .. 2. 3 .1 Identification of attributes ............................. 2.3.2 Evaluation of alternatives on attributas ................. 2.3.3 Prioritization of the attributes (weighting) ............. 2.3.4 Comparison of the alternatives ........................... 2.3.5 Sensitivity analysis .................... : ................ 2 2 3-64 3 4-65 8-69 9-70 11 3. RESOURCE ALLOCATION: AN R&D BUDGETING EXAMPLE ........................ ]3-74 3 .1 Introduction ..................................................... 13 3. 2 Method Description. . . . . . . . . . . . 13 3.2.l Describe projects and assess costs ........................ ls-76 3.2.2 Describe benefits ......................................... 15 3.2.3 Assess benefit contributions of project options ........... 17-78 3.2.4 Deter.nine cost-effective budget allocations ............... 19-80 3.2.5 Sensitivity analysis ...................................... 19 3.3 An R&.D Budgeting Case Study ...................................... 20-81 3.4 Use of the Results ............................................... 27-88 4. CONTINGENT DECISION AIDS. . . . . . . . . . . . 29-90 4.1 Introduction ..................................................... 29 4. 2 A Hypothetical Example. . . . . . . . . . . 31-92 4.3 State-of-the-Art ................................................. 33-94 .. 4. 3. 1 ONR sponsored work ........................................ 33 4.4 Issues in the Development of Contingent Aids ..................... 36-97 4.4.1 Factors affecting implementation .......................... 36 4.4.2 Applicability of decision aids for an attack submarine commander ....................................... 37-98 4.5 Future Prospects ................................................. 38-99 4.6 Recommendations .................................................. 39-100 APPENDIX A: EXPERIENCE OF TilO R&D ORGANIZATIONS IN DEVELOPING CONTINGENT DECISION AIDS .................................................. 41-102 A.l Decisions and Designs, Inc. (DDI) ................................ 41 A.1.1 Template-based aids ....................................... 41 A.1.2 Knowledge-based decision aids ............................. 42.-103 A.2 Decision Science Consortium, Inc. (DSC) .......................... 42 A. 2 .1 Submarine attack aids ..................................... 42 A.2.2 Concepts in development ................................... 44-105 REFERENCES ........... ................ . . . . . . . . . 46-107
PAGE 64
1. SCOPE OF TASK 4 Task 3 of this project for OTA broadly reviewed uses, Loth actual and potential, by the executive branch of computer-aided decision analysis and support (DAS), and more especially, of the tools of personalized decision analysis (PDA). In Task 4 we explore in greater detail, and illustrate with case studies, two application areas of particular interest. The first area is government, and particularly defense, procurement, which is perhaps the area where decision analysis has most widely and most successfully been used in the past. We have distinguished two sub-areas: using multiattribute utility analysis for evaluating military systems (Section 2); and resource allocation for research and development, budgeting, and similar purposes (Section 3). In both of these cases, an example is developed in enough detail to demonstrate the approach. The second area is contingent decision analysis, i.e., preparing for a possible contingency (such as a military threat or a natural disaster) by, at least partially, preprogramming the decision ahead of time (Section 4). A broader review of potential applications is given, with discussion of various issues bearing on directions for future development. By contrast to the procurement area, contingent decision aids have no standardized approaches, as yet, and few implemented applications. However, they may have the greatest potential promise of any area of application, in the field of computer-aided decision analysis and support. -1-f '1 ,0 ;__,
PAGE 65
2. SYSTEMS ACQUISITION: A MILITARY HELICOPTER EXAMPLE Military systems acquisition is one of the most prevalent and successful areas of application of DAS, and the technique of Multiat~ribute Utility Analysis (MAU) is the dominant technique used. 2.1 Multiattribute Utility Analysis MAU analysis is a PDA techn:que for assessing the value of alternatives that have multiple effects. MA1J models explicitly reflect the relative important of each objective, and, therefore, the tradeoffs among them. By doing so, a MAU model enables the evaluator to develop a summary measure of value reflecting many kinds and degrees of impacts on these objectives. The key stages in a MAU approach are: Identification of what is to be evaluated (alternatives or options); Definition of the components, or attributes of value (what is important); Evaluation, or "scoring" based on the attributes (how is each alternative rated on each attribute?); Prioritization of the attributes of value; Comparison of alternatives being evaluated (which system scores highest on all factors combined?); Sensitivity analysis on assumptions and judgments (what if priorities change?). Multiattribute utility analysis has been used by various organizations in the Department of Defense to aid in the evaluation of major military systems, often for purposes of procurement. Some of these systems include the Advanced Scout Helicopter (ASH), Light Armored Vehicle (LAV), Mobile Protective Weapons System (MPWS), and SINgle Channel Ground and Airborne Radio System (SINCGARS). 2.2 Evaluating the Advanced Scout Helicopter The steps of a MAU analysis are described in detail in the sections below in an application to evaluate advanced scout helicopter (ASH) candidates. This illustration is a highly simplified presentation of an actual analysis con--2.--) --, (./1 ,--
PAGE 66
ducted for the ASH Program Manager. The illustration here compares only five alternative designs along only twenty-two attributes. The full analysis compared thirteen alternative designs along seventy-six attributes. TI1e full, detailed analysis is contained in Donnell and Ulvila (1980). This illustration considers a subset of advanced scout helicopters that were evaluated. These are: 1. A new development with twin advanced technology engines and side-byside seating (BT2); 2. A new development with twin advanced technology engines and tandem seating (BTT); 3. OH-1 with a mast-mounted sight (OHM); 4. A new development with twin advanced technology engines, side-byside seating, and capability to operate at 4K/95 (feet/F) with one engine inoperative (B4K); 5. OH-SBC (SSC). In the full analysis, thirteen candidates were evaluated. These included minor and major modifications of existing U.S. and foreign helicopters as well as several completely new designs. 2.3 Steps in the Analysis 2.3.l Identification of attributes. Attributes in a MAU analysis consist of those features, objectives, and co~cerns that serve to different~ate :he va~ue of the alternatives. It is desirable for a complete set of attribute~ to have the following characteristics: Be comprehensive enough to account for most of wnat is important in evaluating the options; BE: able to highlight the differences among options; Reflect separate, nonoverlapping features to avoid double-counting. Deviations from these characteristics can be accommodated, but with an increase in the complexity of the analysis. BESl CGPY AVAILA~LL
PAGE 67
It is often convenient to define and arrange the attributes in a hierarchy, especially if there are many attributes. In the ASH analysis, seventy-six attributes were arranged in a fairly complex hierarchy. The top level of the hierarchy, as shown in Figure 2-1, contained the broadest characterization of attributes; military worth; life cycle cost; attainability (a measure of how well the funding profile fit with the projected budget); force structure personnel impact; and rationalization, standardization, and interoperability (a measure of fit with NATO forces). Each of these attributes was typically subdivided. Figure 2-1 shows the subdivision of military worth into: operational acceptability, technical system, and technical system risk. This "top down" process of subdividing attributes continued until attributes were iden-. tified on which alternatives could be readily compared. Figure 2-2 shows the complete subdivision of the technical system attribute. The most detailed description of attributes in this case contained such items as: navigation equipment, vertical rate of climb (VROC), rotor diameter, and crashworthiness. (For purposes of this illustration, the structure contained in Fugures 2-1 and 2-2 will be treated as complete.) 2.3.2 Evaluation of alternatives on attributes. In order to use the MAU model to evaluate alternatives, it is necessary to develop a measurement scale for each bottom-level attribute. Such measurement scales are developed using natural standard units whenever possible (e.g., dollars, knots, feet per minute), but it is often necessary to use more subjective, relative scales. The two major procedures for scoring, relative scoring procedures and absolute scoring procedures, could be used in the model. Since relative scoring is the most widely used, it is the focus of our discussion. Relative scoring is the easier technique to use. A common measurement of value is used, relative utility, and it is measured on a standard scale (e.g., 0 to 100). The numerical endpoints of the scale are somewhat arbitrary in that any numbers could be used; however, once fixed, the endpoints serve as a reference point for other assessments. In relative scoring, for each factor, the alternative that is "best" on the factor is assigned a score of 100, while the "worst" alternative is assigned a score of 0. The range of such a scale thus measures the difference between options--a score of 100 can be thought of as 100% of the potential improvement on a factor over and above the worst case, which scored 0. Note that a score of O does not necessarily imply that the alternative has no value. Rather, it indicates that the alternative is -4--~i '~ ~ ~\t')-\'' .. D .. 11, ~ur. ... ,. 1~" (p S
PAGE 68
(.7',1 ,,. .. ...., .t ,c. q> -r ;:... -. )a, ;::-it i: I V1 I Figure 2-1. Simplified Attribute Hierarchy for Evaliating ASH Candidates Opcrntlonul Acceptability Overall Value 1 Mllitn ry Worth Lif c Cycle Cost Technlcul System 1 Sub-divided as shown in Figure 2-2 I Technical System Risks At toinability re, Ip. Fore:? Structure Personnel Jmpact Rntlonalizatlon, Standardization, IntcropernhJllty
PAGE 69
I I I -"' ... 0, ~-~" Figure 2-2: Subdivisions of the Technical System Attribute Milit,ry Worth 1.1.l Ttehnlc1I Sy11rm 1.U.I MEO P1th;t 1.1.2.1.I N 1vig1lion E q uipmtnl 1.1.l.1.2 Communiu1ion [quipmtnl 1.1.1.l TA/US 1.1.1.1.4 MEO Graw1h 1.1.U.1 r11la,m1nC1 1.1.1.1.1.1 Man11mabilily 1.1.2.2.U Spud 1.1.U.l.l F u,1 I.U.2.1.4 vnoc From: Donnell and Ulvila (1900) 1.1.l Ttthnlul Sy11,m 1.1.11 Airf11m1 ,.,.1.1.l rhy,iul Chu 1clrri11ic1 ,. 1.1.1.1., T 11111port1bili1y 1.1.U.U Ra101 Oi111111 tr 1.U.1.1.l n,1,i,v,bility 6? l.t.U.l Surtiv,bilily 1.1.1.1.l.1 0(1 1.1.U.J.2 Vulnuability I. U.1.l.J Cr 11h-wor 1hi11m 1.UJ.l.4 ASE 1.1.l.l S,11,m lntr9111inn
PAGE 70
the baseline for comparison. All other alternatives are scored on the Oto 100 scale relative to how they compare with the endpoints. A score of 50 on the above-defined Oto 100 utility scale means that the sat~sfaction level, or utility of the alternative is midway between the best and worst. Note that a score or 100 on one attribute cannot be directly compared with a score of 100 on another attribute since they may not be equally important. In order to compare attributes, a weighting procedure must be applied, as described in a later section. As an example of relative scoring, consider vertical rate of climb (VROC). This scale can be measur~d in feet per minute (at constant environmental conditions, such as 4K/95 (feet/F)). These candidates had the following VROCs: Candidate BT2 BTT OHM B4K SSC VROC (feet per minute) 900 900 500 2000 0 Next, the relationship between VROC and value must be established. This is the process of assigning a value (or utility) to different performance levels. On the Oto 100 point value scale, the utility for VROC was as follows: 100 75 Utility 50 25 0 0 500 1000 1500 2000 VROC (feet per minute) This indicates a decreasing marginal return for increased VROC. -7-
PAGE 71
Sometimes there are no natural units for the attributes being compared. In these cases, other descriptors are needed to establish a score. For example, crash-worthiness was scored by reference to the following value scale: Crash-Worthiness Prevents post-crash fire, absorbs most vertical energy, maintains cockpit integrity, delethalized interior Prevents post-crash fire, absorbs some vertical energy, retains cockpit integ~ity Prevents post-crash fire, absorbs some vertical energy Utility 100 40 25 Only prevents post-crash fire 0 Scores for all candidates were assessed for all attributes. In this application, a wide variety of sources were used _to provide assessments. These included: results of life cycle cost studies, results of simulations, expert judgments of helicopter pilots, expert judgments of engineers (avionics and electrical), and judgmen~ of the program management team. This is typical for a MAU analysis. The analysis often serves as a way to develop a comprehensive evaluation that accounts for all sources of input, data, and judgment. 2.3.3 Prioritization of the attributes (weighting). In the scoring system described above, an evaluation scale from Oto 100 was developed for each attribute. However, each scale was defined independently of all others, so the resulting scores are not directly comparable. In reality, some attributes carry more importance in the evaluation than others, and a measure of the priority, or relative importance, of each factor is necessary. This is accomplished through a weighting system. As with the scoring system, weighting judgments are personal, and different decision makers could have different sets of weights. The most common perception of a weight b that it answers the question, "How important is attribute A relative to attribute B?" Unfortunately, such a measure is often misleading. A more pertinent question to ask is, "How impor canc is che difference in the range of values for attribute A versus the dif ference for attribute B?" The subtle difference between these two questions -8-,.f' '--( ~.:..., ,. ,., u ... I .A,; t'' Jh ,. ~.\a Ll.
PAGE 72
is extremely important. The latter question incl1Jdes both the importance of the attribute as well as the "swing" in the range of values on the attributes. As an example of this distinction, speed is very important in the abstract. However, the difference in speed between 88 knots and 165 knots (which is the range represented by candidates) is relatively unimportant because a scout helicopter would not operate at a speed in excess of 88 kno:s in most missions. Thus, speed received a relatively low weight. Weighting can be accomplished top-down or bottom-up. Top-down weightin~ 5s easier and is more widely used. In the top-down approach, the analyst begins at the top of the hierarchy, and assesses the relative importance of differences among attributes. For example, at the top level of the ~ierarchy in Figure 2-1, the question is, "How important are differences in military worth compared with differences in life cycle costs; attainacility; force structure personnel impact; and rationalization, standardization, and interoperability?" A common approach is to assign a weight of 100 to the most important swing. Other weights are then assigned usi~g ratio judgments-that is, if the s~ing on an a~tribute is judged to be twice as important as the swing on a~other attribute, the former would carry twice the weight of the latter. In this example, the following weights were assessed: Attribute Category Military Worth Life Cycle Cost Attainability Force Structure Personnel Impact Rationalization Standardization, and Interoperability Weight 100 60 30 6 4 For comparability, the weights are then normalized to sum to 1.00 by adding the assigned weights and dividing each by the sum. This process if then repeated for each node in the attribute hierarchy. 2.3.4 Comparison of the alternatives. After all alternatives have been scored on the attributes, and weights have been assigned to the attributes, the overall measure of value for each alternative is determined. Since the MAU analy;is described here has independent attributes, the overall score will be an additive combination of scores and weights. (In more complicated -9-; (l BEST COPY AVAJlA~iL.
PAGE 73
structures, where attributes interact, a more complicated model is appropriate.) Starting at the bottom-level nodes, a weighted-average score is calculated for each alternative. The table below shows the calculations for the airframe performance node. 1 1 2 2 1 ASH>MILWORTH>TECHSYS>AIRFRAM.E>PERF ATTRIBUTE WT BT2 BTI' OHM B4K SSC 1) MANUVRABTY .30 100 100 90 100 0 2) SPEED .10 93 96 62 100 0 3) FUEL .20 70 70 60 70 100 4) VROC .40 75 75 SO 100 0 COMBINED 83 84 6S 94 20 Scores of each candidate are shown against each attribute (maneuverability), speed, fuel, VROC) in the table, and each attribute's weight is given in the "WT" column. The weighted-average evaluation of .each alternative is given in the bottom row, labeled "COMBINED". These evaluations are then carred to the next higher :evel and combined with the results of other nodes. The following table shows the calculations for the airframe node: 1 1 2 2 ASH>MILWORTH>TECHSYS>AIRFRAME ATTRIBUTE WT BT2 BTT OHM B4K SSC 1) PERF .48 83 84 65 94 20 2) PHY CHAR .14 85 85 33 85 75 3) SURV 38 80 88 42 82 0 COMBINED 82 85 52 88 20 Notice that scores in the "PERF" row are the same a!. in the "COMBINED" row of the previous table. The combined scores at this level are carried up to the next level and combined with other results, and so forth, until the top node is reached with the following results: 1 ASH A'.TTRlBUTE WT BT2 BTT OHM B4K SSC 1) MIL WORTH .so 77 79 69 80 24 2) LCC .30 39 39 41 38 97 3) ATTAINABTY .15 26 26 25 26 96 4) FSI .03 so so 0 so 98 5) RSI .02 69 69 9 69 0 COMBINED 57 58 .51 SE. 58 -101' I
PAGE 74
In this example, all candidates have overall evaluations that are approximately the same. However, they exhibited widely varied performance on some of the attributes. For example, compare the scores of BT2 with those of SSC in the table above. More detailed descriotions of differences are provici~l by examination of scores. A more systematic examination is obtained by an orden:d attribute-by-attribute comparison between candidates. Such a comparison bet.Veen BT2 and SSC .is shown below: ATTRIBUTE-BY-ATTRIBUTE COMPARISON BE'IVEEN BT2 AND sac BT2 sec sr2 sec NODE LABEL wEIGHTED DIFF CUM WTD DIFF -----------------------------1 1 1 OP ACCEPT 13-275 13.275 1 1 2 1 3 TADS 6.075 19.350 1 1 2 1 2 COMMS 2. 700 22 .050 1 2 1 NAV 2.363 24.413 1 2 3 SYS lNT 2.025 26.438 1 5 RSI 1-380 27.81 8 1 1 2 1 4 GROWTH 1 .080 28.898 2 2 1 1MANUVRAE'l'Y 972 29,870 1 2 2 1 4VROC 972 30,842 2 2 3 2VULVERABTY 821 31 .662 1 2 2 3 3CRASH '.'HY .513 32. 17 5 1 1 2 2 3 10EI .462 32 .6 37 1 1 2 2 1 2SPEED .301 32,938 1 1 2 2 3 4ASE .257 33.195 2 2 2 2ROTOR DIA .108 33.303 1 2 2 2 3RETRIEVBTY .000 33,303 1 1 2 2 2 1 TRANSABLTY -.013 33.289 1 1 2 2 1 3FUEL 194 33.095 1 4 FSI -1.440 31 6 55 1 3 SYS RISK -5.000 26. 6 5 5 1 3 ATTAI.N!ETY -.500 16. 155 2 LCC -1'7 .400 -1 .245 This shows, for example, that the most important differences favoring BT2 are operational acceptability and the target acquisition/designation system (at the top of the table). Conversely, the three most important differences favoring SSC are life cycle costs, attainability, and system risk (shown at the bottom of the chart). An analysis such as this often serves to guide a sensitivity analysis. 2.3.5 Sensitivity analysis. It is usually desirable to perform sensitivity antlyses with any decision support model. MAU models are no exception. Often, in working with multiple sources of input, there are disagreements that BEST COPY AVAILl,~1.
PAGE 75
may never be resolved through a consensus building process. Rather than spend significant resourced debating the issue, it is better to first determine if a change in the input affects the result. If not, there is little to be gained in further data collection and debate. There are three major types of sensitivity analyses that are often used. First, the scores that have been assessed are modified to determine if results change. Often, however, results are reasonably insensitive to minor changes in scores, and there is usually a high degree of confidenc~ in the assessed values. Second, weights can be changed and the overall scores recalculated. This is useful in examining large-scale changes to models (such as using weights for a different decision maker), but does not make it easy to isolate causes of change. A third sensitivity analysis is to vary one weight systematically through a range and identify the regions where decisions change. For instance, as the weight increases, the total weight of the other f~ctors must decrease, but the weights might be kept in the same relative proportion to each other. For example, the following result is obtained by varying the weight of military worth over the range of 0% to 100%. WEIGHT BT2 BTT OHM B4K SSC .0000 37 37 32 36 93* .1000 41 41 36 41 86* .2000 45 45 40 45 79* .3000 49 so 43 49 72* .4000 53 54 47 54 65* .5000 57 58 51 58 58* .6000 61 62 54 63* 51 .7COO 65 67 58 67* 44 .8000 69 71 61 71* 38 .9000 73 75 65 76* 31 1.0000 77 79 69 80* 24 This result shows that SSC receives the highest evaluation for weights up to 50% and B4K receives the highest evaluation for weights greater than 50%. The highest scoring system for each weight is indicated by the asterisk. -12-1
PAGE 76
3. RESOURCE ALLOCATION: AN R&D BUDGETING EXAMPLE* 3.1 Introduction This section describes a decision-analytic approach to resource allocation, and to R&D budgeting in particular, that has been used by the Defense Nuclear .. Agency (DNA) and by other governmental and business organizations. The method is described and then illustrated in a disguised version of DNA's application. The approach can be (and has been) adapted to a wide variety of resource allocation decisions. 3.2 Method Description The method ccmbines the techniques of decision analysis, multiattribute utility analysis, and cost-effectiveness analysis to provide an R&D planning and budgeting method that: ensures consistency between one-year budgets and longer term goals and objectives; is responsive to budgetary changes and technical changes in emphasis; develops a database for project evaluation, review, and control. Cost-effectiveness analysis is the unifying technique in this method. Basically, the idea of cost-effectiveness analysis is to prioritize project funding increments so as to provide the greatest total benefit within the budget. To work effectively, however, an appropriate measure of benefit must be defined. This is provided by the techniques of multiattribute utility (MAU) analysis and decision analysis. MAU techniques provide a basis for defining an appropriate measure of benefit, one which capturec the goals and objectives of the organization. Decision analysis provides techniques for assessing and measuring benefit as defined by the MAU analysis. Development of an R&D plan and budget follows six s~eps. First, options are specified and costs are estimated. Each project is described in enough detail to assess its probable results and the value of *This section is adapted from Ulvila, J .W., & Chinnis, J.O., Jr. Analysis for R&D resource management. In D.F. Kocaoglu (ed.), Hanagement of R&1) and Engineering, North-Holland, 1985. -13BESl cnpy .4 VA/! AB
PAGE 77
those results. In addition, different possible funding levels are identified for ea~h project. Second, a measure of benefit is defined. Benefit reflects the goals of the organization in pursuing the projects. Benefit might be a single quantity or it might have multiple dimensions. Where multiple dimensions of value are appropriate, multiattribute utility methods are used. Third, benefits are assessed for each project for a selected funding level. Each project's relative contribution to the goals is assessed using the best data and judgment available. Fourth, benefits are assessed for different levels of funding within each project. This assessment allows the development of a complete research plan that is composed of the best level of funding for each project. Fifth, cost-effective priorities of project fundings are determined. The cost-effectiveness of a given funding level is determined considering its benefit (steps 3 and 4) and cost (step 1). Priorities are determined from incremental costs and benefits with the highest priorities going to increments that contribute the greatest benefit per unit of cost. Funding programs in this priority order result in allocations that provide the most benefit for the cost. Efficient allocations, and their relationships to other allocations, are shown graphically in Figure 3-1. At any budget level, an efficient allocation provides more benefit than any other possible allocation. Sixth, sensitivity analyses are performed and the model is revised. Costeffective budget allocations can be sensitive to changP.s in any of the assessments. These might include changes in: cost assessments (step 1), benefit assessments (step 3), and assessments of the relationship between benefit and funding levels (step 4). In addition, some assessments are likely to be highly uncertain or speculative. A sensitivity analysis identifies those parts of the analysis where improvements in the quality of assessment are most important. Another important aspect of the sensitivity analysis is the evaluation of trial budgets. Trial budgets (proposed sets of funding levels for all projects) are evaluated and compared with cost-effective allocations. The comparisons indicate the changes required to achieve more value for the same budget (a better allocation). -14(..-,--) I I
PAGE 78
100% Benefit 0 $0 Efficient Allocation fo~ SX Other Efficient YAllocations t I I I I 1. I I I I I I X Budget ($) ~. : : Total Set of Possible Allocations Budget Required to Fund all Projects Figure 3-1: Efficient Allocations Each step is described in more detail in a section below. 3.2.1 Describe projects and assess costs. The first step in the method is to describe each project and estimate its costs. Descriptions should be as complete and concise as possible to aid in the assessment of both costs and benefits. In addition, to the extent possible, projects should be described in such a way that each is independent of the rest. This may lead to a combination, for purposes of this analysis, of otherwise separate projects. The method accommodates different funding level possibilities for projects, and it is especially important that several possible levels of funding be described for each project. This provides the basis for a pre-analysis of the program changes to make in the event of budget changes and it forces a closer scrutiny of the proper level of funding for each project. 3.2.2 Describe benefits. The second step in the method is to describe the benefits that are expected from the Re~ projects. This description should include the entire range of contributions that projects are expected to make to the goals and objectives of the organization. In cases where multiple objectives are important, it is important to choose the set of projects that strike the best balance among all objectives. In cases such as this, the techniques -15BEST COPY AVAILAWLl
PAGE 79
of multiattribute utility analysis (see Section 2) can provide the means for achieving the proper balance. Multiattribute utility (MAU) analysis provides an appropriate procedure for assessing the value of programs in cases where multiple objectives are important. For each attribute, a scale is developed that relates improvements on the scale to the value to the organization. Such a scale could be developed using natural ~tanda~d units (e.g., dollars for cost, months or years for time, etc.) when such units exist. Next, the relationship between changes on the scale and the value of the changes is established, and the value is incorporated in the model using a standard scale, such as a Oto 100 scale. In cases where no natural units exist, a completely relative value scale, such as a Oto 100 point scale, could be used directly. Here it is important to define the points on ~he scale carefully in terms of the attribute being represented. In eith~r case, the relationship between variations on the scale and the value should be established. The other step in the development of an MAU model is the specification of weights for each attribute. The interpretation of weights and the procedure for assessing them depends strongly on the form of the model to be employed to aggregate the single-attribute scales. The theoretical bases for a variety of aggregation models have been developed (see Keeney and Raiffa, 1976), but often in practice an additive aggregation rule is appropriate, or a sufficient approximation, so that is our focus here. Weights are assessed by considering the relative importance of moving from the worst to the best level on each attribute. The assessments are ratio judgments, meaning that if the importance of going from Oto 100 on one attribute scale is twice that of another attribute scale, the first scale is assigned a weight twice that of the second. This assignment of weights on the basis of "swings" is an important feature reflecting the theoretical basis of MAU that differentiates it from weighting procedures based on some ill-defined concept of "importance." These ratio judgments are normalized to sum to one to facilitate further calculations. The benefit of each project is then computed by summing its contributions to each attribute weighted by the normalized weights, as illustrated in the example below.
PAGE 80
It is desirable for the list of attributes to exhibit the following characteristics: comprehensiveness, separateness, and manageability. The set should be comprehensive enough to account for most of the important objectives of the R&D program, at least as represented by the project~ under consideration. Each criterion should also measure a separate non-overlapping objective. If some objectives overlap, they might be combined into a single attribute. In addition, as a practical matte~. the number of attributes should be kept to a manageable number. For analyses of the type described here, we suggest that 15 is about the maximum manageable number of attributes. A couple of techniques might be used to pare down a larger list. First, some attributes might be combined. Second, less important attributes might be dropped. For example, attributes with weights of less than 10% of the largest weight might be dropped. Alternatively, attributes with normalized weights of less than 4%-5% might be dropped. The resulting list should then account for most of the value yet have a manageable number of attributes. 3. 2. 3 Assess benefit cJntributions of proi ec t options. The third step is to assess the benefit contributed by each project. Benefit is assessed in two parts, the benefit contributed by different "target level" projects and the change in each project's benefit at different levels of funding. In cases where benefit is a multidimensional quantity, the project's contribution to each attribute is assessed. Benefit contributions might be assessed in a mun ber of ways depending on the nature of the measure (e.g., whether or not it is easily characterized in natural units) and the availability of data and expert judgment. Sources of information include statistical data, analytic or simulation models, field experience, or judgment. For attributes with natural units (such as dollars or months), assessments are straightforward, being made for each project in terms of the natural units. These assessments are then translated into benefits using the scales developed in step 2. For attributes that lack natural units, the following two-stage process is often a useful assessment procedure. First, programs are grouped into four categories, category A (containing programs with the most favorable performance on the criterion) through category D. (Although this first stage is not required, respondents often find it easier to provide this grouping than numerical scores, at least at a first pass.) Numerical scores are then assigned to the projects on a 100-point scale with a score of 100 being assigned to the best performance on the attribute, which is often exhibited by -17Bf ST COPY AVAiLAYLL
PAGE 81
Benefit one of the projects under consideration, and a score of zero to the worst performance on the attribute or to no positive contribution to the attribute. Scores between O and 100 are assigned to projects based on their relative performance on the attribute. For example, a score of 50 is assigned to a project that exhibits half the contribution to the attribute as a project receiving a score of 100. This scoring procedure addresses only the performance of projects within each attribute. The relative importance between attributes is assessed by the weighting procedure described above. The Oto 100-point scales on the attributas ensure consistency in the scaling. 'When assessing the scores for each program, the respondents should refer to a specific description of the project since the benefits of a program depend on what is done in that project (e.g., different benefits would accrue from a program if it involved just calculations than if it involved calculations, prototype development, and a field test). Next, assessments are needed of how a project's value changes as the level of funding changes. This can be done by asking respondents to specity the percentage of a program's total value that could be attained at each of the levels described. Two patterns of benefit changes are most common, a decreasing increme,tal contribution and a linear contribution. These patterns arc illustrated in Figure 3-2. With decreasing incremental benefit, each higher level of funding adds less than a proportional benefit. With linear increased benefit, each higher funding level adds proportionately the same benefit Another pattern is sometimes encountered which might be called a "dip." With this pattern, some lower funding levels provide proportionately less benefit than higher levels. This pattern is sometimes encountered when "placeholder" funding levels are proposed or when a project consists of complementary parts. Benefit Benefit O l 2 3 Funding Level ($) 0 l 2 3 Funding Level ($) l 2 3 4 Funding Level ($) Decreasing lncret11ental Benefit Pattern Linear Incremental Benefit Pa-ttern "Dip" Pattern Figure 3-2: Benefit Patterns -18-1((
PAGE 82
3.2.4 Determine cost-effective budget allocations. The fifth step of the method is to uze the assessed benefit contributions and costs to determine cost-effective allocations, those uses of funds that maximize benefit within cost constraints. The cost-effective order of increasing the funding of projects is determined by ordering the transitions between levels on the basis of their benefit-to-cost ratios. The transition with the highest ratio is first, the transition with the.second highest ratio is second, and so forth. Choosing to fund the projects in this order ensures maximum benefit within a budget (Everett, 1967). (This statement is approximate if the allocation does not use the entire budget.) Typically, only a small fraction of possible allocations will be costeffect~ve. For example, in the description of a complete analysis present~d below, cost-effective allocations were determined for twenty-two projects, each with about ~hree possible levels of funding. The analysis indicated 60 cost-effective allocations out of over 9,000,000,_000,000 possibilities. 3.2.5 Sensitivitv analysis. Often, components of tbe analysis, cost and benefit assessments, are not known precisely. In addition, some assessments are usually made judgmentally and are subject to disagreement. These conditions make it especially important to investigate the sensitivity of the results to variations in the input. Several inputs might be varied: cost assessments, benefit assessments between projects, or benefit pattern assessments within projects. Each input influences the incremental benefit-to-cost ratio of transitions and thus influences the order of transitions and cost-effective projects at different budget levels. Sensitivity analyses might be conducted by varying groups of parameters and re-calculating results or by selectively investigating the extent that certain inp~ts would need to change in order to give a different result. An important set of sensitivity analyses to conduct is one that spans the range of opinions where there are disagreements. Often, disagreements are resolved when a sensitivity analysis shows that they make no difference in the final conclusion. Another type of sensiti"'rity analysis is provi.ded by examining trial allocations or sets of funding levels for all projects. Comparing the benefit and cost of the trial allocations with that provided by cost-effective allocations -19f?()
PAGE 83
can indicate areas for revision if the trial allocation is not efficient. These types of sensitivity analyses are readily accommodated by the computer software developed for this purpose. 3.3 An R&D Budgeting Case Study The following is a description of an application of the method tc, prioritize R&D program funding alternatives as a basis for a one-year budget plan. The description is a disguised version of an actual analysis used by a division of the Defense Nuclear Agency (DNA). Table 3.1 is a summary display of program scores on multiple attributes and weights for the attributes. In the example, twenty-two R&D programs (labeled 1-22) were considered. Each program was evaluated on its contribution to each of eleven attributes (shown as "obj. a" through "obj. k"). These correspond to the attributes in the MAU analysis that was deyeloped as described above. Some attributes were indexed on objective, quantitative scales with ~atural units, and some were strictly judgmental. In cases where scales were strictly judgmental, a relative scale was developed and scores of O and 100 were defined by the programs. The weights shown correspond to tradeoffs across attributes. n,ese weights represent the relative importance of the Oto 100-point variations on the objectives. The weights show, for example, that the 100-point variations on objectives c and d were judged to be equally most important. The 100-point variations on objectives a and e were next in importance, and each was about 85% (.12/.14) as important as objective cord. For each program, a score of Oto 100 is shown against each attribute; these are the assessed contributions of "target level" programs, those levels with the "target level" costs shown. (Rationales for the scores were recorded where appropriate.) The "BENEFIT" column shows the overall weighted-average contribution of the "target level" program to the attributes. For example, BENEFIT of Program 2 is calculated as: (.12)(0) + (.08)(100) + (.14)(30) + (.14)(60) + (.12)(0) + (.08)(25) + (.08)(75) + (.06)(10) + (.06)(50) + (.04)(45) + (.10)(75) 41 (Both weights and benefits are rounded in Table 3.1). -20<1:.J ()
PAGE 84
Tabl~ 3.1: Program Scores and Attribute Weights 0 0 0 0 0 0 0 0 0 0 0 b b b b b b b b b b b j j j j j j j j j j j "TARGET LEVEL" a b C d e f g h 1 J k COST BENEFIT -------weights: O. 12 0.08 0.14 0.14 0.12 0.08 0.08 0.06 0.06 0.04 0.10 1 )program 1 0 100 80 50 0 100 75 60 30 30 70 700 53 2)program 2 0 100 30 60 0 25 75 10 50 45 75 1000 11 1 3)program 3 0 100 90 55 20 50 50 30 50 75 60 800 51 11) program 4 0 100 20 90 0 90 85 110 50 80 75 800 52 I 5)program 5 0 100 50 50 0 75 75 60 50 50 65 1000 i1a N t-6)program 6 92 0 25 50 0 60 75 20 25 25 700 110 I 75 7)program 7 0 20 80 0 0 100 20 100 20 40 0 350 31 8)program 8 11 20 50 75 0 90 20 70 110 90 20 600 40 9)program 9 92 0 20 40 75 90 60 20 30 90 30 700 119 10)progrom 10 92 0 20 80 100 100 60 20 30 80 25 1100 57 ll)program 11 1 0 20 60 0 90 25 20 ., 0 100 25 500 28 12)progrrun 12 12 50 0 30 0 90 25 30 25 80 25 800 27 13)program 13 0 50 10 60 0 100 25 10 10 100 25 500 31 14)program 14 1 0 10 110 0 90 50 20 20 50 10 350 23 15)program 15 12 50 100 40 100 110 50 90 30 80 65 1000 60 16)program 16 2 30 10 30 0 75 50 50 30 90 75 600 33 17) program 17 100 20 30 60 20 90 50 110 20 50 75 300 52 CD 18)program 18 0 20 80 0 0 100 50 80 0 100 0 350 33 a 19)program 19 \1 0 20 60 0 90 35 25 0 100 50 275 32 20)program 20 1 20 10 65 0 35 60 15 50 100 50 300 32 -0 21)program 21 92 20 30 110 0 0 75 10 90 75 40 600 111 -< > 22)program 22 1 10 0 110 0 100 0 30 30 60 0 300 20 < ,-> Note: Weights and lleneCits shown ore rounded. s. r-r-g,;)_ __
PAGE 85
Table 3. 2 1how1 the costs and benefits of different levels of funding for each program. For example, Program l is shown with three possible levels of funding, $250,000, $500,000, and $700,000. The relative overall benefits of the three levels were assesaed to be 551, 801, and iOO\ of the "target level" (Level 3). This gives a benefit pattern for Program las shown in Figure 3-3. 100% X X Relative 50% X Benefit 0% 1 2 3 (250) (500) (700) Funding Level ($000) Figure 3-3: Benefit Pattern for Program l The cost-effectiveness priority of the funding levels is determined by multi plying the relative benefit increases shown in Table 3.2 by the appropriate weighted-average BENEFIT cf Table 3.1 and then dividing the product by the cost increments of Table 3.2. The multiplication places the incremental benefits on a common scale, and the division determines the cost effectiveness. For example, the increase in benefit of increasing the funding of Program l from nothing to Level 1 is cal~ulated as (551) x (53) -29. The benefit of increasing from Level 1 to Level 2 i= (801-551) x (53) 13, and the benefit of increasing from Level 2 to Level 3 is (1001-801) x (53) 11. Similarly, the benefits for increasing levels of Program 2 are: (371)x(4l)l5 (to Level 1), (621-371)x(41) (Level 1 to Level 2), (871-621)x(41) (Level 2 to Level 3), and (1001-871)x(41) (Level 3 to Level 4). Dividing these incremental benefits by their respective incremental costs gives the following benefit-to-c~st ratios for each increase in Program l's funding: 29/250.12 (Level 1), 13/(500-250).05 (Levell to 2), and 11/(700-500).05 (Level 2 to 3). These ~ompare with Prngram 2's benefit-to-cost r~tios of: 15/250.06 (Level 1), 10/(500-250)-.04 (Level 1 to?.), 10/(7505C).04 (Level 2 to 3), and 5/(1000-750).02 (Level 3 to 4). -22...., ?>? The highest
PAGE 86
Table 3.2: Program Funding Levels Level 1 2 3 I 4 I I I Relative Benefit < '> I 55 80 100 Program 1 I Cost {$000) 250 500 700* I Relative Benefit < '> I 37 62 87 I 100 Proy.ram 2 I Cose ($000) 250 500 750 ~000* I Relative Benefit < '> I 25 50 100 Program 3 I Cost ($QC')) 200 400 800* Relative .Benefit ( I) I 50 75 100 I Program 4 Cose ($000) i 400 I 600 I 800* I Relative .Benefit (I) 15 I 50 I 100 i Program 5 Cost ($000) 250 I 500 1000* I I I Relative .Benefit < '> I 45 80 100 I Program 6 Cost ($000) 200 500 I 700'11 I Relative Benefit < '> I 40 67 100 Program 7 Cost ($000) I 50 100 350* Relative Benefit < '> I 20 40 100 I Program 8 Cost ($000) I 100 200 600* I i Relative Benefit ( I) 17 I 50 100 I ?rogram 9 I Cc st ($000) 100 I 250 i 700 ... I Program 10 : Relative .Benefit ( I) I 45 I 80 100 I Cost ($000) ; 150 300 400* _I Relative Benefit < '> I 12 55 I 100 I I Program 11 Cost ($000) I 50 250 I 500* I .I Relative Benefit C ') I 40 80 100 I Program 12 I Cost ($000) i 300 600 800* I ---. *"Target level" -23-'i~ l/
PAGE 87
Table 3.2 (Continued) Level 1 2 I I 3 4 I I Relative Benefit(\) Program 13 I Cost ($000) 20 so 1 100 I 100 250 I 500* I Relative Benefit (I) 10 60 100 I Program 14 Cost ($000) so 200 350* I Relative Benefit(\) I 30 60 I 100 1' I Prog~am 15 1--------------------....;.... __ .._ __ _, Cost ($000) I 300 600 I 1000* I I Relative Benefit (\) 25 I 75 I 100 I Program 16 Cost ($000) 100 400 I 600* I Relative Benefit (I) 66 I 83 I 100 I I Program 17 Cost ($000) so I 100 I 300* 1 J Program 18 I Relative Benefit (\) 47 77 100 Cost ($000) 150 250 350* I Program 19 : Relative Benefit c') I 20 100 Cost ($000) 1100 275*1 Relative Benefit (\) 50 I 100 I Program 20 Cose ($000) 100 I 300*, I l Program 21 I Relative Sen.?fi c (I) 45 I 85 100 I Cose ($000) 250 500 I 600* Relative Benefit <') I 80 I 100 I I Program 22 Cose ($000) I 200 I 300*! *"Target level" -24'-6
PAGE 88
priority increm~nt is the one with the highest benefit-to-cost ratio, calculated in this manner. Table 3.3 shows the co~t-effectiveness priority for budgeting, an ordering of all level changes in decreas~ng order of incremental benefit to incremental cost. In the illustration, the most cost-effective project, considering its contribution across the multiple objectives, is Levell of Program 17. This is followed 9y Levell of Program 7, Level 2 of Program 17, Levell of ~rogram 10, and so forth. Numbers in Table 3.3 are rounded. In cases of true ties, the convention is to fund lower levels before higher ones, for example, Levels 1, 2 and 3 of Program 4 in orders 24, 25, and 26, respecdvely. In cases where a "di,:," pattern aprears, the "dip" is inefficient and the benefit-tocost ratio is calculated for the double-transition that spans the dip. For example, at order 8 Program 19's funding moves from Level Oto Level 2, bypassing the dip at Level 1. Table 3.3 also shows the total weighted-average contribution to attributes ("Total Benefit") and total cost of following the cost-effective order. At any .dven level of total budget, the most cost-effective combination of project fundings is given by taking all of the changes indicated in the prioritization down to that level. For example, the cost-effective use of a total budget of 1125 ($1,125,000 in the illustration) is to fund the first nine transitions in the order shown. This results in the following set of project fundings: Program l Level l $250,000 l'rogram 7 Level 2 $1UO,OOO Program 10 Level 2 $300,000 Program 17 Level 2 $100,000 Program 19 Level 2 $275,000 Program 20 Level 1 $100,000 All Others Level 0 Notice that, because of the way that program levels were defined, this order accounts for synergies and dependencies among the programs. Also notice that Table 3.3 shows 60 cost-effective combinations over the entire range of budget possibilities from $50,000 to $12,925,000. This contr~sts sharply with the total number of possible funding combinations of: (3)3 x (4) 18 x (5) 9 f 2 77 f 000 I 000 -2sS 0 BEST cnPY AVAIWLL
PAGE 89
Table 3.3: Cost-Effective Priority for Budgeting L.EVEI. BDP.FIT COST T0:11. TC':'.C. ~EN!l'IT ORDER PROCRAM CBANGE CiWIGE CBJJICE !DiE:IT COST COST ----------------1 17)progru 17 0> 1 3-' 50 3t 50 .68 2 7 )progru 7 0> 1 12 50 100 .24 3 :-:, )prograi:: 17 1> 2 9 50 55 150 .18 II 10) rogru 10 0> 1 26 150 C1 300 .17 5 7)progru 7 1> 2 8 50 89 350 .l6 6 20) prop-aa 20 0: 1 16 100 105 1150 .16 7 10)progru 10 1> 2 20 150 125 600 .13 8 19)progru. 19 O> 2 32 275 158 875 .12 9 1 )progru 1 O> 1 29 250 187 1125 .12 10 10)prosraa 10 2> 3 11 100 198 1225 .11 11 18)provu 18 O> 1 15 150 2111 1375 .10 12 18)progru 18 1) 2 10 100 2211 1 .10 13 9)program 9 0> 2 25 250 2 1725 .10 1 II fi)program 6 O> 1 18 200 266 1925 .C9 15 16)prosram 16 0) 1 8 100 275 2025 .08 16 22)program 22 0> 1 16 200 291 2225 .08 17 20 )program 20 1> 2 16 200 307 21125 .os 18 8);,rogram 8 O> 1 8 100 315 2525 .08 19 r )progru 8 1> 2 8 100 323 2625 .08 20 1 J) progra= 8 2> 3 8 100 330 2725 .(18 21 21 )progru 21 O> 1 18 250 3118 2975 .Oi 22 111)program 1.11 O> 2 1 II 200 362 317~ .v:23 11 )progru 11 O> 1 3 50 366 3225 .06 211 ll)program 0) 1 26 oo 392 3625 .06 25 ll)progru II 1) 2 13 200 1105 3825 .06 26 ll)program II 2) 3 13 200 1118 11025 .06 27 21 )progru 21 1) 2 16 250 .-311 11275 .06 28 3)program 3 O> 1 13 200 U7 111175 .06 29 3)progru 3 1> 2 13 200 1160 11675 .U6 30 3)program 3 2) 3 26 1186 5075 .06 31 111)prog:-am 14 2> 3 9 150 1195 5225 .06 32 1 3 ) program 1 3 0) 1 6 100 501 5325 .06 33 13) prop-a= 13 1) 2 9 150 511 5475 .06 311 13)progru 13 2) 3 15 250 526 5725 .06 35 21)progru 21 2) 3 6 100 532 5825 .06 36 2)progru: 2 0) 1 15 250 5117 6075 .06 37 15)program 15 0) 1 16 300 565 6375 .06 38 15)progru 15 1) 2 18 300 583 66i5 .06 39 15)progru 15 2) 3 2.11 1100 607 7075 .06 JIO 8)program 8 2) 3 211 631 71175 .06 11 )program 11 1> 2 12 200 611;? 7675 .Oe li2 16)progru 16 1) 2 17 300 660 7975 .06 3 9iprogru 9 2> 3 25 1150 6811 8425 .06 Jill 1 )program 1 1) 2 13 250 697 8675 .05 1 )program 1 2> 3 1 1 200 708 8875 .05 ll6 11 )progra= 11 2> 3 12 250 720 9125 .05 1'7 5)progru 5 0) 2 211 500 7411 9625 .05 8 5)prog:-u 5 2> 3 211 500 768 10125 .05 49 6)program 6 D 2 111 300 782 101125 .05 50 17)progru 17 2> 3 9 200 791 10625 .Ci4 51 16 ) program 1 6 2) 3 8 200 sno 10825 .04 52 2)progru 2 1> 2 10 250 810 11075 .04 53 2)progru 2 2> 3 10 250 820 11325 .04 511 7)progru 7 2> 3 10 250 830 11575 .04 55 22) program 22 1) 2 I( 100 834 11675 f'\4 56 6 )progru 6 2> 3 8 200 842 11875 .04 57 12)prograi: 12 0> 1 11 300 853 12175 .04 58 12)progru 12 1) 2 11 300 8611 121175 ,04 59 12)program 12 2> 3 5 200 869 12675 .03 60 2)prograi: 2 3> 5 250 875 12925 .02 Note: Nut:ibers in this table are rounded. -26-I BEST COPY AViiLAilt
PAGE 90
Figure 3-4 shows a plot of the percentage contributions and costs of cost effective program combinations. For ~xample, Table 3.3 shows that after 21 transitions, a total benefit of 348 is attained at a total cost of $2,975,000. This is plotted as point A on Figure 3-4, since 348 is 40\ of the total benefit attainable (348/875-.40). Similarly, after 31 transitions, 57% (495/875) of the benefit is attained at a cost of $5,225,000, which is plotted as point B. 3.4 Use of the Results Results from this application were used in a variety of ways. The initial development of the analysis brought a focus to the R&D planning process development of the analysis by delineating goals, objectives, and tradeoffs and by framing the contributions of R&D projects in terms of those goals and objectives. Exercising the analysis through sensitivity analyses helped to resolve disagreements. The continued use of the analysis aided DNA in adjusting the plan to respond to changes. The method provided a means for rapid responses to budget changes and rapid evaluations and prioritizations of new project opportunities. -27-RL~T ~OPY AVAILAiLL
PAGE 91
Figure 3-4: ?lot of Cos:-Effect~~~ ?riori:y PLOT OF TOTAL BENEFIT VS. TOTAL COST FOR EFFICIENT ALLOCATIONS 10080-, p C T M 60A X I H 0 M B E N 40-E F I T 20-+ I+ I+ I + + + + Point A 0 + ++ + + + + + + ++ Point B 0 + + + + ++ ++ + + + + + + + + + + ++ + + ++ + + + + o-~-------------l------------1------------:------------:------------I 0 2600 5200 7800 10400 13000 TOTAL COST Note: Precision in this graph is limited by the character Yidth and height. -28c; ~ (/ am cePY AVAtlt\lii.l
PAGE 92
4. CONTINGENT DECISION AIDS 4.1 Introduction A contingent decision aid is one which has been designed ahead of time to help a decision maker in some anticipated future contingency, for example, to help him respond to a military threat or an environmental emergency. It contrasts with the more conventional kinds of current decision aids discussed above, which are developed to solve a problem which has already arisen, such as deciding which weapon system to acquire, or which budget to adopt. In principle, contingent decision aids can be in the form of verbal decision rules ("shoot when you see the whites of his eyes"), or mechanical devices (e.g., that shut down a boiler when it gets too hot). However, we are concerned primarily with a particular class of contingent decision aid, one where a decision rule is preprogrammed in advance of the contingency for which it is designed, such that in the event, the decision analysis can be performed very rapidly and almost invariably with the support of a computer. In all of our tasks on this project for OTA, we are primarily concerned with personalized decision analysis, i.e., analysis which includes the quantification of personal judgment. In the present task, this means aids where judgment is incorp~~ated, not only in the spe~ification of the aid (in advance of any contingency), but also in the execution of the aid (for example, in supplying "last minute" inputs for overriding preset inputs). The essence of PDA is that cocplex human judgments can be broken down into simpler components of uncertainty and value, which can be quantified and reassembled. The logic of this process--manipulating probabilities and utilities--is the province of mathematical statistics and has occupied much of the early work in the field. The human factors issues of PDA--capturing judgmental input and communicating the implied output--is the province of psychology, and has of late assumed critical importance for implementation. The dividing line between current and contingent decision analysis is somewhat fluid. Current decision analyses are built on some prior analytic development (for example, the generic resource allocation model discussed in Section 3), and contingent decision analyses allow for some analytic judgmental interven--29/,j /': I L,/ BES"'f COPY .~VAilABLt.
PAGE 93
tion after the contingency has occurred. (Otherwise, they would not be decision "aids," they would be decision "makers.") The main distinguishing characteristic is reaction time. It might be a matter o! seconds (responding to an incoming air or torpedo attack), or minutes (deciding when to attack a target/thre~t or how to respond to a reactor incident) or hours (responding to indications of an impending conventional attack from the Warsaw Pact, or choosing protective measures in handling a toxic substance). ~'by should one want to analyze a decision before knowing whether it will be needed? Surely, if we wait, we can save ourselves possibly unnecessary effort (if the contingency does not arise) ~r (if it does arise), we will have more up-to-date information about what exactly the circumstances of the decision are. There are two broad motivations for contingent decision analysis. One is that the si~uation will be repeated often enough that the cost of any anticipatory effort can be spread over many applications and, therefore, be cost effective. This would be typified by an inventory control system. These are relatively straightforward aids to develop and test because, since the task is repetitive, illustrative applications are likely to be plentiful. The other main motivation is that the reaction time is so short and the cost of a mistake is so high, that when the contingency arisP.s (if it does), there will not be enough time to do the analysis required; therefore, as much as possible of the analysis must be clone ahead 'lf time. A prime example would be, in response to a nuclear attack, a decision whether to make a counter or preemptive strike. A topical, and most complex, variant of this is the multiplicity of defensive re~ponses addressed by the Strategic Defense Initiative ("Star Wars") effort, where the expenditure of billions of dollars is envisaged. (This has some elements of the repetitive decision situation, since each of perhaps thirty thousand incoming warheads will present comparable decision problems.) In addition, the reaction time may be too short to permit higher level policy makers to intervene, &u~h that their inputs on values and tradeoffs can be represented. A classic exawple is the P~eblo incident, where a U.S. Captain, threatened by North Korean naval fore.es,. had to decide, within a matter of a -30-, 7' BlST cnP~ AVAILAWLL
PAGE 94
few hours, whether to surrender, resist or evade. In the absence of contr~ry guidance, he was essentially left to his own judgment on the value tradeoff between his men's safety and more global national interests. In cases such as this, there can be no guarantee that the local decision makers' priorities will be dominated by national priorities. Some kind of contingent decision aid might serve to segregate value from technical judgments, with responsibilities for them split between appropriate parties. 4.2 A Hypothetical Example* A ship's commander, under conditions of international crisis, engaged in a routine exercise, spots an unidentified plane approaching threateningly. He cannot be sure whether it is friend or foe and, if foe, whether it is intent on attack. Should he interrupt the ship's routine maneuvers and, if so, should he preemptively attack the incoming plane? He must decide within minutes. He calls up on his computer screen the program TACAID, which is a contingent decision aid designed for just such a situation. He is shown the display in Figure 1. The small circle in the triangle represents his current uncertaiTIy about the threat--each corner of the triangle represents one of the three hypotheses about the plane (routine surveillance, full-scale preparations to atteck, hostile action impending). The probability of each corresponds to the distance of the circle to the corresponding corner. In the position shown, since it is nearest the "routine" corner, that is the most probable hypothesis. As new information comes in, the probabilities will change and the circle will move (e.g., along the path shown). The triangle is divided up into three regions which prompt certain actions to be taken as the probability circle roams over the triangle: routine operations, stage to attack and attack. In the situation shown in the figure, the circle is initially in the "routine operations" region and, therefore, that is what the aid is prompting the captain to do. Where the boundaries are drawn depends on basic policy (value judgments). For example, if the "attack" region is small, this represents a policy where the risk of shooting down a *The material in this section is a~apted from Brown, et al. (1975). -31-q'l BEST COPY AVAILAWLL
PAGE 95
friendly plane is treated more seriously than failing to shoot an enemy attacking plane. "'' ILi CT 10,, .. .. ,-000 .... "'"' ..... Figure 4-1: A PDA Contingent Decision Aid Reproduced from Brown, et al. (1975) The definition of the regions and the loca~ion of the probability circle reflect judgments of value and uncertainty, respectively. The captain can set them himself (in which case the aid is only helping him nrganize his own thinking); or they can be set automatically by the computer as a function of preset value judgments, intelligence items as they are received, and technical judgments of the diagnosticity of those items. If he chooses, the captain can call up the corr~sponding "driver" programs and override either their inputs or their outputs, th.ereby giving him several ways to introduce his own judg ment into the process. He is also not obliged to act on the prompting of the decision aid. This is a hypothetical scenario, in that no such contingent aid is currently in use in the Navy (or, for that matter, anywhere else so far as we are aware). However, the specific tactical decision aid referred to above was, in -32cr 3
PAGE 96
fact, developed as a working prototype by Decisions and Deigns, Inc., and 11 described in some detail in Brown, et al. (1975). 4.3 State-of-cbe-Art In the past few years, substantial resources have been spent on developing contingent decision aids, especially by the Department of Defense. The main thrtl9t has been toward operational or tactical decision aids, though, so far as we know, none have yet been incorp.--rated into standard practice. In any case, thoae aids which are closest to implementation are not, in general, released for public dissemination. 4.3.l ONR sponsored work. One of the early pioneers in this field was the Office of Naval Research, which initiated an Operational Decision Aid Program (ODA) in the early l970's. This provided much of the initial stimulus for the development of contingent decision aids based on ,personalized decision analysis. The goal of the ONR. ODA program was to develop and evaluate techniques that could help overcome some of the known human limitations in decision making under high information loads characteristic of the Navy Task Force Command level. These limitations include faulty memory, stubbornne~s about hypotheses even in the face of contradictory information, limited comput.tio.:,a. ability, rose-colored hindsight (that is, a tendency to remember past decisions as invariably correct), and narrowed focus under stress (that is, a tendency under stress to ignore information about low probability threats and to rely on habitual rather than innovative, and perhaps more appropriate, responses). The program exploited and melded three technology areas: decision analysis, operations research, and computer science. An example of concept based on decision analysis, developed under the ODA program, is the TACAID concept described in Section 4.2 above. It represented an attempt to combine a Bayesian inference model with a 111Ultiattribute utility model, provide flexibility to the decision maker by allowing him to modify the input variables to test sensitivity or to express his own on-the-spot judgments, and present the computed results in simple graphical form. Experimental findings identified three design deficiencies in the original concept: -33-
PAGE 97
1. Th original Bayian inference model did not allow for udden change in an enemy' intent; the moving circle would re1pond luggi1hly to change in input value, especially after a long period of relatively con1tant probabilitie1. In thi1 rpect, the model acted jut like a 1tubbom human. Separate alerting indicator, were needed to prompt th human when significant new evidence of changing threat wa1 received. The decision maker could then introduce step change into the Bayein model. 2. Changes to the probability and utility input values could be made only by reviewing tabular di1play1 and entering preci1e nwaerical data, which proved to be tedious and time consuming. Modem technology would per11it th entries to be made more eaily, *ither by graphic approximation or even by voice. 3. A more complex di1play wa1 needed to portray graphically a 1ituation in which more than three hypothe1ea were being cons~dered. Modern computer technology could undoubtedly overcome thi1 problem Th ODA program utilizd operations re1earch technique, to develop model that could provide a tactical commander with rapid etimate1 of the relative worth or outcome of several alternative tactics that he might be considering. For example, if the objective i to plan a course for a ta1k force to transit from point A to point B, through an area with enemy threat in unknown but e timated poitiom, algorithm can be created to compute criterion measures 1uch aa tranit time and degree of vulnerability for alternative courses drawn on the di1play creen. If a new enemy threat 1 sighted, new probability contour can be computed and displayed, new alternative cour1es can be aaesed, and tradeoff1 can be made in 11lecting a new course. Outcome calculators embodying th technique were developed for air trike operations, anti submarine warfare and electronic emission control tactics (in which different pattern. of emiaion control would result in different tradeoffa between in formation gained and information divulged to an enemy). Th tactical decisions can often involve more variables than can ily be dealt with mentally or on a acratch pad. Th main challenges are to ensure that the appropriate variable are included in th model, or that new variable can be incorporated needed, and to allow th decision maker to rule out obvioualy undeairabl alternative 10 that if he requ11t1 a di1play of the beat alternative, the problem apace 11 realistically constrained. -34-~-' CO,y AVAJJ .41i
PAGE 98
Comp~ter science techniques were developed to facilitate human interaction with tactical data bases. The concept here was that the computer system should not only respond to a user's requests, bur actively alert him to new information and jog his memory about previous events and decisions. The University of Pennsylvania developed a syste~ known as DAISY (~ecisior tiding Information llstem) embodying these features. It could keep track of ~he requests and decisions made by a particular user, call his attention to sub sequent decision situation which are similar in nature, and remind him of the information he requested and the decisions he made previously. It allowed him to set alerting mechanisms for information he considered especially critical, and to remove them when no longer necessary. It also allowed him to set triggers, or thresholds, for decisions he wished to implement rapidly as soon as certain information was received. In summary, the objective of the ODA program was to develop and test a variety of decision aiding techniques that could help ove.rcome human information processing limitations, adapt to th~ needs of individual users, provide flexibility to incorporate new informacion or judgments with old, and act either as advisors recommending courses of action or as evaluators of humaninitiated alternatives. Several of these co~cepts, in much improved form, are now being incorporated into tactical systems, expert systems, and intelligent computer-assisted instruction (ICAI) systems, and have become part of the body of technologies now known as Artificial Intelligence. Currently, a number of research and development organizations are working on a wide variety of contingent aid concepts. The work of two such contractors is illustrative of the current state-of-the-art. Some of the earliest ork under the ONR-based ODA program was conducted by Decisions and Designs, Inc. (DDI), which remains active in this field (Br~wn, et al., 1974, 1975; Peterson, et al., 1976). A brief account of their more recent work, prepared by DOI staff, appears in Appendix Al. Decision Science Consortium, Inc. (DSC), has also been developing decision aiding concepts stimulated by the ONR ODA program, and its work is discussed in Appendix A2. -35-
PAGE 99
" .. 4.4 Issues in the Development of Contingent Aids 4.4.l Factors affecting implementation. The potential value of personalized decitiion aids appears to be very great, which accounts for the high degre~ of attention to them in research and development programs. However, the technical problems which need to be overcome before they become part of standard operating procedures, are also very substantial. Generally, they are greater than those commonly encountered in "current" decision analysis, mainly having to do with the very short reaction time for contingent aids. In a typical current decision analysis, the risk of disastrous error is largely avoided because there is time for ~he decision maker to absorb the "implications of the analysis and see that any major problems are fixed.If necessary, he can disr5,1ard the analysis entirely if he feels it is irremedially flawed. In the typical application of 3 contingent.decision aid, on the other hand, there is / rarely more than a few minutes to evaluate and, if necessary override, the ac-tion prompted. In addition, suer. scarce time as there is will be made even more costly by th~ stresses and pressures of, say, a co~bat situation. To be successful, an aid must eithe~ be on Les own, reliably superior to the decision maker's unaided judgment; or there must be an effective and timeefficient way to incorporate the decision maker's judgment. Moreover, by contrast to current decision analysis, the decision aid designer does not know all the considerations that will be relevant when and if the contingency ~rises. He cannot know everything that the decision maker will know ~t the time a decision is to be made, for the aid to be a reasonable competitor to the decision maker's unaided judgment. There needs to be some way of updating the preprogrammed analysis with new information. Returning to our initial hypothetical example of an aid to respond to an unidentified aircraft, even though the captain can supply his own values and probabilities, there is no guarantee that the options to be compared or the hypotheses to be assessed will be limited to those three (in each case) that the aid is designed for. The net result is that a great deal of development is needed on a contingent decision aid, especially on user engineering, before such an aid can replace, or even compete for the attention of, a human decision maker in a high stakes, short reaction time situation. -36-.'1 -I
PAGE 100
4.4.2 Applicability of decision aids for an attack submarine commander. The following discussion (adapted from Cohen, 1982), relates specifically to attack submarine aids, but has mor~ general applicability. To what extent can decision aids actually help attack submarine commanders solve problems? Ideally, they should improve the quality of inferences and decisions, reduce workload, and enhance organizational morale and efficiency. Yet, reservations about decision aids on the part of potential users are often quite strong regarding every one of those claims: Validity of output. Soft factors--for exblilple, the competence and skill of personnel, judgments about the int~ntions of a foe, or the tactical intuitions of an experienced commander--can often tip the balance in an engagement. An aid based on "objective" factors alone may be seriously misleading. Often it is (quite legitimately) ignored. Yorkload. Will a decision aid become merely another input to the commander, who is already inundated with more data than he can properly handle? Will it demand laborious inputs and continual updating to function accurately? In an actual engagement, would there be enough time to relieve the user's burden, will tasks be turned over entirely to the machines, sacrificing the benefi~s of human participation in the decision process? Organization. Dependence on a decision aid may diminish the status of the CO and implies an unacceptable relinquishing of his authority (and perhaps even r~sponsibility) to the machine. In designing decision aids for passive approach ASW, we hor0 to have dealt successfully with these considerations. Soft inputs. The aids we describe utilize whatever information is available and relevant. The commander and crew are not faced with the choice of disregarding their own experience in favor of a more "factual" approach. Since actual decisions almost always mix objective and subjective factors, decisionanalytic aids supplement objective data by the judgment of decision makers (see Watson, 1980). Such judgments may concern the total picture (when to shoot), a minute detail (bearing errors), or anything in between--depending on the personal style of the CO. Managing workload. The CO is at present virtually inundated with data, yet the task of extracting the implications of these data need not be handed over entirely to the computer. A more promising approach Lwolves selectively
PAGE 101
sharing workload, in a highly flexible manner, under the control of the user. When time permits, and if the CO regards aspects of the current situation as unique, he may substitute his own judgment in place of stored values. Yet, under conditions of stress (as with multiple targets), he can rely on automatically supplied values, based either on prior research or on his own previous adjustments. Command role. In the present combat center, very little (except the periscope) is devoted exclusively to the use of the CO. The provision of com mand decision aids, tailored to his personal inference and decision making style, may focus and reinforce his critical organizational role in the combat center. Decision aids of the sort envisioned here do not replace the Commander--on the contrary, they utilize his inputs and judgments, and display the implications of such inputs and judgments for his consideration and for his ultimate decision. 4.5 Future Prospects When the challenging issues of logic, human engineering and military science have been resolved and integrated, we believe that personalized decision aids will have a major impact, perhaps a revolutionary one, on the conduct of highstakes, short-reaction time decision making. In the short and medium term, military tactical decision making at the local field command level (such as a submarine or aircraft commander) is likely to provide the most extensive applications. The most dramatic and most challenging applications, however, may well come from centralized strategic decision making, notably in the context of the Strategic Defense Initiative. On the civilian front, the handling of emergencies which are natural (e.g., earthquakes), or man-made (e.g., reactor accidents), probably have the greatest promise. The case here is strengthened, both by the need for various commercial and civilian government organizations to demonstrate adequate preparedness for disasters (the lack of credible emergence response has often held up the granting of reactor operating licenses), and by the need to demonstrate, after the fact, that the response was appropriate (as in the TMI incident). The key ingredient to early success is the effective fusion of behavioral and engineering technologies. The obstacles to this, however, may be more institutional and cultural than technical. In the relevant technical -38-fi~I
PAGE 102
communities, the development and acquisition of innovative systems has often been dominated by engineering and hardware oriented considerations. For example, the overwhelming predominance of research sponsored by the Nuclear Regulatory Commission has been in the hard sciences, whereas NRC's own safety appraisals have indicated that as ~1ch as 50% of the risk of severe accidents at most reactors is due to human error. It might be argued that behavioral research is less productive per unit of effort than engineering, but the disparity in the levels of =c~earch effort is too great to be explained by this alone. It is not so much that insufficient be ioral research and development is being done--major programs are being funded, for example, by Engineering Psychology Programs at ONR and by the Army Research Institute. The gap is rather on the merging of the social and hard sciences. For organizational and institutional reasons, projects tend to have either a predominantly engineering or predominantly human behavior orientation._ A military command and con trol system, for example, will typically be the primary responsibility of engineers, with human factors experts brought in only at the end to make fine tuning adjustments. Even in those cases where a deliberate effort has been made to create an interdisciplinary program of research and development (as in the case of ONR's Operational Decision Aiding Program), the tendency has still been for individual projects to fall largely within a single discipline, cor responding to the background of the technical monitor. In many cases of system design, this technical s~gregation is not critical. Personalized decision aids are a notable exception. Their successful development requires the intimate integration of what engineering and mathematical technology can offer with an appreciation of what the limitations and strengths are of human decision makers and how this constrains and influences the design of a contingent aid. 4.6 Recommendations A main recommendation to Congress, then, would be to foster truly interdisciplinary development of decision aiding methodology, both in the training of aid designers (counteracting the disciplinary fragmentation of most academic institutions) and in the direction of research and development projects. -39'--~PY.' ,Iii'\ JlL ~,~, ~u --
PAGE 103
Contingent decision aids permit the carefully considered specific judgments of recognized authorities in selected topical areas to be readily available to on-the-spot crisis decision makers. These judgments do not, and should not replace the judgments of the crisis decision maker--rather, they enhance those judgments. The U.S. Government has invested in research and developing contingent decision aiding concepts and has accomplished the first step--contingent decision aids have been developed and their utility demonstrated. Congress should proceed to the next step--consolidation of the methodologies and pursuit of specific pilot applications. -40-/
PAGE 104
APPENDIX A: EXPERIENCE OF TWO R&D ORGANIZATIONS IN DEVELOPING CONTINGENT DECISION AIDS A.l Decisions and Designs, Inc. (DDl)* Decisions and Designs, Inc. (DDI), has developed and is currently developing contingent decision aids. These aids fall into two broad categories: template-based aids and knowledge-based aids. A.1.1 Template-based aids. In most decision problems, a few well-defined alternative courses of action can be characterized and compared in terms of a potentially large number of evaluation criteria. Research and clinical evidence indicates that unaided decision makers experience difficulty in formulating and organizing the evaluation criteria and considering and aggregating relevant information with respect to the criteria to arrive at a coherent choice. This is particularly true in times of crisis with attendant urgency and stress. A template-based aid can alleviate this difficulty. A decision template is a pre-formatted logical structuring of the evaluation criteria supported by definitions and additional information that will assist the evaluator in comparing the alternative courses of action with respect to the criteria. Templates are situationally oriented and, because they are carefully considered and structured in advance by competent substantive situational experts in a relatively relaxed non-crisis mode, they allow the crisis evalud.tor to focus on the relative comparis'on of the courses of action under consideration, thus making the best use of the evaluator's time while at the same time making availab1.e relevant substantive advice on the critical factors. DOI has developed such decision template-based aids for the military services that address crisis evacuation planning, the projection of military force, response in counter-terrorist situations, and contingency planning. Most relevant is the delivery of CONSCREEN-II, a microcomputer-based contingency screening aid, to the U.S. Army War College. The decision template incorporated in this aid was developed by the Army War College faculty assisted by retired General Officers. It is being used in the course of instruction. *This material was contributed by staff of Decisions and Designs, Inc. -41-/ {) ~-
PAGE 105
A.1.2 Knowledge-based decision aids. In addition to incorporating prestructured evaluation criteria in a decision aid, it is also possible to incorporate specific judgments on relevant topics from informed substantive experts. The user of such an aid can thus benefit not only from advice on what he should be thinking about when comparing difficult courses of action (the evaluation criteria), but also how a reputable expert might assess the alternatives and choose a course of action. In particular, such an aid can provide constructive criticism of a proposed plan of action and even suggest explicit preferred alter11ative plans for consideration by the planner. DOI has developed several decision aids in this category. One, the Target Prioritization Aid, supports the Air Force targeting officer in planning strikes against enemy air bases. Another aid supports an Air Force planner in formulating a command, control, communications (C3 ) countermeasures plan. Both aids focu~ the planner's time, help ensure the completeness of the judgmental assessments necessary to evaluate alternatives and select a course of action, and ensure that t~e final choice coheres with the judgments. A. 2 Dec is ion Science Consortium. Inc. (DSC) A.2.1 Submarine attach aids. In 1979, a effort was initiated, aimed at validating the practical value of some of these concepts. 1he intent was to foster in the fleet the actual implementation of at least one concrete variant. This was both to provide a stringent test of the applied merit of the concept and to uncover the most promising direccions for further development. Attack submarine command and control was selected as the testbed and within it the approach and attach antisubmarine warfare (ASW) scenario (Cohen and Brown, 1980, 1981; Cohen, 1982). The submarine context was favored because: The decision process of the submarine is largely self-contained and can often be considered in isolation from outside influences, at least during an engagement; Major clearcut decisions are available whose enhancement is worth seeking, such as to fire a torpedo; The Naval Underwater Systems Center offered collaboration, which would provide the necessary access to current state-of-the-art of Navy knowhow and to key Navy personnel and facilities. -42-/' () ~)
PAGE 106
The initial thrust of the effort was to review the range of command and control activity in an attack submarine in order to identify situations of particular promise. In particular, the team looked for command judgments with significant room for improvement, and where any improvement would have high impact on success of a mission. Two types of judgment were considered: assessment and action selection. The situation identified for detailed attention was approaching an enemy submarine with intent to attack, which requires both typ~s of judgment. Specifically, the judgmental tasks of assessing target range and choosing the time to fire a torpedo were addressed. In each case, the object was to explore and develop a technology for aiding submarine commanders and staff in making these judgments; specifically, to help them to assimilate the data and the expertise available to them and to combine them effectively with contributions from their own judgment. There are three essential components to an effective decision aiding technology: establishing the primary inputs to the decision process (such as the output of the fire control system Jr key judgmental assessments); deriving the logical implications of those inputs (say, about probable target range); and making those implications effectively available to the command staff (say, through interactive graphic software). The first and third components draw most heavily on the discipline of engineering psychology and the second on statistical decision theory, but all three interact strongly. In this pruject, the specific questions addressed w~re: 'What do I know about the distance of the target? When should I tip my hand by shooting? However, the principles eMployed for the development of aids were themselves quite general. They were intended to be illustrative of decision-analytic tools designed to deal with uncertainty, risk, and competing goals, and to demonstrate a sec of highly flexible human-computer interaction techniques. Thus, if decision aids prove useful in the passive approach setting, they are potentially applicable to a variety of other submarine inferences and decisions, and indeed to military command and control more generally. The current status of this effort is that: -43I I /() t_j_
PAGE 107
a fully operational graphic interactive aid to submarine commanders for assessing target range has been developed and teste1 on potential users in a laboratory setting, but it has not yet leen adopted by the Navy (Bromage, et al., 1984); an aid to prompt torpedo firing and maneuvering decisions has been conceptually designed, in collaboration with a major Navy contractor for Submarine Advanced Combat Systems, and is awaiting funding to proceed further (Brown & Ulvila, 1984). A.2.2 Concepts in development. A spec~fic objective in DSC's research has ber,n the development of decision aid design concept.s for the blending of pre criptive and descriptive concerns: i.e., the effort _to ensure that aiding concepts are comparable with the cognitive representations and informationprocessing strategies of users while at the same time providing effective and normatively defensible guidance. These two objectives can easily conflict: on the one hand, catering to the preferences of the user, whatever they may be; on the other hand, avoiding the errors or biases to which his preferred information-proc,2:ssing strategies might give rise. Because of this conflict, we suspect, high-level users of computer-based information systems typically find that either too little or too much help is offered. Users are caught between systems that serve as passive tools (i.e., which automate routine functions like data storage, sorting, and retrieval) and systems which tend to dominate any dialogue with the decision maker. To deal with these conflicting objectives, DSC has applied three generic decision aid design concepts: Facilitation by the aid of user-preferred information-search and choice strategies; Monitoring the computer of selected human-performed tasks, and prompting to guard against potential biases and fallacies where computer contributions might be of value; Monitoring by the computer of selected computer-performed tasks and prompting for incompleteness of evidence or conflicting results where human contributions might be of value. The first principle maximizes the tailoring of person-computer interactions to the particular style of a user. The second and third principles provide a prescriptive counterbalance: they enable the computer to sense weaknesses in a line of reasoning, whether its own or the user's, and to call for (or offer) help. Thus, they are designed to compensate for deviations from optimality -44-. ... '-~ ~' -~ I,
PAGE 108
that may emerge from the first principle, and do so in the most non-obtrusive way possible. In the work on submarine attack planning (described in Cohen, et al., 1982; Cohen, 1984), DSC has identif~ed five interface functions whose presence is typically required to ensure that an aid satisfies both prescriptive and descriptive constraints. These functions also serve as a (partial) framework for the evaluation of a decision aid design, (1) Planning--users may provide inputs or express queries at any level of "fuzziness" or precision, and at any level of specificity or aggregation; they may organize their exploration of the data base according to any preferred search scheme; (2) Select--the user can examine (preferably through interactive graphics) any significant input, inference rule, intermediate conclusion, or result in the data base; (3) Adjust--a user can alter values of any data base element and immediately observe the impact on results downstream in a chain of reasoning; (4) Alert-the system prompts a user when events occur or fa_cts are learned which may cause significant changes .in user decisions, e.g., which disconfirm currently favored hypotheses or which significantly affect expected payoffs for options; and (5) Advisory--the system prompts a user when a user cognitive strategy or a user-computer task allocation scheme appears to be employed which may be suboptimal. -45-/~=.:lt.,
PAGE 109
REFERENCES Bromage, R.C., Brown, R.V., Chinnis, J.O., Jr., Cohen, M.S., and Ulvila, J.W. Decision aids for submarine command and control (Phase III) Concept implemen tation (U) (Draft Technical Report 83-2). Falls Church, VA: Decision Science Consortium, Inc., 1983. (Confidential) Brown, R.V., Hoblitzell, C.M., Peterson, C.R., and Ulvila, J.W. Decision analysis as an element in an operational decision aiding system (Phase I) (Technical Report 74-2). McLean, VA: Decisions and Designs, Inc., September 1974. (NTIS No. AD A001110) Brown, R.V., Peterson, C.R., Shawcross, W.H., and Ulvila, J.W. Decision analysis as an element in an operational decision aiding _system (Phase II) (Technical Report 75-13). McLean, VA: Decisions and Designs, Inc., November 1975. (NTIS No. AD A018109) Cohen, M.S. Decision support for attack submarine cocrmumders: Target range pooling and attack planning (Technical Report 82-1). Falls Church, VA: Decision Science Consortium, Inc., April 1982. (Confidential) Cohen, M.S. Research on cognitive collaboration between pe~sons and computers. In Proceedings of the 6th HIT/ONR Workshop on~ Systems, 1984. Cohen, M.S., and Brown, R.V., 1981 Cohen, M.S., and Brown, R.V. Decision support for attack submarine commanders (technical Report 80-11). Falls Church, VA: Decision Science Consortium, Inc., October 1980. Donnell, M.L., & Ulvila, J.W. Decision analysis of advanced scout helicopter candidates (Final Technical Report PR 80-1-307). McLean, VA: Decisions and Designs, Inc., February, 1980. (NTIS No. AD A081483) Everett, H., III. Generalized Lagrange multiplier method for solving problems of optimum allocation of resources. Operations Research, 1967, 399-417. Peterson, C.R., Randall, L.S., Shawcross, W.H., & Ulvila, J.W. Decision analysis as an element in an operational decision aiding system (Phase III) (Technical Report 76-11). McLean, VA: Decisions and Designs, Inc., October 1976. Ulvila, J.W., and Chinnis, J.O., Jr. A method for allocating resources across R&D programs. Falls Church, VA: Decision Science Consortium, Inc., 1983.
PAGE 110
DECISION ANALYSIS AS A TOOL OF CONGRESS May 10, 1985 Prepared for: Dr. Fred Wood Office of Technology Assessment United States Congress Prepared by: Rex V. Brown Decision Science Consortium, Inc. 7700 Leesburg Pike, Suite 421 Falls Church, Virginia 22043 (703)790-0510 Under Contract No. 433-0315.1
PAGE 111
TABLE OF CONTENTS Page ABSTR,ACT. 1 1. INTR.ODUCTI ON . . . . . . . 2 1. 1 Scope ................ ,-. . . . . . . . . . . 2 l. 2 Source of Material in Thia Report. . . . . . . . 2 2 CURR.EN'T snros . . . . . . . . . . . . . 4 2.1 Broad Trenda of Computerized Information Technology ............. 4 2.2 Use of DAS: An Anti-Crime Funding Example ...................... 5 3.. POTE!fTI,AL A:IPLICATIONS. . . . . . . . 9 3.1 Uaea in Ad Hoc Legislation ...................................... 9 3.2 Uses in Recurring Legislation ................................... 11 3.3 Congress u a Source of Value Input to Executive Branch PDA ..... 12 4. LIMITATIONS ON THE USE OF DECISION ANALYSIS AND SUPPORT .............. 15 4.1 The Problem of Communication .................................... 15 4. 2 Other Limi tationa. . . . . . . . . . . . 15 5 RECO~ATIONS . . . . . . . . . . 17 REFmEl'ICES. . . . . . . 18 BlST CDPY AVMLAiLL
PAGE 112
ABSTRACT Congress is currently well supplied with computerized and other resouces for obtaining available information for decision making, and for developing certain types of new information such as the economic impacts of options. \Jhat is lacking is struccured assistance in assimilating this information along with other less formal judgments and usessments, in the process of making a sound and defensible decision. An appropriate technology exists for ~ystematically incorporating objective and controversial considerations (such as impact on national interests and importance tradeoffs between them), in the fora of decision analysis and support (DAS) tools, and especially personalized decision analysis (PDA). However, it appears not to have been applied to con gressional purposes, except for one pilot project. An i.mporunt potential limitation on the value of DAS tools is the extent to .. ~;.ich their nee can be colllllNni.cated briefly and clearly to a eypical congressman/suffer, and the extent to which this essence can, in turn, be ex plained by the con~ressman to the public. This consideration has often stoud in the vay of more effective use of other analytic tools, such as eeonometric models. In many ways, personalized decision analysis models are easier to communicate, because they track relatively closely the normal processse~ of intelligent reasoning. Nevertheless, there is still much work to be done on making the communication process more effective. Promising new congressional initiatives might include: CRS developing customized decision models to parallel issue briefs and putting them at the disposal of congressional staffs; CBO developing rep4atable budget evaluation tools that build on non economic impacts and tradeoffs becween them into existing aids. -1.. BES J COPY AVAILA~lL
PAGE 113
1. INTRODUCTION 1.1 Scope The prl.lDary purpose of this task report is to explore existing and potential applications of computer-aided decision analysis and support (DAS) and particularly personalized decision analysis (PDA) for the direct purposes of Congress. This is a small but significant subset of all congressional uses of computers, a topic which has been explored by Frantzich (1985) under another OTA project. tJhat is distinctive about our effort is that we are limiting our attention to those uses of computers which involve weighing the pros and cons of decisions, particularly using concepts and tools of personalized decision analysis. This excludes from our consideration, for example, the use of computers to retrieve data, for which there has been an explosive demand over the past decade. As Frantzich has pointed out, the necessary institutional arrangements for getting congressional computer systems up and running has largely been achieved and there is every indication that productive congressional use of them will proceed, and on a large scale, without any special further initiative. On the other hand, the higher order process of turning data into usable information is much less well established; and the organization of that information and merging it with relevant personal judgment has hardly benefitted at all from computerized information technology (IT). The realization of the potential of computerized IT in this use (such as it is) may require more thorough deliberation and stimulation. 1.2 source of Material in This Report The material on which this report is based was obtained from three distinct sources: the contractor's own professional experience in developing decision analysis and support tools, and applying them to issues of interest to Congress and the executive branch; relevant documents (in particular, Frantzich, 1985); and interviews with informed parties. Particular attention was p&id to a single representative issue, energy regulation. The interviews included: Dr. Ed Zschau (a congressman expert in this information technology); Dr. Marvin Esch (an ex-congressman trained in social -2-/ I I
PAGE 114
science); Mr. Ed Mason and Dr. Robert Civiak (Congressional Research Service); Mr. Paul Gilman (Staff, ~~nate Sub-Committee on Energy Regulation; Mr. John Swearingen, Mr. Robert Hicks and Mr. Jim Eastep (Senate Technical Service); Mr. Fred Tathwell (Department of Energy, Congressional Liaison Office); Mr. Ed Fay (Nuclear Regulatory Commission, Congressional Liaison Office); Dr. Steve Frantzich (U.S. Naval Academy, Political Science Department); Mr. Bob Harris and Dr. Steve Zeller (Congressional Budget Office). ~ile our conversations with these gentlemen provided many valuable insights, the point of view expressed here is the author's alone. The National Science Foundation Policy Sciences Section provided some support for this enquiry. -3-1 ( 7/
PAGE 115
2. CURRENT STATUS 2.1 Broad Trends of Computerized Information Technology Over the past decade, Congress has become plentifully supplied with computers, both mainframe and personal, and with information technology more generally. The status of current use has been well-described and appraised elsewhere (Frantzich, 1975). Institutionally, Congress has ready access to abundant data and to numerous data base systems to facilitate their effective and efficient sifting. The Congressional Research Service, the ~ongressional Budget Office, the General Accounting Office, and the Office of Technology Assessment and support groups attached to the House and to the Senate, provide a highly effective service, which puts available information and extensive expertise at the service of Congress. They also develop special-purpose information (for example, econometric mod~ls provided by CBO). However, these information service developments have gone along with other developments which point up a need, now pressing, for a different kind of technical support to Congress to help focus information on specific decisions. While information has expanded, the time of congressmen and their staffs to absorb it and digest its implications has become much scarcer. More specifically, the issues which any given congressman has to address have become more numerous, for example through the proliferation of subcommittees, and technically more complex. In addition, there have been political trends to make Congress a more active partner in government decision making. As a result, Congressmen have to make up their own minds or critically evaluate more government choices that, in earlier times, might have been left more passively to the discretion of the executive branch. As a result, there is a common sentiment among congressmen and their staffs (certainly among the eight or so we talked to), that they would value help in organizing information into a form where. its implications for a particular choice at hand are made clear, provided that it takes no more of their own time to do so. This, of course, is the distinctive function for which decision support and analysis tools have been designed, and personalized decision analysis, in particular. Its essence is to take a particular choice (say, whether to support funding for the MX missile or aid to the Contras), and to structure all the considerations which may affect its evaluation. The con-
PAGE 116
siderations will include not only factual information and proj e".Jh:,n:; .b;3,~ed_ o~ expert studies, but also impacts ~hich can only be assessed judgmentally (and usually controversially) and value judgments, e.g., the tradeoffs between very different impacts (like defense preparedness, the nation's health, and environmental protection). It can certainly be argued that such subjectivities and intangibles are best left to the individual congressman or staffs to weigh informally. Moreover, it can also be argued that, when a congressman is called upon to justify his actions before his constituents and others, simple words, untainted by reference to computerized analysis, are likely to be most effective. Nonetheless, burgeoning computer-aided decision analysis and support may help fill the gap between information and decision. It is being used on an increasingly widespread and effective scale by the executive branch (see Task 3 report). Many of the motivations relevant to the executive branch seem to apply equally well to Congress: to make decisions more sound; co focus information and judgment efficiently; to justify recommendations ambiguously; to pinpoint technical views on which more deliberation and information is needed. 2.2 Use of DAS: An Anti-Crime Funding Example To the best of our knowledge, the tools of personalized decis.ion analysis or other forms of decision support and analysis, are not being used in any significant scale to the direct service of Congress. However, at least one pilot exercise has been attempted which illustrates the approach and hints at its potential value for Congress. In 1979, our company conducted a brief, low-budget decision analysis for the Law Enforcement Assistance Administration (LEAA) to evaluate congressional funding options in the community anti-crime program which LEAA administered. It consisted essentially of a small program on a desk top graphic computer, which evaluated the main congressional ~ptions in terms of a dozen or so impacts (like crime and the fear of crime) and the value tradeoffs between them. It was demonstrated to the staffs of interested congressional subcommittees. The basic format of the demonstration was to take half an hour to present the evaluation model with the most plausible inputs we were able to obtain; and then to spend another half an hour for the staffers to try out their own inputs and play "what if" games. -s' ( J. I r l.'.IIIJ'J I, ,j {f. ,., i.,t ,iJ i l'I, ... 11.!;:(;1 ,.
PAGE 117
The follcwing account. of the exercise is taken from our .re.port: to LE.A.A (Brown, et al., 1980). With the advice of LEAA staff, a decision of topical interest was selected (to test the usefulness of this type of decision aid): the level of government funding for community anti-crime programs for fiscal 1980 and beyond. The decision would emerge from a complex pr~cess culminating in Congressional action in Fall, 1979. Several parties to that process were expected to be interested in a systematic evaluation of options. They included LEAA's Office of Community Anti-Crime Programs (OCAP) that would administer the programs, and certain key Congressional committees. Three funding levels were chosen for special attention: $10 million, $20 million, and $40 million a year. The first and last corresponded approximately to to positions advocated at that ti.me by the Senate and House Judiciary Committees respectively. Each level was assumed to remain constant from fiscal 1980 through 1984, the period for which LEAA was expected to be reauthorized. DSC's analysis measured the cost-effectiveness of the three funding levels (compared with no program at all), and was based on available data about the past and future impacts of CAC programs and about national priorities. That data consisted primarily of informed opinions and a few case studies, since the results of extensive field evaluations were not yet available. A "multiattribute utility" model was constructed, within which a dozen measures of program effectiveness were identified and weighted by importance. They included measures of social effectiveness such as: crime and fear of crime; level of community activity and other social processes; underlying causes of crime; relations between public and police; quality of life; institutionalization of funding and people resources; and crime prevention know-how. They also included measures of special interest to particular actors in the decision process, such as the goodwill of communities affected and of other groups. Each option was assessed on each measure according to the fraction of a hypothetical "ideal". impact it would achieve by 1984. A combined index of effectiveness was calculated for each option; translated into dollar worth (based on direct judgments of social value and on cost savings to the criminal justice system); and compared with the actual cost of the option--$10 million, $20 million, or $40 million a year, as the case may be. The resulting DSC analysis suggested that by mid-1979 the OCAP programs had almost justified their cost after two years of operation. They would be worth $50 million a year by 1984 if continued at $10 million a year; rising to $75 million by 1984 for a $20 million cost; and $105 million for $40 million cost. In 1984, therefore, the $40 million level would generate the greatest surplus of effectiveness over cost. However, its cost-effectiveness is poor in earlier years and the $20 million level shows highest cumulative net value over the five year period. Net value, as a difference, is a reasonable criterion if the total budget is variable. If it is fixed, cost-effectiveness ratio becomes more relevant and would
PAGE 118
favor the $10 million level. Thus, the final preference depends both on budgetary constraints and on the time horizon of interest. Such analyses are, of course, completely dependent on the particular impact assessments and value tradeoffs used as inputs. These are by no means definitive and were synthesized by DSC staff from judgments elicited from several experts (researchers, program ev~luators, program personnel) and from anecdotal evidence from field evaluation studies in progress. To test the effect of alternative assessments and perspectives, the analysis was computerized to facilitate sensitivity analysis. It showed, for example. that in terms of annual effectiveness the $40 million option would still be preferred in 1984, provided that an "ideal" program would be worth at least $200 million a year (where DSC had assessed it at $300 million). To validate the realism of the analysis and its formulation, DSC presented it to various key actors in the executive and legislative decision processes, such as the head of OCAP at LEAA, and the staff of selected Congressional committees. The latter included: the Senate Judiciary Committee and its Subcommittee on Crime; the Senate Appropriations Subcommittee on the Judiciary; the House Judiciary Subcommittee on Crime; and the House Appropriations Subcommittee on the Judiciary. Those briefed found that the analysis was a useful device for clarifying their positions even with the tentative judgmental inputs used. Furthermore, several felt that more extensive studies of this kind, grounded in major empirical evaluation studies, could provide an agency like LEAA with a valuable vehicle for supporting recommendations to Congress. Analyses like the present one could also help determine the form and scale of larger studies. Following the completion of this study, in Fall 1979, Congress in fact authorized a funding level of $25 million for fiscal 1980. The analysis described here used a relatively low level of effort to show how decision analysis can focus evaluation efforts to assist a particular programmatic decision. It demonstrated that a formal analysis of a significant decision, where available data are largely subjective, can be responsive to the needs and interests of key players in the decision process. In this case, those players included line managers and staff within LEAA and members of all major Congressional committees involved. Among the latter, several senior staffers expressed the view that a decision-analytic procedure such as that to which we exposed them would have a valuable role to play in higher level decision making. For example, one senior staffer for the Senate Committee on the Judiciary felt it might be a suitable framework fo~ analyzing and communicating decisions on the total funding of LEAA and the partition of that funding among major programs. Much of the same view was expressed by staffers on the House and Senate Appropriations Committees. Since Congressional committees have to consider a gre.at many issues in a short period of time, they found it helpful to have recommendations presented to them in a compact form such as decision analysis provides, particularly since it is capable of explicating the supporting arguments in a systematic and structured fashion. In cases such as the one under consideration, where there is substantial controversy and conflict of opinion, it was valuable to be able to rest the implications of different assumptions. This is a major feature of the kind of analysis performed. -7-I I, ")
PAGE 119
Prompted by this exercise, a senior staffer on the Senate Subcommittee on Crime suggested that decision analysi.s might be used to specify and/or implement "Sunset" criteria to evaluate major programs at the time they become due for reauthorization (such as LEAA was at the time of the study). This might involve: specifying a multiattribute utility func :ion with which to evaluate a program at the time a program is initially authorized; specifying threshold performance on that function to the responsible administrators; periodically tracking the performance of the program, thereby providing feedback to the administrator; and ultimately determining the program's .fate. The scores on arguments in the utility function (say, various social and economic objectives) might be assessed using empirical and analytic techniques such as those developed in DSC's overall CAC evaluation effort (Snapper and Seaver, 1978; 1980). -a-
PAGE 120
3. POTENTIAL APPLICATIONS There are a number of congressional functions and motivations that DAS and PDA might serve. The functions include: authorizing legislation, budget determination, and agency oversight. Motivations for a congressman may include: making sound decisions for the public, making the executive branch operate better, looking good to his constituency, getting his own way on particular issues. 3.1 Uses in Ad Hoc Le~islation What we learned from the pilot anti-crime funding exercise is primarily that there is a potential demand by some congressional staff at least, for a computerized aid that can help them comprehensively and compactly use best available information and judgment on a decision issue, and also splice in their own assessments and judgments. In this particular case, the view of the staffers was that this particular issue was a sufficiently small part of their total concerns, that they would not want to spend much more than an hour deliberating on it, and this seemed to be a useful way to spend that hour. In the five years since then, the technology of personalized decision analysis has advanced substantially, particularly in the user engineering of communication (but it still has a long way to go). The logic and substance of the model must be sound, of course, but it will not be successfully adopted by busy congressmen and staffers unless it 1.\i:iplays very clearly what it has to say and is very easy to manipulate. Much of this decision analysis function is already being performed_informally by the Congressional Research Service and othe~s. Through issue briefs and personal communication, CRS can, and does, evaluate the pros and cons of an issue or choice at about the right level of detail and with much the same content that a computer aided PDA model of the type we have discussed might have. It does, however, lack some features which a computerized decision aid would supply. -9
PAGE 121
One is assistance in the final decision making stage. It is not normally CRS' role to argue for a particular decision--that would probably be viewed as usurping an essential role of Congress--but rather of presenting information that would facilitate that decision. A PDA decision aid provides a vehicle for congressmen and staffers to incorporate their own judgment where they please (for example, on the relative importance of impacts) and shows them the action implications of those judgments combined with whatever information and assessments they do noc choose to second guess. It also, provides the congressman/staffer with the ability to keep, in a sense, "two sets of books." He can run the model once to reflect only society's values, such as crime rate and cost to the taxpayer, and this would be the argument he would feel most comfortable acknowledging publicly. He may also rerun the analysis, taking also into account some more private considerations, such as political goodwill and electoral security. More than one staffer noted this as an attractive feature in the anti-crime funding example discussed above. Being able to i-un the model in this mode in the privacy of a committee office has some obvious advantages over having to rely on an outside expert to test out alternative inputs (which might be sensitive or embarrassing if leaked). The strength of such a "macro model" (so ca~led because it deals with decision considerations at a highly summarized level) is also its weakness. It is focused on a particular choice, which makes it very valuable for that choice, but by the same token, it cannot be reused (at least not in exactly the same form) for other decisions that may arise. Being customized, it has only one setting in which to recover its cost. On the other hand, general purpose modeling technology and modules can be (and are being) developed from which customized macro models can be created at low cost. Perfectly usable models have been constructed in half a day (to structure the existing knowledge and judgment of a small group of decision makers and experts), and a man-month of effort by a decision analyst will be ample for a wide variety of other situations. The most natural institutional arrangement for offering a macro model PDA service to Congress might be for CRS todevelop such models as an adjunct to the preparation of issue briefs. The models could be used to help develop material for the issue briefs, and they could also be made available to con--10BEST COPY AVAiLAWli_
PAGE 122
gressional staffs (probably in the form of a disk to be used on a committee's personal computer). CRS might also take responsibility for putting on seminars for congressmen/staffs as part of its "institutes" program. A more extensive three-day course would give staffers the skills they need to work comfortably "'ith, and possibly construct their own macro models. (Analogous courses are already being offered to senior staff in the executive branch.) The role of the executive branch in "feeding" inputto such models deserves some discussion, since in virtually all cases, some part of the executive branch will have a keen interest in the decision in question and it wUl have available much relevant expertise and other resources. As we suggested in our Task 3 report, the executive branch can be encouraged to support its requests and recommendations in decision analytic terms, which makes it easier for Congress to identify and second-guess critical. as sump ti, ns and assessments. However, this will probably not supplant the need for Congress to develop its own models. Since the executive branch is an interested party in general, it cannot necessarily be relied upon to produce models which are most responsive to the needs of Congress. I~deed, the legislative branch can be expected to be resistant to congressional use of effective decision aids which might increase the control which the legislature wields. (This concern was confirmed in discussion with the ex-administrator of a major executive agency.) However, it would certainly seem reasonable to elicit inputs from the executive branch for congressional decision models. At the very least, the executive branch could be given the opportunity to challenge or suggest revisions to whatever predictive, value, or other inputs have been proposed for the congressional model. 3.2 Uses in Recurring Legislation There are certain congressional decisions, notably those to do with budget allocation, which recur often enough, in substantially the same form, that a standardized decision aid can be seriously considered. One form of decision analysis commonly used in the executive branch (see reports on Tasks 3 & 4) is a resource allocation tool, which could be readily adapted to the needs of budgeting. -11r1fl
PAGE 123
As described in greater detail in our Task 4 report, what is distinctive about this aid (as contrasted with other commonly used budget analysis programs) is that it explicitly addresses the linkage between candidate budgets and the dimensions of national interest they impact (such as defense preparedness, nation's health, etc.), and it also addresses the relative importance of these impacts. This requires difficult and sensitive assessments to be make (which in any case have to be made, at.least implicitly), but it allows a total budget, or a specific variation within it, to be evaluated by a single number for comparison purposes. Such a budget evaluation model could be developed and maintained by the Congressional Budget Office for use by Budget and other committee staffs. We understand that currently CBO does not take budget analysis further than assessing the economic impacts (e.g., unemployment, federal outlays, and interest on the National Debt) of budget alternatives. FDA models would adapt such assessments as input and combine them with the non-economic impacts of a given budget, using available technical and value judgments. 3.3 Congress as a Source of Value Input to Executive Branch PDA A potential revolutionary use of PDA by Congress, if not very likely to come about in the near future, is as a conduit of societal value judgments, to be used in decision analysis by the executive branch. (In our Task 3 report, we discuss the case for Congress fostering the use of PDA in the executive branch.) Government policy decisions, like any other decisions, involve consideration of two types of issues: What do we know and what do we want? When handled by PDA, these are represented by the measures of probability and utility, respectively. It has been argued by at least one prominent decision analyst (Howard, 1975) that although it may be proper for administrators and technicians to contribute the former (e.g., predicting the impact of alternative options), it is for society as a whole, who are the supposed beneficiaries of social policy, to determine the latter (e.g., value tradeoffs between different types of consequence). Howard argues that Congress is the appropriate spokesman for society on such issues. In particular, Congress could specify (e.g., by legislation) a set of quantified value judgments to which the executive branch would be expected to conform in making its decisions. A simple -12-1 ~/
PAGE 124
example would be to specify value of a human life for regulatory purposes. More specifically, such a "value statute" might specify that it was inappropriate to incur costs up to $1 million in order to save one "statistical" life through the imposition of a regulatory requirement (such as a safety feature on a nuclear reactor). At its most ambitious, this would be a massive technical, administrative and political task. The principle coufd, of course, be established quite modestly by issuing very limited guidance--say, specifying the value of a statistical life, or by-describing criteria in qualitative terms (as is, in fact, already done in a piece-meal way as part of many existing authorizing statutes). Appealing though this idea might be to the social philosopher, we do not underestimate the immense political impediments to put even a modest version of this idea into effect. Private conversations wit~ t:wo unusually qualified players in the political process were not encour~ging. A House Representative, who has taught decision analysis at major schools, expressed__the~view that a congressman would run a serious risk of political embarrassment by coming out in favor of any particular tradeoff, (such as $1 million for a human life) or even by taking the position that such things can be quantified at all. On the other hand, he saw no reason why Congress would not go along with such value coefficients being set by a designated arm of the executive branch, as part of its executive function. There are, in fact, piecemeal precedents. l'he FAA, for example, specifies the value of the order of $700,000 per life and the Nuclear Regulatory Commission does something similar by specifying (NUREG 0880) $1,000 per person-rem averted. Title 2 of the Ritter Bill (possibly to pass into law in the 99th session) calls for the setting up of a body under the aegis of the National Academy of Sciences to determine questions of fact in the assessments of risk for regulatory purposes. Possibly a comparable body could be set up to make value .determinations, with more or less loose guidance from Congress. An executive branch perspective was provided, in private communication, by the ex-administrator of a regulatory agency, who is well familiar with decision analytic concepts (he was a member of the prestigious National Academy of Sciences Committee on Risk Analysis and Decision Making). He felt that even if politically feasible, it would be inappropriate for Congress to make its -13-I (__, L
PAGE 125
guidance so specific (and, therefore, inflexible) as to provide quantitative value tradeoffs, like the value of a life. His feeling was that this would amount to more specific direction by Congress to the executive branch than the Constitution prescribes. The appropriate degree of congressional input on societal values he believes is provided by current practice. This consis:s largely of qualitative statements of what factors should be considered and with what relative importance in various enabling statutes (for example, authorizing different programs at EPA) or by the exercise of congressional oversight, which provides feedback on whether agencies and programs are, in fact, reflecting appropriate value judgments. More generally, we have detected some reluctance in executive agencies to ac cept more direct congressional involvement in the decision making processes of the executive branch. -14.-, Bf.Sl CDPY AVAll~iLL I .... --,
PAGE 126
4. LIMITATIONS ON THE USE OF DECISION ANALYSIS AND SUPPORT 4.1 The Problem of Communication The key to successful use of decision analysis by Congress is com:n1.1nication. There seems to be little doubt that many members of Congress and their staffs would welcome the benefits of tools which help them and others structure the information and judgment that go into decisions. These are mainly the benefits of focusing attention economically on the essentials of an issue: of having a vehicle for organizing and clarifying the implications of multiple points of view; and of justifying the soundness, or at least the reuonableness, of decisions to the outside. There may also be significant value to the public in reducing the role of irrelevancies and rhetoric in favor of essentials and rationality in legislative decision processes. However, none of th benefits can be realized ~less the tools are easy and tranaparent to use and understand. They have a long way to go in this respect, though it may be easier to make them comprehensible than other types of modeling and simulation which are intrinsically more foreign to everyday ways of thinking. The problem of effective communication--both to Congress and then from Congress to the public--may be the principal reason why Congress looka to support agencies and others primarily for facts and, much less co1m1only. for analysis. The analysis takes more time to absorb; and even when it has been absorbed, it cannot be effectively communicated to the public in twenty second television clips or even in more leisurely media. Up until now, most development effort in the field of decision analysis And support bu been on making them technically sound. It is now time, we believe, for aignificant effort to be devoted to improving the communication properties of th tools. both generally and in the specific case of Congress. which bu its specific needa. 4.2 Other timitacions Required in parallel with better communicability of the tools themselves is enhanced familiarity. and therefore comfort, with DAS techniques by Congress and ataff. (Ultimately also by the general public, but that is a topic more properly reviewed as a topic in public education). Some of this may take -15-/ /. J I
PAGE 127
place spontat1eously as Congress and its staff become increasingly peopled with newer generations of technocrat3. In the shorter run, short courses and seminars, possibly sponsored by CRS, would make a significan~ contribution to the communicability of DAS. A further potential limitation on the increased use of DAS is thac it may not lend itself readily to being incorporated into the instituticn~l processes of Congress. This issue is discussed at a more general level in Brown (19830. Members of Congress may, for example, be resistant to any but the most prhe.te use of PDA, on the grounds that it limits their flexibility to decide on the basis of considerations that they would not care to acknowledge publicly. (This, of course, may be an advantage from the public's point of view!) It is possible that the introduction of comprehensive decision aids which purport to capture explicitly all the pros and cons of a decision, would make for greater consistency and public responsiveness in. the decisions that get made (and not just in the process of making them). Much of this effort would be lost if the use of these tools were hidden from public view, but this may be a necessary condition for their widespread adoption by Congress. -16c.--/ l .)
PAGE 128
5. RECOMMENDATIONS 1. Familiarize members of Congress and staffs with information on the capabilities and limitai:ions of personalized decision analysis a.-:.1d other computer-aided decision support and analysis. 2. Encourage the Congressional Research Service to offer a macro modeling PDA resource and user training for congressional committees and members. 3. Encourage the House and Senate Budget Committees and the Congressional Budget Office to consider developing a PDA-based budget evaluation tool. 4. Spvnsor research and development on the technology of decision aids for congressional purposes, especially those aspects which simplify communication with the user. 5. Commission a study on the proper role and development of decision aids for Congress, with particular attention to institutional factors (see Brown, 1983). 6. Explore what role Congress should play in providing input on social values for decision analysis carried out within the executive branch. 7. Consider legislation which fosters use of DAS by the. executive branch (e.g., which calls for "total impact statements," analogous to the NEPA requirement for environmental impact statements, or which gives Congressional authority of Executive Order 12291 for certain classes of decisions). -17-
PAGE 129
REFERENCES Brown, R.V. Towards a prescriptive organization theory of decision aidsing for risk management: Phese I: Conceptual development (Technical Report No. 82-7). Falls Church, VA: Decision Science Consortium, Inc., November 1982. Brown, R.V., Seaver, D.A., & Bromage, R.C. An analysis of the community anti crime funding decision (Technical Report 80-2). Falls Church, VA: Decision Science Consortium, Inc., May, 1980. Frantzich, S.E. Congressional applications of information technology. Con gressional Data Associates, February 1985. Howard, R.A. Societal decision analysis. Hanagement Science, 197:i. -18-/ z_ 7
xml version 1.0 encoding UTF-8
REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd
INGEST IEID EMCL3EK42_WB02CB INGEST_TIME 2017-06-05T16:39:45Z PACKAGE AA00055709_00001
AGREEMENT_INFO ACCOUNT UF PROJECT UFDC
FILES
|