Department of Defense Budget Level Analysis of Science and Technology (DoD BLAST) Team

Size: px

Start display at page:

Download "Department of Defense Budget Level Analysis of Science and Technology (DoD BLAST) Team"

Russell Robertson
5 years ago
Views:

Department of Defense Budget Level Analysis of Science and Technology (DoD BLAST) Purpose of Brief Provide an overview of the DoD BLAST team s work on the Budget Exhibit Analysis Tool (BEAT).

1 Department of Defense Budget Level Analysis of Science and Technology (DoD BLAST) Purpose of Brief Provide an overview of the DoD BLAST team s work on the Budget Exhibit Analysis Tool (BEAT). The team will cover where we started, what we did, and recommendations for future work. SYST/OR 699 Master s Project Team Craig Jayson (SE) Josh Singh (SE) Phillip Sutton (OR) Sponsor Systems Planning and Analysis, Inc. (SPA) is a professional services contractor located in Alexandria, VA. Project This project is a continuation of a capstone project selected last year that supported categorizing and analyzing past, present, and future budget submissions. 1

2 Agenda Problem Scope Architecture Data & Visualization Way Forward Key s Background Problem Statement Stakeholders Scope Requirements Use Cases Structured Analysis BEAT Methodology BEAT Database Update Program Analysis Dashboard Project Analysis Dashboard Deliverables Third Offset Example Recommendations Conclusion Reporting Functions 2

Key Defenitions Budget Activity A category within each appropriation and fund account that identify the purposes, projects, or types of activities financed by the appropriation or fund Program

Presidential Budget The president submits a comprehensive budget request to Congress in early February which outlines the administration s policy and funding priorities and economic outlook for

3 Key Defenitions Budget Activity A category within each appropriation and fund account that identify the purposes, projects, or types of activities financed by the appropriation or fund Program Element The primary data element in the RDT&E budget and the smallest aggregation of resources used by the Office of the Secretary of Defense (OSD) for analysis. Presidential Budget The president submits a comprehensive budget request to Congress in early February which outlines the administration s policy and funding priorities and economic outlook for coming year. Source: OMB DoD Budget The annual appropriation of discretionary spending that goes to any military related expenditures. RDT&E RDT&E appropriations finance research, development, test and evaluation efforts performed by contractors and government installations to develop equipment, material, or computer application software. 3

Background (1 of 3) Figure: FY 2016 DoD and OCO budget (source: DoD.

Military Strategy (NMS), the Department of Defense (DoD) budget places emphasis on a number of items to

4 Background (1 of 3) Figure: FY 2016 DoD and OCO budget (source: DoD.Gov) Supplemental Funding Research, Development, Testing & Evaluation (RDT&E) makes up 11.8% of the total DoD budget in FY Figure: DoD Budget from in FY16 dollar (future years estimated) In support of the National Military Strategy (NMS), the Department of Defense (DoD) budget places emphasis on a number of items to include the need to invest in future capabilities to ensure the U.S. remains a modernized, technologically superior gloabal force that is able to project power across the globe. This funding is found in RDT&E appropriation. 4

Background (2 of 3) Budget Activity BA-1 Basic Research BA-2 General Purpose Force DOD Programs (Found in DoD RDT&E) BA-3 Advance Technology Development PE Number Key 06 DoD Program 3 Budget Activity

5 Background (2 of 3) Budget Activity BA-1 Basic Research BA-2 General Purpose Force DOD Programs (Found in DoD RDT&E) BA-3 Advance Technology Development PE Number Key 06 DoD Program 3 Budget Activity 28 Serial Number Assigned 6 Equipment / Activity Type E- Code for Service / Agency 01 Strategic Forces 02 General Purpose Force 03 Intelligence and Communication 04 Mobility Forces 05 Guard and Reserve Forces 06 Research and Development 07 Central Supply and Maintenance 08 Training, Medical, and Other 09 Administrative and Associated Activities 10 Support of Other Nations 11 Special Operations Forces BA-4 BA-5 BA-6 BA-7 PE Number E F Advance Component and Prototypes System Development and Demonstration RDT&E Management Support Operational System Development PE Name Advanced Aerospace Systems Aerospace Technology Dev/Demo Last Year: System Planning and Analysis teamed with students in the Master s of Systems Engineering and Operations Research to initiate Phase 1 of a multi-year effort to develop a Decision Management System that assesses trends in Science & Technology (S&T) investing within DoD. 5

Background (3 of 3) PE: 0603286E Advanced Aerospace Systems FY17 Total $182,327,000 PE / BLI Program Element / Budget Line Item (BLI) Title Include in TOA FY 2015 (Base &

173,631 0 173,631 182,327 0 182,327 U 0603287E Space Programs and Technology Y 172,504 126,692 0 126,692 175,240 0 175,240 U 0603739E Advanced Electronics Technologies Y

Warfare Technology Y 350,323 425,861 0 425,861 428,894 0 428,894 U 0603767E Sensor Technology Y 283,905 240,127 0 240,127 241,288 0 241,288 U BEAT Version 1.

6 Background (3 of 3) PE: E Advanced Aerospace Systems FY17 Total $182,327,000 PE / BLI Program Element / Budget Line Item (BLI) Title Include in TOA FY 2015 (Base & OCO) FY 2016 Base Enacted FY 2016 OCO Enacted FY 2016 Total Enacted FY 2017 Base FY 2017 OCO FY 2017 Total Classification E Advanced Aerospace Systems Y 123, , , , ,327 U E Space Programs and Technology Y 172, , , , ,240 U E Advanced Electronics Technologies Y 81,119 76, ,021 49, ,807 U E Command, Control and Communications Systems Y 229, , , , ,081 U E Network-Centric Warfare Technology Y 350, , , , ,894 U E Sensor Technology Y 283, , , , ,288 U BEAT Version 1.0 Budget Exhibit Analysis Tool (BEAT) is a Decision Support Tool that that allows users to visualize RDT&E data based on selective queries in order to conduct follow-on exploratory analysis. 6 6

Problem Statement There is no known method or available tool to digest and depict multiple budget cycles (POM) to facilitate the analysis and characterization of past, current, and future budget

DoD Budget data is disbursed into many separate documents/locations Manual formatting of data standard of practice No efficient way exists to isolate and visualize data to meet user-specified

7 Problem Statement There is no known method or available tool to digest and depict multiple budget cycles (POM) to facilitate the analysis and characterization of past, current, and future budget information. The Fall 2016 Capstone team was successful in aggregating budget data into a database and creating an Excel tool that performed such analysis at the PE level. DoD Budget data is disbursed into many separate documents/locations Manual formatting of data standard of practice No efficient way exists to isolate and visualize data to meet user-specified criteria Understanding that there is important information available at the more granular project level, the BEAT has the capacity to provide further information and insights, but currently that capability has not been realized. 7

8 Industry: Private entities whose interests and level of involvement with the DoD vary significantly Stakeholders FFRDCs/UARC Research partnerships in which the DoD is the sponsor of a research facility which is under the administration of either a corporate entity (FFRDCs) or a University. Community of Interest: A network of technical groups that serve as a lasting structure to integrate with experience and technical expertise. DoD: This includes the Services and the S&T community. They receive the tech. and support and collaborate with multiple stakeholders. OSD: Primary defense policy advisor to the President and responsible for formulating and executing policy through the DoD. Figure: Stakeholders include publishers, resourcing, and recipients (source: gmuseorspa.wordpress.com) President/Congress: The DoD submits an annual budget request to the President for review. The President then submits a budget request to Congress. This request is reviewed and subjected to a process of revisions before it is returned to the President, at which point it is either signed into law or vetoed. 8

Scope/Requirements The project team for Spring 2017 Capstone team will focus on taking the PEs and continuing to add to the database by including all project level data (excludes classified data).

Highest Level Requirements MR1: The BEAT system shall be able to ingest project level information and data from DoD documents that summarize PEs.

9 Scope/Requirements The project team for Spring 2017 Capstone team will focus on taking the PEs and continuing to add to the database by including all project level data (excludes classified data). Time Frame of Study ( ) this covers FY 16 PB and POM. Highest Level Requirements MR1: The BEAT system shall be able to ingest project level information and data from DoD documents that summarize PEs. MR2: The BEAT system shall ingest project level description information for each project. MR3: The BEAT system shall ingest five future years of expected project level data. MR4: The BEAT system shall include accomplishment level data for specific projects. Link to Requirements Diagram MR5: The BEAT system shall visualize RDT&E, specifically project level data, over a pre-determined period. MR6: The BEAT system shall be able to display Program Budget (PB) new program/project/accomplishments have started anew or ended. 9

10 Structural Analysis A Process Oriented Approach of BEAT Use Cases (Two Primary) Activity Modeling IDEF0 (A-0, A0, A5 diagrams) Data Modeling Entity Relationship Diagram Rule Modeling Decision Table Dynamic Modeling Sequence Diagrams Use Cases Source: SYST 520 course at 10

R-1 document contains budget information for all RDT&E

For each FY, the document lists the budget for the previous

R2 Data: The R-2 document is specific to each program,

11 BEAT Version 1.0 (Fall 2016 Capstone Project) R1 Data R2 Output R1 Data: The R-1 document contains budget information for all RDT&E programs undertaken by the branches of the DoD. For each FY, the document lists the budget for the previous two FY, as well as the budget for the current FY. R2 Data: The R-2 document is specific to each program, identified by a PE number in the R-1. The R-2 serves as the item justification for each program, as the proposed budget for the next four FYs. 12

Database Update Presidential Budget XML file parsing was migrated from Java to VBA and the database was moved from a MySQL database to MS Access The BEAT has been linked to the Access Database

12 Database Update Presidential Budget XML file parsing was migrated from Java to VBA and the database was moved from a MySQL database to MS Access The BEAT has been linked to the Access Database and two new worksheets have been created to query the data The current data set goes back to 2011 and contains 4366 Program Element Records across the DoD comprising of 9388 Project Records 12

13 BEAT Version 2.0 Methodology Download PB data from DTIC (Manual Process) Open BEAT and Select Upload new R2 files BEAT (Excel) Opens Access and runs the XML parsing Code Input Cells for scoping data Visualization User conducts exploratory analysis and informs management Descriptive Information Visualization Tool within BEAT that allows user to graphically display data based on inputs Excel Sheets Update Existing 13 Queries and Connections

14 Program/Project Analysis Dashboard BEAT Program and Project Analysis Dashboards allows the User to analyze a specific program or projects funding trends and relate those variations to the notes from the budgets all in a single environment 14

15 Program Analysis Dashboard (1 of 6) The User can narrow the organizations and programs by selecting a (or all) Budget Activity of Interest From there, the User selects among the organizations and agencies that have programs under the selected Budget Activity The Select Program options are then updated and the user can select a Program belonging to the organization or agency by name of Program Number 15

Program Analysis Dashboard (2 of 6) Additionally, the User can search for a program using the Program Search function The User can filter all available Programs by Service/Agency,

16 Program Analysis Dashboard (2 of 6) Additionally, the User can search for a program using the Program Search function The User can filter all available Programs by Service/Agency, Budget Activity, or Keyword Searches Once a Program has been identified, the User can select that Program and the Program Analysis Dashboard will update to the selected Program 16

Program Analysis Dashboard (3 of 6) Information about the Program selected is populated from the database: Program Description Earliest/Latest PB Years Available

17 Program Analysis Dashboard (3 of 6) Information about the Program selected is populated from the database: Program Description Earliest/Latest PB Years Available Program Notes Per PB Year Additionally there is a button below the Program Description to Select the current Program for analysis of its Projects, described later 17

18 Program Analysis Dashboard (4 of 6) Chart plots funding across the various Presidential Budget years providing the User a visual representation of the changes in funding across the budgets Includes the data in a table below the chart to provide values for data 18

Program Analysis Dashboard (5 of 6) Summary statistics are generated to provide metrics describing the behavior of each Fiscal Years budget projections Statistics

19 Program Analysis Dashboard (5 of 6) Summary statistics are generated to provide metrics describing the behavior of each Fiscal Years budget projections Statistics include Mean, Standard Deviation, Variance, Min, Median, Max, and Range A stable budget will have minimal Standard Deviations and Variances between budget submissions 19

20 Program Analysis Dashboard (6 of 6) A heat map highlights the variance in the deltas between Presidential Budgets (PB) The N/A is used to indicate that there is no delta to calculate: (In PB14 we have a prior year data point in FY 2013, however in PB15, there is no value for FY13, resulting in an #N/A) Note: The PB covers current FY and next FY and then 4 years out (tied to the POM) 20

21 Project Analysis Dashboard (1 of 4) The Project Analysis allows the User to investigate the different projects that belong to each program The down selecting process follows the same as the Program Analysis Dashboard, but adds a Project Selection by Project Name or Project Number 21

Project Analysis Dashboard (2 of 4) Similar to the Program Analysis Dashboard, the User can filter all available Projects by Service/Agency, Budget Activity, or Program

22 Project Analysis Dashboard (2 of 4) Similar to the Program Analysis Dashboard, the User can filter all available Projects by Service/Agency, Budget Activity, or Program and Project Keyword Searches Once a Project has been identified, the User can select that Project and the Project Analysis Dashboard will update to the selected Project 22

23 Project Analysis Dashboard (3 of 4) The Project Mission, (Earliest and Latest PB Data Years), and Notes are all imported for the select project, the same as the Program Analysis Dashboard Additionally, the tool provides the funding chart and heat map to highlight the deltas in funding 23

Project Analysis Dashboard (4 of 4) Removed from the Project Analysis Dashboard are the summary statistics In their stead is a pie chart that describes the all

24 Project Analysis Dashboard (4 of 4) Removed from the Project Analysis Dashboard are the summary statistics In their stead is a pie chart that describes the all the Projects under the governing Programs control This pie chart provides insight as to how the Project selected compares to the other Projects in that purview 24

Reporting Functions: Delta Analysis The BEAT also has an analytical feature that will analyze all Programs and Projects to identify Programs/Projects that have user defined budget variations as a

25 Reporting Functions: Delta Analysis The BEAT also has an analytical feature that will analyze all Programs and Projects to identify Programs/Projects that have user defined budget variations as a percentage of the budget Those Programs and Projects that exceed the delta threshold are logged in a report that can be reviewed and then explored further within the Program and Project Analysis Dashboards 25

Program and Project Level data to identify programs

26 Reporting Functions: ID New or Recently Ended Programs The BEAT will also parse through both Program and Project Level data to identify programs that are new to this years PB data or have ended in the last 3 years 26

27 BEAT v1.0 BEAT Comparison BEAT v2.0 JAVA XML Parsing MySQL Database Database not directly linked to BEAT Manual data extraction, transformation, and loading from MySQL into BEAT Visualizes R-1 and R-2 data at the Program Element level Requires the purchase of Smart Charts for Excel by DEVSCOPE for R-1 Visualizations Compares multiple R-2 Programs in a single Presidential Budget year VBA XML Parsing MS Access Database Access database directly linked to queries within BEAT (MS Excel) Queries automatically refresh after new data is uploaded into Access Program Analysis Dashboard allows for indepth analysis of a specific program Project Analysis Dashboard allows for indepth analysis of a specific project Reporting Functions Delta Analysis analyzes all data to identify programs and project with significant variation New/Ended Program Identification The updates to the database and XML parsing resulted from sponsor request during initial meetings 27

28 Deliverables BEAT continued refinement and project level data uploaded BEAT user interface improvement with ability to visualize new information BEAT user manual updated BEAT test case for final presentation (example of BEAT the JLTV) Written report and briefing summarizing results of the analysis Website with all products ( Listing of all sources Visualization Tool that identifies when projects begin and end Identification of future budget trends of projects that are nested with Office of Secretary of Defense s offset strategy (Next Slides will Provide Example) 28

29 Third Offset Strategy Technological superiority has been a staple of the DoD strategy since post WWII. This philosophy and follow-on investment has allowed U.S. to out pace its peers and remain at the forefront of battlefield innovation. However, since the current budget drawdown and the military mission shift to counterinsurgency (COIN), it is no longer safe to assume technological superiority. TOTAL RDT&E BUDGET AUTHORITY BY SERVICE (FY 2001 FY 2017) Counterinsurgency Drawdown Opportunity DoD investing $3.6 billion in FY 2017 and $18 billion over the FYDP (in current year dollars) in capabilities related to the Third Offset Strategy. Source: Third Offset Strategy (if maintained) provides significant opportunities for future RDT&E investment- should observe trends of increased investments in these capabilities. 29

30 Third Offset Strategy (PE and Project Examples of Tool) The President s FY2017 budget resulted in DoD investing in anti-access technology, undersea warfare capabilities, human-machine collaboration, precision munitions, and asymmetric capabilities (EW, Cyber, Space, Laser). PE and projects of interest (should see an uptick in funding) I) U.S. Army High Energy Laser (PE: Advanced Weapons Technology) Existing Project with Uptick in future spending II) U.S. Airforce- Cyber Technology Pattern Recognition and Autonomous Learning (PE: Aerospace Sensors) New Project 30

31 Recommendations Retrieve and house accomplishment data in the database Improve the database by creating more tables and linking them appropriately Explore and incorporate predictive analytics tools such as Linear Regressions, Support Vector Machines, K-Nearest Neighbors, etc. to better analyze data (may require more advanced analytical software packages/programming languages Julia, Python, R, SAS) Explore and incorporate textual analytics on the program/project descriptions and notes to investigate correlations between keywords/phrases and budget projections (may require more advanced analytical software packages/programming languages (Julia, Python, R, SAS) Continue to link policy (National Military Strategy, Defense Innovation Initiative) to budgetary trend analysis using BEAT (Significant opportunity for future RDT&E investment if current administration and DoD continues to invest in a Third Offset Strategy) Upgrades to BEAT: Heat Chart with conditional formatting based on relative changes Link each program to its page in the corresponding budget file 31

Conclusion Decision Management System Begin with the Decision in

improvement Mile 20 Next Step Mile 10 Program/Project/

itself to data analytics because: Abundance of training data

5 Program/Project Data Create repeatable process for importing

32 Conclusion Decision Management System Begin with the Decision in Mind Transparent & Agile Predictive Test, learn, continuous improvement Mile 20 Next Step Mile 10 Program/Project/ Accomplishment Data Projects, Accomplishment, and Plans data fields are verbose E.g. NLP, Support Vector Machine, K- Nearest Neighbors FYDP Data Lends itself to data analytics because: Abundance of training data Structured Released annually PB17 Mile 5.0 Mile 1.0 Mile 2.5 Program/Project Data Create repeatable process for importing future data Run surface level analytics to identify trends, significant changes, or beginning/ending of funding BEAT can present data in minutes 32

33 Acknowledgements System Planning and Analysis, Inc. Andy Dichter, Slider Griese, Chris Anderson, and team SEOR Faculty Prof. Laskey OR /SYST 699 Classmates 33

34 Questions 34

35 Backup 35

36 Acronym Name Project Acronyms BA Budget Activity A Budget Activity (BA) is a category within each appropriation and fund account that identify the purposes, projects, or types of activities financed by the appropriation or fund BEAT Budget Exhibit Analysis Tool Decision Management Support Tool that allows users to visualize RDT&E data based on selective queries in order to conduct follow-on exploratory analysis DoD Department of Defense The executive branch department of the federal government of the United States charged with coordinating and supervising all agencies and functions of the government concerned directly with national security and the United States Armed Forces FFRDC Federally Funded Research and Development Centers Federally Funded Research and Development Centers (FFRDCs) are public-private partnerships which conduct research for the United States Government FY Fiscal Year Budgetary cycle for the DoD that starts in October and ends in September. FYDP Future Years Defense Program Summarizes resources (equipment, manpower, forces) associated by fiscal year, with DoD programs, as approved by the Secretary of Defense. PB President s Budget The president submits a comprehensive budget request to Congress in early February which outlines the administration s policy and funding priorities and economic outlook for coming year PE Program Element The primary data element in the FYDP and normally the smallest aggregation of resources used by the Office of the Secretary of Defense (OSD) for analysis. PEs are designed and quantified to be comprehensive and mutually exclusive. POM Program Objective Memorandum The 5-year FYDP and presents the Services and Defense Agencies proposal on how they will balance their allocation of available resources. The POM includes an analysis of missions, objectives, alternative methods to accomplish objectives, and allocation of resources. R1/R2 R1/R2 Data The R-1 document contains budget information for all RDT&E programs undertaken by the branches of the DoD. R-2 document is specific to each program, identified by a PE number in the R-1. RDT&E Research, Development, Test & Evaluation RDT&E appropriations finance research, development, test and evaluation efforts performed by contractors and government installations to develop equipment, material, or computer application software 36

37 Refrences American Association for the Advancement of Science 2016 [3] - Govini. No Date. DoD Financial Management Regulation Volume 2B, Chapter 5 Department of Defense, Future Years Defense Program (FYDP) Structure, April 2014 at Department of Defense (DoD) Releases Fiscal Year 2017 President s Budget Proposal Defense Technical Information Center (DTIC) GMUSEORSPA at Military budget of the United States, Wikipedia U.S. House of Representatives, Committee on the Budget at "US Department of Defense Third Offset Standard Market Taxonomy." Analyst Report, Washington, D.C. 37

Need Statement There is a need to advance

mechanism for categorizing and analyzing

38 Need Statement There is a need to advance the BEAT system to analyze project level budget data in order to project future trends for better decision making. Mission Statement The Budget Exhibit Analysis Tool (BEAT) will provide a mechanism for categorizing and analyzing past, present, and future Department of Defense (DoD), Research, Development, Testing & Evaluation (RDT&E) Presidential Budget (PB) submissions. User BEAT SYSTEM 38

39 Requirements Diagram Return to Scope/Requirements 39

Use Case 1: Load R-2 PE and Project Level Data into BEAT System Boundary Schedule: Main Use Case; must be tested before providing to sponsor Priority: Top, must be executed Performance Target: All

40 Use Case 1: Load R-2 PE and Project Level Data into BEAT System Boundary Schedule: Main Use Case; must be tested before providing to sponsor Priority: Top, must be executed Performance Target: All project level, unclassified budget level data is available from FY 11 through FY 20 Frequency: Every time a download is required due to release of new Presidential Budget for corresponding FY Return to BEAT Functional Description 40

Use Case 2: Perform Visualization of Project Level Data in BEAT Query data

unclassified budget level data is available from FY 11 through FY 20

41 Use Case 2: Perform Visualization of Project Level Data in BEAT Query data in order to determine what to visualize Schedule: Main Use Case; must be tested before providing to sponsor. Priority: Top, must be executed Performance Target: All project level, unclassified budget level data is available from FY 11 through FY 20 Frequency: Every time a query and filter for specific project level data is executed Return to BEAT Functional Description 41

42 IDEF0 for BEAT (A-0 Diagram) Return to Structural Architecture 42

43 IDEF0 for BEAT (A0 Diagram) Return to Structural Architecture 43

44 IDEF0 for BEAT (A5 Diagram) Return to Structural Architecture 44

45 Entity Relationship Diagram for BEAT Return to Structural Architecture 45

46 BEAT Decision Tables (1 of 2) (Based on IDEF0) Return to Structural Architecture 46

47 BEAT Decision Tables (2 of 2) (Based on IDEF0) Return to Structural Architecture 47

48 Sequence Diagrams for BEAT (Based on Use Case 1) Return to Structural Architecture 48

49 Sequence Diagrams for BEAT (Based on Use Case 2) Return to Structural Architecture 49

50 Types of Equipment / Activity 50

51 Types of Service /Agency 51

SYST/OR 699 Master s Project Final Report

SYST/OR 699 Master s Project Final Report Department of Defense Research, Development, Test and Evaluation Budget Exhibit Analysis Tool Enhancements Prepared by Craig Jayson, Josh Singh, and Phillip Sutton