Design of a Budget Exhibit Analysis Tool for Analysis of the Department of Defense Research, Development, Test and Evaluation Budget

Transcription

1 SYST/OR 699 Master s Project - Report Design of a Budget Exhibit Analysis Tool for Analysis of the Department of Defense Research, Development, Test and Evaluation Budget Prepared for Systems Planning and Analysis, Inc. Chris Anderson, Kris Cruz, Jhonnattan Diaz, Said Masoud

2 TABLE OF CONTENTS PROBLEM DEFINITION... 1 SPONSORSHIP... 1 BACKGROUND... 1 PROBLEM STATEMENT... 3 PROBLEM AND NEEDS STATEMENT... 3 OBJECTIVES AND PURPOSE... 4 STAKEHOLDERS... 4 SCOPE... 6 ANALYSIS... 8 REQUIREMENTS MISSION REQUIREMENTS SYSTEM REQUIREMENTS FUNCTIONAL REQUIREMENTS DATA EXTRACTION AND ANALYSIS DELIVERABLES BUDGET EXHIBIT ANALYSIS TOOL (BEAT) R-1 BUDGET DATA EXPLORATION (PAST YEARS) R-2 BUDGET DATA EXPLORATION (FUTURE YEARS) FUTURE OPPORTUNITIES ACCOMPLISHMENTS DECISION POINTS PROGRAM MANAGEMENT EARNED VALUE MANAGEMENT SYSTEMS REFERENCES APPENDIX A BUDGET ACTIVITY DEFINITIONS... A-1 i

3 APPENDIX B - LIST OF ACRONYMS... B-1 APPENDIX C USE CASES... C-1 USE CASE 1: RUN R-1/R-2 LOAD BATCH JOB... C-1 USE CASE 2: RUN EXCEL GENERATOR JOB... C-4 USE CASE 3: PERFORM R-1/R-2 DATA VISUALIZATION... C-7 APPENDIX D SYSTEM ARCHITECTURE... D-1 ii

4 Table of Figures Figure 1. DoD Budget from in FY16 dollar (future years estimated)... 1 Figure 2. Average annual growth of select countries R&D budget... 2 Figure 3: Federal Spending on R&D... 3 Figure 4. Stakeholder Diagram... 6 Figure 5. Scope... 8 Figure 6. Use Case: Perform R-1/R-2 Data Visualization... 9 Figure 7. Sequence Diagram: OV-6C Data Visualization... 9 Figure 8. Requirements Diagram Figure 9. R-1 Budget Data for a specified President's Budget year Figure 10. R-2 XML document example Figure 11. BEAT system R-1 graphing options Figure 12. Spark Charts within BEAT (Smart Charts) Figure 13. DARPA drill-down to show PE's within Figure 14. R-2 excerpts from PB13-PB17, identifying FY Figure 15. R-2 Graph of Electronic Warfare Technology Figure 16. BEAT (R-2) system user-interface Figure 17. Advanced Aerospace Systems data for FY17 by PB submission year ($ Millions) Figure 18. Multiple program elements graphed Figure 19. Analysis ACROSS BAs, Services, Agencies Figure 20. Excerpt from PB17, showing FYDP Figure 21. Excerpt from R-2, showing Accomplishment description Figure 22. BEAT Work Breakdown Structure Figure 23. Earned Value Management Figure 24. Use Case 1 - Run R-1/R-2 Load Batch Job... C-2 Figure 25. Sequence Diagram: OV-6c Run Data Load Batch Job... C-3 Figure 26. Use Case 2 - Run Excel Generator Job... C-5 Figure 27. Sequence Diagram: OV-6c Run Excel Generator Job... C-6 Figure 28. DIV-1 Conceptual Data Model... D-1 Figure 29. OV-5b: Operational Activity Model... D-2 Table of Tables Table 1. Categories of Appropriation... 7 Table 2. Budget Activity Numbers and Names... 7 Table 3. Use Case 1 - Success Criteria... C-1 Table 4. Use Case 1 - Scenario Extensions... C-2 Table 5. Use Case 2 - Success Criteria... C-4 Table 6. Use Case 2 - Scenario Extensions... C-4 Table 7. Use Case 3 - Success Criteria... C-7 Table 8. Use Case 3 - Scenario Extensions... C-8 iii

5 Problem Definition Sponsorship Systems Planning and Analysis, Inc. (SPA) is a professional services contractor located in Alexandria, VA. SPA has permitted the Joint Air Force & Analysis Division (JTAAD), and specifically the Operational Support Analysis Group (OSAG) to become the first sponsor of an educational capstone project. The problem statement outlined below has been derived from conversations with OSAG, the official client of this project. OSAG is trying to search out new analytical methods and approaches to categorizing and analyzing past, present, and future budget submissions. The proposal offered by OSAG was selected as a capstone project for our team Background Since 2000, the budget of the Department of Defense (DoD) has ebbed and flowed (See Error! Reference source not found.) in line with national security concerns and overseas military activities. For example, after 9/11, the DoD R&D budget increased by approximately 60% through FY2007, but declined 25% between FY2010-FY2014, coinciding with the drawdown of American forces and presence in Iraq and Afghanistan. (American Association for the Advancement of Science 2016) Base Funding Supplemental Funding Figure 1. DoD Budget from in FY16 dollar (future years estimated) Formally announced in 2014 by then Secretary of Defense Chuck Hagel, one major focus of the DoD has been the third offset strategy. Third offset is an effort to maintain technological superiority; focusing on such activities as but not limited to; Human-machine learning, Robotics, Undersea warfare, Hypersonics, semi-autonomous weapons, resilient and/or alternative methods of communication, and cyber 1

6 capabilities (both defensive and offensive). There is also an emphasis on agility and sustainability in manufacturing new systems, decreasing the logistical load of deployment by utilizing techniques such as modular manufacturing and 3D printing. In December of 2015, Deputy Secretary of Defense (DEPSECDEF) Bob Work stated that 2017 will be an important year to lay the groundwork for this strategy. (American Association for the Advancement of Science 2016) Third offset strategy represents more than $18 billion of budget requests across the Future Years Defense Program (FYDP), constituting an overhaul in US military operating concepts and technological investments. The concepts and investments are intended to address emerging capability gaps as well as deter potential near-peer adversaries, such as China and Russia. (.govini Accessed 2016). These strategic goals, however, are being pursued in an increasingly constrained fiscal environment. The Budget Control Act of 2011 created several mechanisms for enforcing stricter control of spending, including sequestration, and its effects can be seen in the flat future budget projections of the final years depicted in Figure 1. DoD Budget from in FY16 dollar (future years estimated). The US no longer has a monopoly on power projection, precision guidance, or the enabling infrastructure that made it the dominant military power since the end of the Cold War. China and Russia have observed the US s capabilities and have been making strategic investments that counter US advantages. These efforts are evidenced by the average annual R&D growth of these two countries, which is significantly greater than that of the US (See Figure 2. Average annual growth of select countries R&D budget). (.govini Accessed 2016) Figure 2. Average annual growth of select countries R&D budget 2

7 The domain of this project is limited to Research and Development (R&D) budgeting within the Department of Defense. The DoD typically accounts for one-half of the federal Research and Development budget approximately $76.6 billion in 2016 (See Figure 3: Federal Spending on R&D). Problem Statement Problem and Needs Statement Figure 3: Federal Spending on R&D Despite the magnitude of DoD R&D investment, there is currently no known tool for housing and visualizing budget data. The data are dispersed into many separate documents and locations. Manual formatting of data is time-consuming, and there is no efficient way to isolate and visualize data meeting a set of user-specified criteria. Currently, if one wanted to visualize a forecasted budget trend over time using Future Years Defense Program (FYDP) projected data, they would need to take the following steps: Download one R-2 document from Defense Technical Information Center (DTIC) website for each year in the 5-year RDT&E budget cycle o These documents are usually several hundred pages long, as they contain information on the progress and future goals of each Program Element (PE) Find the PE of interest within each document Export budget data to a program (such as Microsoft Excel) for visualization Repeat process for multiple PEs if comparison is desired 3

8 This is a time-consuming process. So, while it is possible to track a PE budget over time by accessing budget documents for each year, there is no known way to access that information rapidly or isolate PEs which meet a set of user-specified criteria without searching for them manually. The purpose of the Budget Exhibit Analysis Tool (BEAT) is to meet this need by allowing users to quickly identify PEs of interest using historic data and visualize their individual budget trends over time using Future Years Defense Program (FYDP) projected data. Objectives and Purpose The objective of this project is to develop a methodology to analyze past, present, and future budget trends within the DoD S&T community. Different stakeholders will approach the DoD RDT&E budget with different questions depending on their role and areas of interest. It is of paramount importance to the team to develop a decision support system with parameters that can be set by the user to provide a tailored analysis. This tool must also be able to be updated with new data as it becomes available. The requirements of this tool as defined by the team, and agreed upon by the sponsor, are defined in the Requirements section of this document. Stakeholders There are many stakeholders in the DoD budgeting process. Some of these stakeholders are decision makers who are responsible for determining the budget, others are responsible for allocating it, and most are the recipients of those allocated funds. It should be noted that, with the diversity of interests at work within the budgeting process, not all of them are of a nature such that they would consider this project of value. For example, someone who is responsible for only one project and whose primary responsibilities are to document that project s progress and justify future funding may not be interested in trend analysis. The primary stakeholders those who would consider this project of value to them are those who are interested in budget trend analysis. This might include, for example, a company which is considering how to best allocate its resources to projects which are likely to receive increased attention and funding in the future. The following is an overview of the stakeholders and their roles in the DoD budgeting process: The President and Congress The DoD submits an annual budget request to the President for review. The President then submits a budget request to Congress. This request is reviewed and subjected to an iterative process of revisions before it is returned to the President, at which point it is either signed into law or vetoed. Understanding trends within R&D is vital for decision makers to be able to rally support for the expenditures they support or against the expenditures they oppose. 4

9 Services (Department of the Navy, US Air Force, US Army) The primary organizations through which military policy is carried out, the services each have research laboratories which are organized under them. They are the principal end users of the technologies developed, and are therefore essential for determining the usefulness of any technology. Communities of Interest (COIs) A network of technical groups that serve as a lasting structure to integrate the diverse technological efforts throughout the DoD. COIs were established in 2009 as a mechanism to encourage multi-agency coordination and collaboration in cross-cutting technology focus areas with broad multiple-component investment (Research & Engineering Enterprise: EXCOM 2016). Having a powerful analytical tool at their disposal will enable COIs to better facilitate the coordination of R&D efforts across the DoD and monitor the general state of development for specific areas of technology. Industry A diverse collection of private entities whose interests and level of involvement with the DoD vary significantly. Some are recipients of contracts for specific research projects, while others are contracted to provide an array of consulting services. Naturally, industry players are in competition with one another for the limited resources apportioned to the services they provide. Being able to identify and predict future budgeting developments is invaluable as companies seek to determine how to best allocate their resources and direct their future endeavors. Federally Funded Research & Development Centers (FFRDCs), and University-Affiliated Research Centers (UARCs) Research partnerships in which the DoD is the sponsor of a research facility which is under the administration of either a corporate entity (FFRDCs) or a university (UARCs). Office of the Secretary of Defense (OSD) The Secretary of Defense is the primary defense policy advisor to the President and is responsible for formulating and executing policy through the DoD. The OSD, as the staff of the Secretary, shares in those responsibilities by providing management and oversight for the numerous components of the DoD. Leadership and decision making effectiveness can be improved with the development of decision support tools. The Assistant Secretary of Defense (Research & Engineering) ASD(R&E) is one component of the OSD. It provides S&T leadership and strengthens coordination efforts Science and Technology (S&T) Laboratories In addition to the laboratories organized under the Army, Navy, and Air Force, the DoD allocates funding to other laboratories, some of which are independent (directly under the DoD), and some of which are within other DoD agencies such as DARPA. 5

10 Figure 4. Stakeholder Diagram Lastly, it is important to grasp the profound impact that even slight improvements to information accessibility and visibility of trends can have. It is easy, but incorrect, to look at each year s budget as the product of a brief decision process with a definite beginning and end. In actuality, determining the future budgets for the many components of the DoD is an ongoing undertaking that many of the different stakeholders participate in. Also, the description of stakeholders in this report is greatly simplified, and there is a multidimensionality to the organizations represented, with organizations each having several levels of decision makers interested in identifying budget trends. Given the interminable nature of the budgeting process, and the sheer number of people involved in it, the benefit of increased access to information and greater visibility of trends cannot be understated. Scope There are natural scoping mechanisms built into the budget nomenclature and taxonomy used to build the budget documents. The first scoping mechanism is the separation of Research, Development, Test, and Evaluation (RDT&E) funding (See Table 1. Categories of Appropriation) from the other types of funding. There are five major categories of appropriation: 6

11 Table 1. Categories of Appropriation Appropriation Category Scope Funding Policy Research Development, Test, Activities and expenses related to research, Incremental and Evaluation (RDT&E) development, test, and evaluation Procurement (PROC) End items >= $100K system unit cost, All centrally managed items, Initial spares, Labor for certain Full production-related functions (e.g., item assembly, quality assurance) Operations and Maintenance (O&M) Replenishment spares, Fuel, Civilian salaries, Construction projects < $750K, Travel, Noncentrally managed end items < $100K system unit cost Annual Military Personnel Expenses Military Personnel Expenses Annual (MILPER) Military Construction (MILCON) Construction Projects >= $750K Full S&T funding is primarily within the RDT&E funding component. We therefore chose to examine only that component of the budget. Furthermore, by examining the categories under RDT&E, we can further determine the budget categories that are likely to determine the most important trends in DoD research funding. There are seven BA s that define the total of the RDT&E budget. (See Table 2. Budget Activity Numbers and Names) Table 2. Budget Activity Numbers and Names Budget Activity Number Budget Activity Name 1 Basic Research 2 Applied Research 3 Advanced Technology Development 4 Advanced Concept Development & Prototypes 5 System Development and Demonstration 6 RDT&E Management and Support 7 Operational System Development The scope of this project will reduce the research and analysis of all seven budget activities to just three of them; BA-2: Applied Research, BA-3: Advanced Technology Development, BA-4: Advanced Concept Development and Prototypes. The following BAs will be out of scope for this project: BA-1: Basic Research does not immediately yield a technology that could be produced to assist the warfighter BA-5: System Development and Demonstration include programs having passed Milestone B within the acquisition process leading to full rate production BA-6: RDT&E Management and Support includes funds to sustain and/or modernize installations 7

12 BA-7: Operational System Development includes the developmental efforts to upgrade systems that have previously been fielded or received approval for full rate production Figure 5. Scope Analysis After obtaining input from the project sponsor, a preliminary concept of operations (CONOPS) was developed to elicit requirements for the BEAT system. In addition to the input from sponsors, use cases were developed to formalize the CONOPS, and the viability of the system design. We present a sample of a use case [Perform R-1/R-2 Data Visualization] to show the main success scenario for displaying data to the analyst on demand. The included case has to do with the realization that the add-on tool part has to be running concurrently with MS Excel. The extend case delineates what happens when the tool does not find a Program Element in the query list (See Figure 6. Use Case: Perform R-1/R-2 Data Visualization). Other use cases were developed in order to account for the functionality of the BEAT tool, and can be found in the appendix. 8

13 Figure 6. Use Case: Perform R-1/R-2 Data Visualization In addition to the use cases, sequence diagrams were developed from the use cases to make sure that the order of operations made sense with the dynamics of the tool (See Figure 7. Sequence Diagram: OV-6C Data Visualization). The user launches the Smart Chart Tool, so the ExcelGeneratorJob can populate the R-1, and R-2 data that the ExcelVisualizationTool can graph and display the data to the analyst depending on the his/her input criteria for the query. Additional sequence diagrams for specific use cases can be found in the appendix. Figure 7. Sequence Diagram: OV-6C Data Visualization 9

14 Requirements Mission Requirements MR1: The BEAT system shall be able to ingest RDT&E budget information from R-1 and R-2 documents. MR2: The BEAT system shall have the ability to ingest a minimum of five years of RDT&E budget exhibit documents, when data is available MR3: The BEAT system shall visualize RDT&E budget data over a pre-determined time period System Requirements SR1: The BEAT system shall generate R-1 budget trend data by Program Element for budget activities 2, 3, and 4 SR2: The BEAT system shall generate R-2 budget trend data by Program Element for budget activities 2, 3, and 4 SR3: The BEAT system shall Aggregate budget data by Program Element for budget activities 2, 3, and 4 Functional Requirements FR1: The BEAT system shall create a plot that visualizes outliers of R-1 budget data FR2: The BEAT system shall identify outliers that exist within currently loaded data and newly loaded data FR3: The BEAT system shall identify new Program Elements by first fiscal year reported FR4: The BEAT system shall identify discontinued Program Elements by last fiscal year reported FR5: The BEAT system shall Monitor a minimum of five RDT&E budget cycles, when data is available FR6: The BEAT system shall aggregate Program Elements in order to conduct tailored analysis 10

15 Data Extraction and Analysis Figure 8. Requirements Diagram The data used for the purposes of this project is available to the public, and is obtainable at the website of the Defense Technical Information Center (Defense Technical Information Center 2016). Two types of documents were used to extract and analyze data. For the purposes of this project, only unclassified data was retrieved. R-1: The R-1 document contains budget information for all RDT&E programs undertaken by the branches of the DoD. For each fiscal year, the document lists the budget for the previous two fiscal years, as well as the budget for the current fiscal year. See Figure 9. R-1 Budget Data for a specified President's Budget year for an example of an R-1 document. 11

16 Figure 9. R-1 Budget Data for a specified President's Budget year R-2: The R-2 document is specific to each program, identified by a program element (PE) number in the R-1. The R-2 serves as the item justification for each program, as the proposed budget for the next four fiscal years is listed. In addition, a detailed description of each subprogram covered by the PE is included. See Figure 10. R-2 XML document example for an example of an R-2 document. Figure 10. R-2 XML document example The R-1 and R-2 documents are available in Excel format and XML format, respectively, among others. The data was extracted from the two file types via a Java batch application, and loaded into a MySQL database. The R-2 documents all have the same schema; however, the format of the columns are slightly different in the R-1 documents for each fiscal year, therefore some manual manipulation was required before the documents were loaded via the batch application. 12

17 In the database, the data from the R-1 documents was loaded into a single table, using the PE number and the fiscal year reported as the unique identifier for each row. The schema for the R-2 documents is similar to the R-1 schema where the fiscal year reported and PE number are the unique identifiers for the R-2 table. Once the data was loaded into the database, the raw data was extracted and loaded into an Excel template by the Java batch application. In order to visualize the respective trends, Excel graphs and other visuals were developed and generated automatically using pre-defined Visual Basic for Applications (VBA) scripts that were automatically executed after the raw data was loaded into the template. Within the Java application, templates containing pre-formatted Excel sheets and VBA scripts were packaged together, and, in the future, when new R-1 or R-2 documents are loaded into the database, the results can be loaded into new files via the templates for analysis and visualization. The VBA scripts will automatically be executed by the Java application, such that when the application has completed running, an end user can simply open the Excel file created and instantly see the results. In order to allow the sponsor to have the option to continue to use the application, a separate user manual, which explains how to set up the database and Java batch application, was developed. This manual contains a description of the Java batch application, as well as instructions to set up a MySQL database, if desired. Deliverables The following items will be delivered to the stakeholder at the conclusion of this project: Java Executable (.jar): The Java batch application will be compiled into an executable JAR file that can be readily executed, with the modification of the application properties Java source code: The Java source code will be saved as an Eclipse project and compressed for more efficient delivery. In the event that the sponsor would like to modify or build on the code, the sponsor can download and set up Eclipse, then import the project directory. MySQL database export: A MySQL database export will be provided so that the sponsor does not have to worry about setting up the database manually and trying to load the data that was already loaded during the course of this project. The export will be formatted such that if the sponsor chooses to use another type of database, the data can still be imported without much effort. Excel templates and completed data sheets: The compiled data sheets used to develop the results of the analysis will be provided to the sponsor so that they may continue to perform analysis on the data. In addition, Excel templates will be provided in the Java JAR file and Eclipse project package so that when new data is available, they may create new Excel sheets for analysis. User Manual: A user manual that will provide instructions on how to set up a MySQL database and implement the Java batch application. Budget Exhibit Analysis Tool (BEAT) BEAT was developed to aid in the identification of budget request trends within the RDT&E budget. The BEAT system has the ability to upload new President s Budget documents when they are released (annually). The file formats capable of being uploaded are MS Excel (R-1) and XML (R-2). Previously analysts would have to search on disparate sites for the specific documents in question, download them, 13

18 unpack (WinZip) then, and manually combine them into usable formats for their research. While the downloading takes a long time, and will still be necessary with the BEAT system, the most time-consuming event is in the compilation of the data. The BEAT system is comprised of over 1,000 lines of code. It is this code that BEAT will use to extract, transform, and load the documents in a user-friendly format into the BEAT system (MS Excel Application). The system then runs super-macros (leading code that will run the pertinent routines in the proper order) in order to finalize the transformation of the data using MS Excel VBA. This process happens for both the R-1 and R-2 data. BEAT is a tool that is used in two stages. The first stage utilizes the R-1 data from the President s Budget files. The R-1 s were collected from the Defense Technical Information Center (DTIC). Data collection resulted in files from the year 1998 until As stated in the previous section, the R-1 data contains budget information for the requested year, and the two previous years; hence will be used to identify historic trends in budget requests. The second stage utilizes the R-2 data from the President s Budget. The R-2 s provide greater detail than the R-1 s. The R-2 s provide the current year request, the two previous years of spending data, and four years worth of projections for a specific program element. BEAT uses the future data to show users how projections and requests change from fiscal year to fiscal year, over sequential President Budget submissions. In order to identify historic trends the BEAT system uses an Excel Add-in called Smart Charts. Smart Charts was developed by a company called DevScope and downloaded from the Windows App Store. The cost for Smart Charts was $5.00. Smart Charts allows for time-series data to be graphed in multiple manners and includes a dynamic aspect. The charts selected can be filtered dynamically and a time-series video can be played to show how the data changes over time; in this case the time factor is the President s Budget submission years. R-1 Budget Data Exploration (Past Years) The BEAT system can show data in multiple ways: Tree Maps, Pie Charts, Bar Graphs, and Bubble Charts (See Figure 11. BEAT system R-1 graphing options). At a macro level, this tool is meant to show you how the service s or agency s share of the budget changes over time. We also used this to show the annual change of appropriated funds from year to year. 14

19 Figure 11. BEAT system R-1 graphing options Identifying services or agencies of interest is based on the user s research. Hovering over a service or agency will show the user a spark chart of the data over time. Based on the filters set in Figure 12. Spark Charts within BEAT (Smart Charts), the user is able to identify the absolute change in budget requests from the President s Budget submissions from This hovering spark chart allows the user to quickly scroll through the chart to identify the service or agency of interest. Figure 12. Spark Charts within BEAT (Smart Charts) 15

20 By selecting DARPA, you will be able to see the program elements DARPA is funding. The filters were changed in Figure 13. DARPA drill-down to show PE's within to show the absolute change in budget requests, and includes each program element title. Again, you may hover over the program element names in order to show a spark chart of the funding requests within that specific program element. Selecting a program element will show that specific program element only, which may not be useful as it will be the only item on the chart. R-2 Budget Data Exploration (Future Years) Figure 13. DARPA drill-down to show PE's within After identifying a program element of interest, the user can begin stage two of the BEAT system by delving into the projections of the R-2 budget documents. The R-2 portion of the BEAT system was developed using MS Excel and VBA and provides visual charts to show the user how the projections for a specified fiscal year change as the fiscal year approaches. As shown in Figure 14. R-2 excerpts from PB13- PB17, identifying FY17 the projections for fiscal year 2017 increased approximately 60% over 5 years; from $ Million in 2013 to $ Million in These are the trends shown in stage 2 of the BEAT system in Figure 15. R-2 Graph of Electronic Warfare Technology. 16

21 Figure 14. R-2 excerpts from PB13-PB17, identifying FY17 Figure 15. R-2 Graph of Electronic Warfare Technology 17

22 Identifying changes in projected funding can assist stakeholders in identifying the key technologies that are gaining funding. This will assist companies trying to gain a foothold in federal contracting by giving them the insights needed to identify key opportunities for their business units. There are over 500 program elements to choose from, so it is important to make the tool user-friendly. The R-2 section of the BEAT system allows the user to search and filter to see only the data requested. Figure 16. BEAT (R-2) system user-interface shows the filtering options available to the user. The user can filter the program elements based on the fiscal year of interest (default is 2017) which is the year whose value will be shown as requested across President s Budget submissions. The user can also filter by Service or Agency (e.g. Army, DARPA, etc.), by budget activity (i.e. BA2 BA4), and/or use the text search capability. The listing of program elements will dynamically filter itself upon each selection showing the user the program elements based on the filters selected. Figure 16. BEAT (R-2) system user-interface Once the user has selected the program element(s) to research further, the user presses the OK button. That will populate the graph with the selected program elements using the data requested. As shown in Figure 17. Advanced Aerospace Systems data for FY17 by PB submission year ($ Millions), the user will be presented with the graph, displaying a legend of the selected program elements, data markers to alert the user to the amount depicted, and specific markers. The markers alert the user to the specific budget 18

23 activity of the program element (e.g. Square = Budget Activity 2, Square = Budget Activity 3, and Triangle = Budget Activity 4). This will aid in discerning the data on the graph without cluttering it. Null values have been taken out of the dataset, as they are translated as zero values within the dataset. As such they were skewing the data, and could lead the user to believe that a budget request was made within the President s Budget for that year and no money appropriated to the program element. Values that are actually zero, will be portrayed within the chart. Figure 17. Advanced Aerospace Systems data for FY17 by PB submission year ($ Millions) The BEAT system has the ability to graph multiple selections (i.e. program elements) simultaneously. This is a first step toward the future work recommendations presented in the next section of this report. As shown in Figure 18. Multiple program elements graphed filters were used to select items that were alike. The graph on the upper-left portion of Error! Reference source not found.was filtered using the text search functionality. The search was performed on LCS (Littoral Combat Ship), bringing up two results: LCS mission modules and Littoral Combat Ship (LCS). Both selections are budget activity 3 and Navy programs. The graph on the lower-right portion of Figure 18 used program elements selected using multiple filters; the Service / Agency filter and the Budget Activity filter. The agency selected was DARPA and the budget activity selected was BA 3. Out of the eight program elements remaining (post-filtering), four were selected: Advanced Aerospace Systems, C3 Systems, Network-Centric Warfare Technology, and Space Programs & Technology. While these are just examples of the types of graphs that can be created, 19

24 based on the user s filtering preferences, more in-depth case studies were performed in order to provide a proof of concept for the BEAT system. Future Opportunities Figure 18. Multiple program elements graphed While we are able to graph the program elements from different services and budget activities on the same chart, there are much more in depth and cross-program analyses that are possible. Building the macros to allow such analysis would be a good future project for this class or could be done by the sponsor. It allowed us to show a proof of principle (Figure 19. Analysis ACROSS BAs, Services, Agencies). The future of this tool will be to take the analysis ACROSS budget activities and services / agencies. 20

25 Figure 19. Analysis ACROSS BAs, Services, Agencies Program Elements are the top layer of the analytical onion. Underneath program elements, there are projects listed within (one or more). Each project, in turn, has multiple accomplishments (one or more) that comprise the project. This layering effect provides more detail and different information the further you drill down. For an example, see below: PE: E / Advanced Aerospace Systems (DARPA) Project: AIR-01 / Advanced Aerospace Systems o Accomplishments: Tactically Exploited Reconnaissance Node (TERN) Collaborative Operations in Denied Environments (CODE) Hypersonic Air-breathing Weapon Concept (HAWC) Tactical Boost Glide Advanced Aerospace System Concepts Technology for Enriching and Augmenting Manned-Unmanned Systems Vertical Take-off and Landing (VTOL) Technology Demonstrator Distributed Fires Advanced Full Range Engine (AFRE) The data currently displayed within the BEAT system lends itself well to data analytics. Future work involves the projection beyond the FYDP (four years in the future), (Figure 20. Excerpt from PB17, showing 21

26 FYDP+). There is an abundance of data that can be used as the training data to test the model or models developed. Projections could be used to alert users to trends of interest (i.e. increase, decrease, or steady funding profiles). The trends identified will be useful to government contractors looking to identify the lines of business that represent opportunities to remain in, research the feasibility of getting in, and research the feasibility of divesting. The opportunities identified relate directly to the trends of the Science & Technology fields and whether they are increasing, decreasing, or remaining steady. Figure 20. Excerpt from PB17, showing FYDP+ Another opportunity for future work would be delving deeper into the data within the R-2 budget exhibits using machine learning techniques. The data within the R-2 budget documents below the program element level contains more diverse data than at the program element level. The projects and accomplishments have much more verbiage defining them. This verbiage lends itself to natural language processing, support vector machine (SVM), and K-Nearest Neighbors (KNN) models (See Figure 21. Excerpt from R-2, showing Accomplishment description). The output analysis of these types of models could alert the user to specific projects and/or accomplishments relating to their areas of expertise, for further research by the user. Accomplishments Figure 21. Excerpt from R-2, showing Accomplishment description As the BEAT System team, we have been successful in many ways. We have been able to provide a proof of concept tool for our sponsor that drastically reduces the amount of time spent parsing and compiling the disparate files related to the R-1 and R-2 budget exhibits. The bulk of the set-up time will still be the downloading of the documents. The R-1 documents are readily available in MS Excel format on Comptroller s website ( but the timeconsuming portion will be the download of the R-2 budget documents. The R-2 budget documents are housed in many locations. Services (i.e. Navy, Air Force, Army) are posted on the service specific comptroller website grouped by budget activity, while the agencies (i.e. DARPA, MDA, etc.) are posted in multiple documents indexed by their respective budget activity levels (Defense Technical Information Center 2016). The documents to download are in.pdf format, with the appropriate.xml files attached to the parent document as a.zip file. 22

27 Once the files are unzipped, and located on the computer of the analyst, they can easily be run through the extract, transform, and load (ETL) phases. This portion of the BEAT system does the parsing and compilation work of the analyst. This process is the MASSIVE time-saver that will allow the analyst to concentrate his/her time on the analysis and research portion of their work. Trends are easily identified within the BEAT system using the graphing capabilities and custom userinterfaces within the tool. The analyst has many options, in terms of filters, to gain valuable information quickly of the processed data. This allows the analyst to customize the work being completed, and back up the analysis with graphs, solidifying their position within the analysis. Decision Points The BEAT system team has validated the intent of this project and is able to offer decision points for the sponsor. The sponsor will need to identify the next analytic opportunity based on the decision points chosen or creating their own vision of where to take this tool. Options for the continuation of this project include: Deep dive into the R-2 budget exhibits o Projects o Accomplishments o Plans Data Models o Natural Language Processing (descriptions) o Forecasting (FYDP+) Program Management After requirements elicitation was completed with the customer, and agreed upon, a Work Breakdown Structure (WBS) was developed to gauge the price of the contract. The Statement of Work (SOW) will serve as a delimiter for the work to be performed. The cost for the entirety of the project was estimated to be $66, and designated as a non-fixed cost contract. Any changes by the customer after the SOW has been signed, such as additional work, change of requirements, change in the schedule; will change the cost of the project, and a new SOW shall be drafted, and signed. There are 30 hours allocated in the contract to meet with knowledge experts, and consultants at a rate of $ per hour. If extra consultant hours are needed, the developer will need authority to proceed from the customer. This is the Work Breakdown Structure developed to plan, control, and integrate the BEAT concept. There are four major areas for the successful completion of the project as follows: Analysis Design Implementation Verification & Validation (V&V) 23

28 In the Analysis phase, special attention was devoted to the project overview in order to get a better idea of what needed to be developed. Early collaboration with the sponsor was a major aid in the notion of what the customer expects from the project team. Also, in this phase, being able to define the problem with those who would eventually use the tool was beneficial in eliciting requirements. There were three Customer meetings scheduled to review the progress being made by the developer of the project. Customers are expected to interact with the developers for purposes of achieving the requirements of the project. Figure 22. BEAT Work Breakdown Structure In the design phase, a CONOPS was developed and later on formalized with the aid of Use Cases derived from the customer. Once the use cases were developed, formal Mission requirements emerged, and later on systems requirements flowed naturally to the lower levels. Once a firm base of requirements was established, the data model was developed to form the structure of the BEAT tool. In the Implementation phase, a database was designed to hold the information researched from the different Program Elements (PE) that will reside in the R-1 & R-2 data tables. The database was tested to make sure its components worked correctly, and that CRUD operations could be performed. Also, the visualization capabilities were developed, thereby satisfying one of the main components of the requirements. In the V&V phase, the operations of the database were tested with the data integrity and validation of the visualization. After correct operation of the database was assessed as working as expected, the data 24

29 analysis tools were designed to make the reports and visualization user-friendly. components was tested by the use cases. Each of these Earned Value Management Systems It was important for the group to perform EVMS throughout the life time of project to ensure that cost, and schedule were track. At the beginning of the project, a non-fixed contract was accorded with the customer for roughly $62,000 for a period of 15 weeks. Figure 23. Earned Value Management Although it was not required to implement EVMS for this project, the team found it imperative to have transparency with the customer on how their money was spent, and that the project was on schedule in the systems development life cycle. A customer representative shall be designated to communicate with the customer and any questions shall be directed to this person. 25

30 References.govini. Accessed "US Department of Defense Third Offset Standard Market Taxonomy." Analyst Report, Washington, D.C. American Association for the Advancement of Science Guide to the President's Budget Research & Development FY2017. Washington, D.C.: AAAS. Belasco, Amy Defense Spending and the Budget Control Act Limits. US Defense Policy and Budget, Washington, D.C.: Congressional Research Service. Defense Technical Information Center August 30. Accessed August 30, September 17. Accessed September 17, Research & Engineering Enterprise: EXCOM. Accessed October 3, Rugy, Veronique de High Levels of Government Spending Become Status Quo. Fairfax: Mercatus Center. Systems Planning and Analysis, Inc Systems Planning and Analysis, Inc. September 17. Accessed September 17,

31 Appendix A Budget Activity Definitions (Defense Technical Information Center 2016) Budget Activity 1, Basic Research Basic research is systematic study directed toward greater knowledge or understanding of the fundamental aspects of phenomena and of observable facts without specific applications towards processes or products in mind. It includes all scientific study and experimentation directed toward increasing fundamental knowledge and understanding in those fields of the physical, engineering, environmental, and life sciences related to long-term national security needs. It is farsighted high payoff research that provides the basis for technological progress. Basic research may lead to: (a) subsequent applied research and advanced technology developments in Defense-related technologies, and (b) new and improved military functional capabilities in areas such as communications, detection, tracking, surveillance, propulsion, mobility, guidance and control, navigation, energy conversion, materials and structures, and personnel support. Program elements in this category involve pre-milestone A efforts. Budget Activity 2, Applied Research Applied research is systematic study to understand the means to meet a recognized and specific need. It is a systematic expansion and application of knowledge to develop useful materials, devices, and systems or methods. It may be oriented, ultimately, toward the design, development, and improvement of prototypes and new processes to meet general mission area requirements. Applied research may translate promising basic research into solutions for broadly defined military needs, short of system development. This type of effort may vary from systematic mission-directed research beyond that in Budget Activity 1 to sophisticated breadboard hardware, study, programming and planning efforts that establish the initial feasibility and practicality of proposed solutions to technological challenges. It includes studies, investigations, and non-system specific technology efforts. The dominant characteristic is that applied research is directed toward general military needs with a view toward developing and evaluating the feasibility and practicality of proposed solutions and determining their parameters. Applied Research precedes system specific technology investigations or development. Program control of the Applied Research program element is normally exercised by general level of effort. Program elements in this category involve pre-milestone B efforts, also known as Concept and Technology Development phase tasks, such as concept exploration efforts and paper studies of alternative concepts for meeting a mission need. Budget Activity 3, Advanced Technology Development (ATD) This budget activity includes development of subsystems and components and efforts to integrate subsystems and components into system prototypes for field experiments and/or tests in a simulated environment. ATD includes concept and technology demonstrations of components and subsystems or system models. The models may be form, fit and function prototypes or scaled models that serve the same demonstration purpose. The results of this type of effort are proof of technological feasibility and assessment of subsystem and component operability and producibility rather than the development of hardware for service use. Projects in this category have a direct relevance to identified military needs. Advanced Technology Development demonstrates the general military utility or cost A-1

32 reduction potential of technology when applied to different types of military equipment or techniques. Program elements in this category involve pre-milestone B efforts, such as system concept demonstration, joint and Service-specific experiments or Technology Demonstrations and generally have Technology Readiness Levels of 4, 5, or 6. Projects in this category do not necessarily lead to subsequent development or procurement phases, but should have the goal of moving out of Science and Technology (S&T) and into the acquisition process within the future years defense program (FYDP). Upon successful completion of projects that have military utility, the technology should be available for transition. Budget Activity 4, Advanced Component Development and Prototypes (ACD&P) Efforts necessary to evaluate integrated technologies, representative modes or prototype systems in a high fidelity and realistic operating environment are funded in this budget activity. The ACD&P phase includes system specific efforts that help expedite technology transition from the laboratory to operational use. Emphasis is on proving component and subsystem maturity prior to integration in major and complex systems and may involve risk reduction initiatives. Program elements in this category involve efforts prior to Milestone B and are referred to as advanced component development activities and include technology demonstrations. Completion of Technology Readiness Levels 6 and 7 should be achieved for major programs. Program control is exercised at the program and project level. A logical progression of program phases and development and/or production funding must be evident in the FYDP. Budget Activity 5, System Development and Demonstration (SDD) SDD programs have passed Milestone B approval and are conducting engineering and manufacturing development tasks aimed at meeting validated requirements prior to full-rate production. This budget activity is characterized by major line item projects and program control is exercised by review of individual programs and projects. Prototype performance is near or at planned operational system levels. Characteristics of this budget activity involve mature system development, integration and demonstration to support Milestone C decisions, and conducting live fire test and evaluation and initial operational test and evaluation of production representative articles. A logical progression of program phases and development and production funding must be evident in the FYDP consistent with the Department s full funding policy. Budget Activity 6, RDT&E Management Support This budget activity includes research, development, test and evaluation efforts and funds to sustain and/or modernize the installations or operations required for general research, development, test and evaluation. Test ranges, military construction, maintenance support of laboratories, operation and maintenance of test aircraft and ships, and studies and analyses in support of the RDT&E program are funded in this budget activity. Costs of laboratory personnel, either in-house or contractor operated, would be assigned to appropriate projects or as a line item in the Basic Research, Applied Research, or ATD program areas, as appropriate. Military construction costs directly related to major development programs are included. Budget Activity 7, Operational System Development This budget activity includes development efforts to upgrade systems that have been fielded or have received approval for full rate production and anticipate production funding in the current or subsequent fiscal year. All items are major line item projects that appear as RDT&E Costs of Weapon System Elements A-2

33 in other programs. Program control is exercised by review of individual projects. Programs in this category involve systems that have received Milestone C approval. A logical progression of program phases and development and production funding must be evident in the FYDP, consistent with the Department s full funding policy. A-3

34 Appendix B - List of Acronyms ACD&P ASD(R&E) ATD BCA BEAT BLI COI CSV DEPSECDEF DoD DoN DTIC EVMS FFRDC FYDP JTAAD LCS MILCON MILPER O&M OCO OSAG OSD PB PE PROC R&D RDT&E S&T SDD SECDEF SOW SPA SQL UARC USA USAF VBA WBS XML Advanced Component Development & Prototypes Assistant Secretary of Defense (Research & Engineering) Advanced Technology Demonstration Budget Control Act Budget Exhibit Analysis Tool Budget Line Item Communities of Interest Comma Separated Values Deputy Secretary of Defense Department of Defense Department of the Navy Defense Technical Information Center Earned Value Management System Federally Funded Research & Development Center Future Years Defense Program Joint Air Force & Analysis Division of SPA Littoral Combat Ship Military Construction Military Personnel Operations and Maintenance Overseas Contingency Operations Operations Support Analysis Group of SPA Office of the Secretary of Defense President's Budget Program Element Procurement Research & Development Research, Development, Test & Engineering Science & Technology Systems Development & Demonstration Secretary of Defense Statement of Work Systems Planning and Analysis, Inc. Structured Query Language University-Affiliated Research Center United States Army United States Air Force Visual Basic for Applications Work Breakdown Structure Extensible Markup Language B-1

35 Appendix C Use Cases Use Case 1: Run R-1/R-2 Load Batch Job Characteristic Information Goal In Context: Scope: Level: Pre-Condition: Success End Condition: Minimal Guarantees: Primary Actor: Trigger Event: Load downloaded R-1/R-2 file into database via batch job Java Batch Application, Database Sub-Functionality R-1/R-2 file downloaded from DTIC website Batch job successfully, data loaded into database Database is set up User: analyst interested in performing analysis on budget data R-1/R-2 file downloaded from DTIC website and moved to staging directory Main Success Scenario Table 3. Use Case 1 - Success Criteria Step Actor Action Description 1 User Execute command to R-1 batch job 2 R-1BatchJob Read R-1 batch job 3 R-1BatchJob Write R-1 job to Database 4 Database Send R-1 Job completion Receipt 5 User Execute command to R-2 batch job 6 R-2BatchJob Read R-2 batch job 7 R-2BatchJob Write R-2 job to Database 8 Database Send R-2 Job completion Receipt C-1

36 Scenario Extensions Table 4. Use Case 1 - Scenario Extensions Step Sub Step Condition Action Description 1a 2a Batch job fails because database services is not running 1a1 R-1 BatchJob Notify User of file execution failure 1a2 User Turn on database services Unable to read file 2a1 R-1BatchJob Notify User of read failure 2a2 User Close file Figure 24. Use Case 1 - Run R-1/R-2 Load Batch Job C-2

37 Figure 25. Sequence Diagram: OV-6c Run Data Load Batch Job C-3

38 Use Case 2: Run Excel Generator Job Characteristic Information Goal In Context: Scope: Level: Pre-Condition: Success End Condition: User runs Excel Generator batch job to create data to be exported to Excel Database, Java batch application, Excel visualization tool Sub-Functionality MS Excel must be loaded into machine. BVA tool kit must be enabled Batch application successful, Excel file created, VBS script executed Minimal Guarantees: Primary Actor: Trigger Event: User: analyst who wants to extract data from database and visualize the data User starts batch job Main Success Scenario Table 5. Use Case 2 - Success Criteria Step Actor Action Description 1 User Start R-1 batch job 2 R-1BatchJob Retrieve R-1 data 3 Database Send R-1 File 4 R-1BatchJob Read R-1 Excel file 5 R-1BatchJob Send command to execute R-1 VBA Macro 6 ExcelGeneratorJob Execute R-1 VBA Macro 7 ExcelGeneratorJob Send R-1 File job completion 8 User Receive R-1 completion receipt Scenario Extensions Table 6. Use Case 2 - Scenario Extensions C-4

39 Step Sub Step Condition Action Description VBA Macro fails to execute 6a 6a1 ExcelGeneratorJob Notify User of VBA execution failure 6a2 User Reset Excel Macro capabilities Figure 26. Use Case 2 - Run Excel Generator Job C-5

40 Figure 27. Sequence Diagram: OV-6c Run Excel Generator Job C-6

41 Use Case 3: Perform R-1/R-2 Data Visualization Characteristic Information Goal In Context: Scope: Level: Pre-Condition: Success End Condition: User wants to graph and visualize R-1/R-2 data ExcelGeneratorJob has the capability to graph and display R-1/R- 2 data Sub-Functionality MS Excel add-in SmartCharts (by DevScope) must be installed Visualization successfully loads data and filters Minimal Guarantees: Primary Actor: Trigger Event: User: interested in visualizing R-1 data and filters Visualization requests user to load data Main Success Scenario Table 7. Use Case 3 - Success Criteria Step Actor Action Description 1 User Launch Smart Chart Tool 2 ExcelGeneratorJob Populate R-1 Data 3 ExcelGeneratorJob Populate R-2 Data 4 ExcelVisualizationTool Send data receipt to user 5 User Select data to be graphed 6 User Filter PE list 7 User Search for specific PE 8 ExcelVisualizationTool Graph R-1 data 9 ExcelVisualizationTool Graph R-2 data 10 ExcelVisualizationTool Display R-1 data in smart chart add-on to user 11 ExcelVisualizationTool Display R-2 data in smart chart add-on to user C-7

42 Scenario Extensions Table 8. Use Case 3 - Scenario Extensions Step Sub Step Condition Action Description PE is not in the query list 6a 6a1 R-1 Batch Job Notify User PE cannot be found 6a2 User Search for another PE C-8

43 Appendix D System Architecture Figure 28. DIV-1 Conceptual Data Model D-1