The Government Performance and Results Act: An Analysis of Strategic Goals and Performance Measurements

Transcription

1 American Public University System Master's Capstone Theses The Government Performance and Results Act: An Analysis of Strategic Goals and Performance Measurements David W. Johnson Follow this and additional works at: Part of the Public Affairs, Public Policy and Public Administration Commons Recommended Citation Johnson, David W., "The Government Performance and Results Act: An Analysis of Strategic Goals and Performance Measurements" (2014). Master's Capstone Theses. Paper 22. This Capstone-Thesis is brought to you for free and open access by It has been accepted for inclusion in Master's Capstone Theses by an authorized administrator of For more information, please contact

2 School of Public Service and Health Public Administration The thesis for the Master s degree submitted by David Johnson under the title THE GOVERNMENT PERFORMANCE AND RESULTS ACT: AN ANALYSIS OF STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS has been read by the undersigned. It is hereby recommended for acceptance by the faculty with credit to the amount of 3 semester hours. Christi Scott Bartman April 23, 2014 Christi Scott Bartman, MPA, JD, PhD Associate Professor American Public University System (Date) Recommended for approval on behalf of the program Kristen Osborne Obst (Date) July 31, 2014 Kristen Osborne Obst, PhD Program Director, Public Administration Program Recommended on behalf of the program director (Date) Approved by academic dean

3 THE GOVERNMENT PERFORMANCE AND RESULTS ACT: AN ANALYSIS OF STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS A Master Thesis Submitted to the Faculty of American Military University by David W. Johnson In Partial Fulfillment of the Requirements of the Degree of Master of Public Administration April 2014 American Military University Charles Town, WV

4 Running head: STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 2 The author hereby grants the American Public University System the right to display these contents for educational purposes. The author assumes total responsibility for meeting the requirement set by United States copyright law for the inclusion of any materials that are not the author s creation or in the public domain. Copyright 2014 by David W. Johnson All rights reserved.

5 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 3 ABSTRACT OF THE THESIS THE GOVERNMENT PERFORMANCE AND RESULTS ACT: AN ANALYSIS OF STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS by David W. Johnson American Military University, April 28, 2014 Charles Town, West Virginia Professor Christi Bartman, Thesis Professor The premise of this study contends that the design of Agency Strategic Plans and Agency Performance Plans are inadequate and fail to provide the guidance and structure required to affect leader decision-making. Additionally, Strategic Plans and Agency Performance Plans fail to provide adequate scope, detail, guidance, and tools needed to integrate with an agency s budget- and program-management systems. This study closely examines the content and structure of selected Agency Strategic Plans and Agency Performance Plans to determine if the structure, design, and content of these plans is conducive to supporting leader-based decisionmaking and providing for integration with a comprehensive budget- and performancemanagement system. The study closes with an examination into the relationship and effect the agency s policy focus and its independence may have on the content and structure of these plans.

6 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 4 TABLE OF CONTENTS CHAPTER PAGE I. Introduction... 7 II. A Background on Performance Budgeting in the United States... 9 III. Conceptual Framework Use of GPRA Resource in the Management of Federal Agencies Use of GPRA Performance Measures to Assess Efficiency and Effective Use of GPRA Strategic Plans and Performance Measures for Programmatic Evaluations IV Methodology Design and Instrumentation Documents and Artifacts Data Collection and Analysis Limitations of the Study V. Results and Discussion Agency Strategic Plans Agency Performance Plans Agency Independence and Policy Focus VI. Summary and Recommendations Recommendations... 61

7 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 5 List of References Appendices... Error! Bookmark not defined.2

8 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 6 LIST OF TABLES TABLE PAGE Table 1 - Composite Evaluation of Agency Strategic Plans Table 2 - Analysis of Agency Goal Design Table 3 - Analysis of Objective Specificity Table 4 - Analysis of Objective Measurability Table 5 - Analysis of Objective Achievability Table 6 - Analysis of Objective Realism Table 7 - Analysis of Objective Timeliness Table 8 - Analysis of Performance Measure Budget Applicability Table 9 - Composite Evaluation of Agency Performance Plans Table 10 - Analysis of Performance Measure Usability Table 11 - Analysis of Performance Measure Type Table 12 - Composite Evaluation Based on Agency Independence Table 13 - Composite Evaluation Based on Agency Policy Focus... 57

9 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 7 I. Introduction During the later part of the twentieth century, leaders in both the public and private sectors became dissatisfied with traditional management practices and the business as usual mindset (Lemay, 2006, pp ). The concept of entrepreneurialism 1 began making its way first into the private sector and later into the public sector. Entrepreneurialism advocates innovative thinking and provides managers with the power to effect changes that increase efficiency, effectiveness, and customer satisfaction. In the public sector, entrepreneurialism promotes concepts such as competitiveness in the public sector and adoption of private sector approaches to management. Osborne and Gaebler (1992) captured these concepts and laid out a path to achieve these goals in their book Reinventing Government. Their book was recognized by many political leaders of the time as a roadmap for fixing much of what was wrong with government. President Clinton commented, This book gives us the blueprint (p. 1). So influential was the book that it led to the publication of the National Performance Review in 1993 (Gore, 1993). While entrepreneurialism became the focus for much of the Clinton Administration, Congress sought to implement a government that, while capturing the spirit of entrepreneurialism, was more structured and focused on tangible objectives. Through the passing of several legislative acts, Congress established a framework based on the concepts of performance-based management and performance-based budgeting. Perhaps the most significant piece of legislation related to this effort was the Government Performance and Results Act of 1993 (GPRA). The GPRA has been one of the most enduring and arguably the most successful performance focused legislation written by Congress. It directs agencies within the federal 1 Term first coined by Gordon & Milokovich (1998) Public administration in America, p. 167

10 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 8 government to set goals, define objectives, and measure performance that is tied to results and tied to agency budgets. Because of the significance of the GPRA and its ties to an agency s funding, numerous studies have been conducted to assess the implementation, effectiveness, and challenges associated with the act. Few studies, however, have examined the GPRA mandated products produced by federal agencies to determine their ability to meet the entrepreneurial and performance vision Congress engrained into the GPRA. The premise of this study contends that the design of Agency Strategic Plans and performance measures are inadequate and fail to provide the guidance and structure required to effect leader decision-making. Additionally, Strategic Plans and performance measures fail to provide adequate scope, detail, guidance, and tools needed to integrate with an agency s budget- and performance-based management systems. This study closely examines the content and structure of selected Agency Strategic Plans and Agency Performance Plans to determine if the structure, design, and content of these plans are conducive to supporting leader-based decision-making and providing for integration into a comprehensive budget-based and performance-based management system. Specifically, as the central question, the study asks, Are Agency Strategic Plans and Agency Performance Plans designed to facilitate effective and efficient results-oriented performance improvements at the agency level and at a program level? In support of the central question, are the following subquestions: (1) do Agency Strategic Plans and Agency Performance Plans have procedures in place to integrate with their performance and budget management systems; (2) do Agency Strategic Plans target core programs within the agency or are they focused on broad agency missions associated with legislative charters; and (3) do Agency Performance Plans capture

11 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 9 results-oriented performance indicators that represent progress towards efficient and effective achievement of goals? II. A Background on Performance Budgeting in the United States A true understanding of the GPRA requires an understanding of the history of performance budgeting concepts in the United States. Long before the enactment of the GPRA, Congress and presidential administrations realized that the growing complexity of government required additional controls. More importantly, the growing cost of government required greater accountability and oversight (Joyce, 2003, pp. 6-9). Early in the twentieth century, Congress began developing performance budgeting legislation to provide accountability and control over agency budgets. The first legislation of this type enacted by Congress was the Budget and Accounting Act of Since the passing of the Budget and Accounting Act of 1921, presidential administrations have been required to submit an annual budget to Congress to fund the federal government. In comparison to the complexity of today s budget management programs, the Budget and Accounting Act was quite simple. It had a primary requirement for federal agencies to account for line-item expenditures. However, with the growth of the federal government, simple line-item accounting proved inadequate to meet a growing need. To address this need, the Roosevelt administration formed the President s Committee on Administrative Management (p. 9). Also known as the Brownlow Committee, the focus of this activity was to identify how the ever-growing federal government could be better managed. From what was first a general overview of administrative management from the Brownlow Committee, grew the concept of performance budgeting (Tyler & Willand, 1997, pp ). Performance budgeting was first formally proposed by the Hoover Commission in Unlike line-item budgeting that had been used previously, performance budgeting focused

12 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 10 more on what the government does, rather than the things it buys (p. 195). This shift from expenditures only to government services and performance called for a more aggressive and hands-on approach to management. The concept of performance budgeting matured through the 1950s into the early 1960s and culminated with the development of the Planning-Programming- Budgeting System (PPBS) under the Johnson Administration. For the first time, budget and management systems depended on each other to tie resources with results (Joyce, 2003, p. 9). While this approach proved to be effective in the disciplined structure of the Department of Defense, it proved to be less effective in other agencies within the federal government. The concept of tying resources to results, however, was not put to rest with the demise of PPBS. Both the Nixon and Carter Administrations in the 1970s sought to build upon the lessons learned by their predecessors to implement a results-oriented budgeting system. In establishing Management by Objectives, the Nixon Administration went beyond the PPBS concepts of managing inputs and outputs to also include outcomes. Outcomes, known as objectives, were established as performance targets for federal agencies with budget expenditures measured against results (Joyce, 2003, pp. 8-10). These concepts matured even further under the Carter Administration with implementation of Zero-Based Budgeting (ZBB). Under ZBB, agencies were required to justify their program budget requests each year in order to receive continued funding. The focus of such tight budgetary scrutiny was driven by President Carter s goal to balance the federal budget during his first term (Schick, 1978, pp ). While ZBB was a results-oriented budget management system, most felt it was too singularly focused on efficiency and balancing the budget. In many ways, ZBB lost sight of other outcomes such as quality of service and long-term program effectiveness.

13 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 11 During the Reagan and Bush (George H. W.) Administrations, no formal budget management systems were implemented for federal programs. This lack of a unified management system laid the groundwork for the sweeping budget reforms of the 1990s. During the late 1980s and early 1990s in both the private and public sectors, reinvention seemed to be the buzzword driving management reform. In the federal government, the Clinton Administration initiated this reinvention with the National Performance Review (NPR) (Thompson, 2001, pp ). The NPR called for entrepreneurialism in government that focused on decentralization, employee empowerment, quality of service, efficiency, and effectiveness. Their efforts in performance budgeting reform began with the implementation of the Chief Financial Officers Act of 1990 (CFO Act). For the first time, Congress legislated a relationship between performance and budget expenditures. The CFO Act also mandated a greater oversight role for the Office of Management and Budget in the planning and execution of these budget expenditures (GAO, 1991, pp. 1-10). Perhaps most significant to other budget reforms of the 1990s was the naming of the twenty-four agencies (fourteen cabinet-level departments and ten independent agencies) required to report under the act. This significance was highlighted in the requirements outlined in the GPRA. The GPRA and the CFO Act were not the only budget-reform pieces of legislation enacted by Congress during the 1990s. During this decade, Congress also implemented a number of bipartisan legislative acts as part of their budget-reform initiatives. In addition to the GPRA and the CFO Act, Congress also enacted the Federal Acquisition Streamlining Act of 1994, the Federal Financial Management Improvement Act of 1996, and the Information Technology Management Reform Act of Collectively, these legislative acts sought to

14 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 12 bring an ever growing and ever more expensive government under control and accountable to Congress, the presidential administration, and the American people (Joyce, 2003, pp. 8-10). The GPRA was the centerpiece of this effort. Through implementation of the GPRA, Congress sought to establish a long term, results-oriented, and efficiency-focused management framework for federal agencies that ties government performance to their budgets regardless of the administration in office (Brass, 2012, pp. 1-3). At the heart of the GPRA were requirements that direct agencies to develop performancebased budgeting systems that define goals and objectives as well as measure performance toward achieving these goals. Unlike other attempts at performance-based budgeting in the past, the GPRA goes beyond focusing singularly on budgets. The GPRA also provides additional guidance for using the resources required by the acts in management decision-making. The GPRA also places programmatic management requirements on agencies that exceed those typically associated with performance-based budgeting programs (Moynihan, 2012, pp ). The combination of performance-based budgeting guidance and guidance typically associated with programmatic management systems resulted in a hybrid management model to assist in both budget management and in overall program management. Since the enactment of the GPRA, numerous studies have evaluated the effectiveness of the GPRA in achieving the goals outlined in the legislation. These studies have shown that the GPRA has been successful in addressing transparency and many budget management concerns; however, studies have also shown that this hybrid construct of the GPRA has resulted in limited success as a programmatic management decision-making tool for most federal agencies. Congress realized that though the GPRA showed some measurable effectiveness, it had fallen short as a management system and as a tool in management decision-making. To improve upon

15 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 13 these weaknesses, Congress passed the GPRA Modernization Act of 2010 (GPRAMA) (Brass, 2012, pp. 1-6). Congress has set high expectations in the ability of the GPRA and GPRAMA to facilitate agency management and performance (Moynihan, 2012, pp ). To meet these expectations, Strategic Plans and performance measurements must be effective as management decisionmaking tools. They must provide more than measurements of inputs, outputs, and simple outcomes. They must also provide for an evaluation into the effectiveness and efficiency of programs that produced the performance measurements (Bingman, 2006, pp ). Rossi, Lipsey, and Freeman (2004, pp ) define performance measurements of agency inputs, outputs, and simple outcomes as formative performance measurements. For performance measurements to be actionable in management decision-making, they must assess effectiveness and efficiency in achieving outcomes. Effectiveness and efficiency are determined when inputs and outputs are assessed against outcomes. The resulting metrics of the assessment are known as summative performance measurements (pp ). Similarly, for the materials and resources produced under the GPRA/GPRAMA to be used in leader decision-making as part of the agency s comprehensive management systems, they must integrate with the management system and provide summative performance measurements that evaluate effectiveness and efficiency down to the program level. III. Conceptual Framework Over the last twenty years, numerous studies have been conducted to determine if federal agencies have developed Strategic Plans and performance measurements that are results oriented, capable of measuring efficiency and effectiveness of agency programs, and capable of being used in leader decision-making. The studies primarily focus on three key areas: 1) use of GPRA

16 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 14 resource in the management of federal agencies; 2) use of GPRA performance measures to assess efficiency and effective; and 3) use of GPRA Strategic Plans and performance measures for programmatic evaluations. Each group of studies represents insight into significant aspects of the GPRA as they relate to one of the three focus areas. Collectively, the following studies help to define background and direction of this study. Use of GPRA Resource in the Management of Federal Agencies Since 1997, the Government Accountability Office (GAO) has conducted surveys every three to five years to assess the implementation and effectiveness of the GPRA (Ellig, 2009, pp. 1-3). The GAO surveys go out to managers and SES executives in federal agencies to ask questions related to management s use of GPRA resources, integration of GPRA resources into agency management processes, and the use of GPRA resources in leader decision-making. 2 A significant number of the studies were conducted by the GAO as part of their Management for Results series of studies. 3 Collectively, these studies show overall improvements in utilization of GPRA resources by managers since the implementation of the GPRA. However, survey results continue to indicate that less than half of the managers in federal agencies integrate GPRA products in their program management systems (Ellig, 2009, p. 2). Studies also indicate that there continues to be a lack of support and endorsement from senior agency executives (GAO , 2013). In addition to the GAO, several academic researchers also use GAO surveys as a source of data to assess the overall effectiveness of the GPRA as a management tool in federal agencies. 2 Studies using the GAO surveys in part or in whole as a source of evaluative data include Bingman, 2006; Ellig, 2009; Ellig, 2010; GAO-04-38, 2004; GAO , 2013; GAO , 2013; GAO , 2013; GAO , 2013; Joyce, 2003; Moynihan & Lavertu, 2012; and Young & Rainey, The GAO developed a series of reports that evaluate various aspects of the GPRA and the GPRAMA. Known as the Managing for Results series, the GAO studies include GAO-04-38, 2004; GAO , 2013; GAO , 2013; GAO , 2013; and GAO , 2013.

17 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 15 In most instances, these studies are much more critical than those conducted by the GAO. For example, Ellig (2010) highlights that one of the key elements that the surveys repeatedly identify is the fact that performance measures associated with agency programs are far removed from, and maybe even unrelated to, the over-arching measures articulated in the agency s GPRA documents (p. 3). The significance of this finding cannot be overstated. If performance measures do not measure performance that is relevant to program managers then managers will not find use for the performance measures. To be effective, performance measures must provide feedback directly related to agency programs. Ellig argues that without such measures, it becomes extremely difficult to determine the effectiveness and efficiency of these programs. Similarly, Moynihan and Lavertu (2012) suggest that analysis of GAO surveys indicates that significant effort is being made to define performance measures and to collect the data associated with these measures. However, there seems to be less effort and less leadership emphasis on defining how the data is useful to leader decision-making (pp ). They maintain that the design of the GPRA places emphasis on goal and data creation (p. 598) and not necessarily on how data will be used by managers. They also contend that use of performance data is directly related to how well performance measures are linked to causal inferences (p. 600). Use of GPRA Performance Measures to Assess Efficiency and Effective A number of studies argue that the GPRA is too encompassing and complex. These studies suggest that the GPRA should narrow its scope to focus on performance budgeting only. 4 The studies focus on examining GPRA Strategic Plans and performance measurements narrowly in their use as performance-based budgeting tools. They emphasize that to be effective as a 4 Several studies examined the performance and effectiveness of the GPRA in its capacity as a performance-based budgeting tool focusing only on cost efficiency and cost effectiveness. These studies include Brass, 2012; Jones & McCaffery, 2010; and McNab & Melese, 2003.

18 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 16 performance-based budgeting tool, the GPRA should limit measurements to areas related specifically to efficiency and cost effectiveness. For the efficiency requirement of the GPRA to be realized, resources must be linked to results (McNab & Melese, 2003, p. 75); meaning, performance measurements must be able to measure the costs required to achieve a result. To do this, McNab and Melese (p. 76) argue that the GPRA must also perform as an integral part of the agency s overall budget-management system. They highlight that a budget-management system that measures efficiency and effectiveness requires summative performance measurements. Efficiency is derived from measurements of inputs required to achieve outputs. Inputs and outputs are then assessed against the desired outcome to determine effectiveness. One of the biggest obstacles that McNab and Melese (2003, p. 78) see with utilization and implementation of the GPRA is that current budget systems reward inefficient expenditures. For example, managers that spend 100% of their budget, without regard to efficiency and effectiveness, are likely to receive at least the same or perhaps more than the previous year s budgets. Managers that reduce costs are likely to be rewarded with a reduction in their budgets. In short, the desired behaviors of efficiency and effectiveness that Congress wished to implement through the GPRA cannot be realized until the system for rewarding positive behaviors, the system of accountability, can be corrected. Similarly, they also believe that the current structure is too broad and top-down oriented (p. 82). To be successful, they argue the system needs to be more flexible to account for variability in agency missions down to directorate and program levels. The Congressional Research Service agrees and contends that the GPRA has fallen short of providing assessments of efficiency and effectiveness at a programmatic level (Brass, 2012). Brass emphasizes that performance measurements do not dig deep enough into individual

19 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 17 program elements to measure the effectiveness and efficiencies of individual programs. He argues that the requirement for an outcome focus does not guarantee that a program is doing the process effectively (pp. 2-3). He points out that to be truly effective, a management system should provide for performance measures of inputs, outputs, and outcomes as well as provide for in depth evaluations of individual program elements. To determine efficiency, most studies argue that it is critical that the benefit derived from the budget investment be measured. Jones and McCaffery (2010, p. 508) found that the complexity of the design of GPRA plans and measurements affects its implementation and use. They emphasize that focusing on budget only, simplifying design, and providing for incremental growth over time reduces errors and improves utilization. Conversely, complexity can be laborious and have a tendency to lead to too many opportunities for failure, which, in turn discourages use by managers (p. 509). To achieve simplicity, the study asserts that agencies should focus less on overall program management goals with the GPRA and focus only on efficiency, effectiveness, and cost reduction. Efficiency also involves the elimination of redundancy. The GPRAMA requires crossagency collaboration to help reduce overlap and eliminate duplicative programs (Brass, 2012). Several studies focused on cross-agency collaboration, goal setting, and performance measurements to assess the efficiency and effectiveness of these efforts. For example, the GAO conducted a number of studies to assess the implementation of the GPRAMA to determine the effectiveness of some of its specific requirements. 5 In particular, the GAO studies sought to determine if changes specified in the GPRMA have had an impact on agency implementation of GPRA requirements. Overall, these studies found that little had been done to improve cross- 5 The GPRAMA included guidance on cross-agency collaboration. Several studies examined agency implementation of these requirements to include GAO , 2012; GAO , 2013; and GAO , 2013.

20 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 18 agency collaborative issues, using GPRA resources in leader decision-making, and engaging Congress in agency performance-management efforts. The Administrative Conference of the United States (ACUS) also examined the implementation of cross-collaboration efforts (Fountain, 2013). The ACUS study found that while cross-collaboration in and of itself is important, to be effective, the process must also include a cost benefit (p. 97). The study found that cross-collaboration efforts too often attempted to follow an Office of Management and Budget (OMB) one-size-fits-all approach. The study noted that the plans implementing cross-collaboration and the related reporting tended to be too complex (pp ). The study emphasized that plans implementing crosscollaborations need to be agile and light (p. 99), and to achieve a cost benefit, each collaboration requires customization to maximize efficiency and effectiveness. Use of GPRA Strategic Plans and Performance Measures for Programmatic Evaluations Another aspect of the GPRA that has been studied is tied to its use in agency program evaluation. Part 6 of OMB Circular A-11 (2013) defines how agencies should develop and implement Strategic Plans and performance measures. The guidance suggests a more comprehensive system of measurements that includes detailed programmatic evaluations to determine which programs are most effective. Several studies examined the use of Strategic Plans and performance measurements as part of agency program evaluations. 6 The GAO draws distinction between performance measures and program evaluation (GAO SP, 2011). They define performance measurements as the ongoing monitoring and reporting of program accomplishments (p. 2). In contrast, program evaluations are defined as detailed systematic 6 Several studies examined the use of strategic plans and performance measurements as part of agency program evaluations. Studies include Bingman, 2006; Brass, 2012; GAO-04-38, 2004; GAO , 2013; Ellig, 2010; Kravchuk & Schack, 1996; Lavertu, Lewis, & Moynihan, 2013; McMurtry, 2005; Young & Rainey, 2005; Walker, 2013; and Walker & Jung, 2013.

21 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 19 studies conducted periodically or on an ad hoc basis to assess how well a program is doing (p. 2). These broad definitions seem to place the requirements outside the scope of the GPRA; however, the GAO also defines four specific types or subcategories of program evaluations that show significant similarities with the requirements outlined in the GPRA. For example, implementation evaluations, outcome evaluations, impact evaluations, cost-benefit evaluations, and cost-effectiveness evaluations direct the gathering of information down to organizational levels that is very similar to the performance measurements required under the GPRA (GAO/GGD , 1996). To provide leader decision-making capabilities, the design of both requires the metrics derived from the processes to be more summative in nature (Bingman, 2006, pp ). In an early study by the GAO, the quality of Strategic Plans and performance measurements from federal agencies were assessed against survey results to determine the relationship of plan quality on utilization (GAO-04-38, 2004, pp ). In this study, the GAO observed consistent improvement over the years in the quality of these resources. They determined that objectives were becoming more results-oriented and more effective in addressing accountability for agency performance. However, the GAO also identified areas in which improvements were still needed (pp ). One of the key issues identified in this study was the lack of aligning strategic goals with agency programs and processes. The study determined that the lack of linking of GPRA resources to agency program processes is a contributing factor in the lack of leader utilization of GPRA resources in planning and decisionmaking. The GAO determined that because GPRA performance measures gather data on processes outside a manager s framework of reference, managers have difficulty in determining applicability of GPRA resources to their programs.

22 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 20 In a similar study, McMurty (2005) outlines GPRA implementation progress for each administration since the enactment of the GPRA. He points out that the GAO has consistently reported that while agencies continue to improve in the development of Strategic Plans and performance measures, they fail to implement the evaluation component (pp. CRS-10). Similarly, Bingman (2006) focused on targeting the ability of Agency Performance Plans to provide evaluative information. He argued that agencies respond to the intent of the act and their legislative charters but fail to design in evaluative metrics in their Agency Performance Plans (p. 143). Bingman argues that the GPRA is more than just a legislative requirement. He emphasizes that the GPRA directs managers to develop products that are comprehensive and integrated (p. 144) with management systems within their organization. He asserts that a comprehensive system of measurements that take a penetrating look at the performance of major public programs (p. 146) is needed for the GPRA to be effective. The studies listed above primarily rely on GAO surveys as a tool to identify deficiencies in implementation and utilization of GPRA directed resources. Most of these studies did not examine the GPRA resources themselves to determine if inadequacies in their design were a factor driving GPRA utilization and implementation. Several studies look beyond utilization surveys to focus on the design of GPRA and GPRAMA products. 7 An early study attempted to define design principles for effective performance-measurement systems (Kravchuk & Schack, 1996, p. 349). Although the study examined performance-management systems in general and did not specifically focus on GPRA issues, the general assessment of best practices in regards to engineering performance-measurement systems outlines a framework that is applicable to GPRA 7 Several studies look beyond utilization surveys to focus on the design of GPRA and GPRAMA products. Studies include Ellig, 2010; Kravchuk & Schack, 1996; Lavertu, Lewis, & Moynihan, 2013; Young & Rainey, 2005; Walker, 2013; and Walker & Jung, 2013.

23 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 21 documents and resources (pp ). This early study found that comprehensive performance-management systems are, by their nature, complex. The study asserts that agencies acknowledging this complexity can engineer performance measurement systems that consider complexity in their design. Perhaps most significant, the study emphasizes that for performance measurements to be relevant, leaders must have an intimate sense [into] the nature of the system which they monitor (p. 350). Walker & Jung (2013) also examined content and structure of Strategic Plans and performance measurements. Their study examined how design and structure affected perceptions of how performance-measures should be used in the management of federal agencies. In particular, they focused on the differences between political perceptions and managerial perceptions (pp ). They found that the audience to which the plan and measures were written significantly effected perception and how the resources would be used. The study also identified the need to align goals, objectives, and data perceptions of the political audience with those of the managerial audience. Perhaps one of the most comprehensive studies into the design of strategic goals was conducted by Young and Rainey (2005). They identified numerous studies that mostly led to the same conclusions regarding the somewhat ineffective nature of federal agency goal setting and performance measurement. They also found that little research into the effectiveness of strategic goals had been conducted. They argued that most studies relied on surveys given to agency managers without examining GPRA product design as a possible reason for their findings (pp ). Their study surveyed managers in federal agencies to specifically assess their perception of goal ambiguity. The study found that the ambiguous nature of strategic goals allowed for significant variation in interpretation of goals. Young and Rainey contended that

24 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 22 most goals were so ambiguous that they lose their significance and applicability to an agency s actual mission. They found that managers were aware of these issues and desired more clarity. They highlighted that agency managers desired clearer goals to better determine agency performance. Additionally, agency managers also desired targeted performance-management tools that could better assist in management decision-making. Even with these studies into the structure and content of Strategic Plans and performance measures, questions regarding the design of Strategic Plans and performance measures remain. Because most studies relied heavily on survey results instead of examining GPRA source documents, they failed to adequately assess the effect design of Strategic Plans and performance measurements have on providing summative results and in facilitating integration with an agency s comprehensive management system. As outlined in the GPRA and in numerous studies, for the materials and resources produced under the GPRA to be used in management decision-making, they must function effectively as part of a comprehensive management system. For this reason, additional study is needed to assess the effect the design of Strategic Plans and performance measures have on the utilization of GPRA related products and resources. More importantly, additional study is needed to determine the effect design has on integrating the GPRA into the agency s comprehensive budget- and performance-management systems. IV Methodology To assess the effects Strategic Plan and performance measure design has on providing a results-oriented tool for leader decision-making and on integrating the GPRA into the agency s comprehensive budget-based and performance-based management systems, this study uses qualitative research methods to examine the Strategic Plans and Agency Performance Plans of twelve selected federal agencies. It assesses the individual elements within these plans based on

25 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 23 their applicability for use in programmatic performance evaluation and on their ability to measure efficiency and effectiveness. The study also examines the effects agency policy focus, independence, and underlying mission has on implementing GPRA requirements. Design and Instrumentation One of the most significant challenges in assessing the structure and content of Strategic Plans and performance measures is the lack of a specific management model or implementation framework that federal agencies must adhere to in their implementation of the GPRA. Though OMB Circular A-11 provides some guidance, it does not provide a model in which an analysis of Strategic Plans and performance measures can be made. Fortunately, most agency frameworks and implementation guidance has evolved from GAO executive guidance published in 1996 (GAO/GGD , 1996). The GAO guidance outlines twelve practices that encompass four requirements related to: 1) defining an agencies mission and desired outcomes, 2) measuring performance, 3) using performance information, and 4) defining leadership practice that reinforce GPRA implementation. This guidance has resulted in a set of practices that most federal agencies use to integrate the GPRA into their management frameworks. Many agency programs also show evolution from well-established performance-based management models such as The Balanced Scorecard (Kaplan & Norton, 1997). As with the Balanced Scorecard approach, these agency plans focus on interconnected business perspectives (Artley & Stroh, 2001, p. 19) that include finance, customer service, internal business practices, learning, and growth. This study draws from these established performance concepts in the development of its qualitative analysis tools. It also draws from concepts outlined in performance-based management tools and models such as Lean Six Sigma (Price, Mores, & Elliotte, 2011) and ISO 9001 (Baranzelli, 2010). As with the Balanced Scorecard

26 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 24 approach, these tools define a model of continuous improvement that requires integration of core programs and linking to strategic plans and objectives. Perhaps one of the most comprehensive frameworks specifically developed to implement the GPRA was the Department of Energy s (DOE) six volume series titled The Performance- Based Management Handbook, (Artley, Ellison, & Kennedy, 2001). In this series, the DOE captures performance-based management concepts associated with the National Performance Review and the Balanced Scorecard as well as other key performance-based management concepts. The guidance captures these concepts in the following six elements that the DOE defines as needed for an effective performance-based management system: 1) defining organizational mission and objectives, 2) integrating the performance measurement system, 3) requiring accountability, 4) collecting data and assessing performance, 5) analyzing performance and reporting results, and 6) using performance information to drive continuous improvement (p. V). The result is a management framework that evolves and shapes recognized performancebased management practices around the requirements of the GPRA. The DOE also developed specific guidelines in support of this framework to be used in developing Strategic Plans and performance measures. The DOE publication titled Guidelines for Strategic Planning (1996) provides detailed guidance for developing Strategic Plans that integrate agency mission, vision, and goals with program planning and budgeting. Similarly, the DOE publication Guidelines for Performance Measurement (1996) provides guidance in the development of effective performance measures that provide insight into an agency s performance down to operational levels. This study uses these two guides as a framework in developing the qualitative evaluations used to assess Agency Strategic Plans and performance measures.

27 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 25 While the DOE guidance defines a framework for developing a qualitative analysis system, it falls short of providing descriptive guidance in which qualitative assessment values can be designed. In their book Government Performance and Results: An Evaluation of GPRA s First Decade, Ellig, McTigue, and Wray (2012, pp ) define a series of best practices that should be followed to develop effective objectives and performance measures. This study applies these best practices against the DOE s performance-based management framework to provide qualitative assessment metrics that assess Agency Strategic Plans and Agency Performance Plans. The results of the analysis are drawn upon to answer the central question and the sub-questions. To examine the affect policy focus, independence, and underlying mission has on implementing GPRA requirements, the analysis examines the data: (1) collectively for all agencies; (2) in groupings based on an agency s policy focus; and (3) in groupings based on whether an agency is independent or subordinate to a cabinet level department. Documents and Artifacts This study examines the Strategic Plans and Agency Performance Plans of twelve federal agencies based on the agency s policy focus (e.g. distributive policy focus, redistributive policy focus, or regulatory policy focus). Distributive agencies are agencies that use general tax revenue or other nonuser taxes to provide benefits directly to individuals (Meier & Bohte, 2007, p. 99). Redistributive agencies use tax money collected from one category of citizens and redistributes it to another category in the form of a benefit or entitlement (p. 92). Finally, regulatory agencies restrict the conduct of individuals and organizations to keep conduct from transcending acceptable bounds (p. 76). While no research has specifically focused on identifying trends related to policy focus, several studies have suggested that this may be a

28 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 26 factor. For this study, four agencies from each of the policy focus areas are selected to determine the possible effect policy focus may have on the design and quality of Strategic Plans and performance measures. Another issue that is touched on but not well explored is the location of an agency within a cabinet level department. Some federal agencies, such as the Environmental Protection Agency, are independent. The missions of subordinate organizations within many independent federal agencies are interrelated to each other and tightly integrated with the overall mission of the agency. Others federal agencies, such as the National Park Service, are subordinate to cabinet level departments and represent only a portion of that department s mission. These federal agencies may have unrelated missions and policy focus to sister agencies within their department and their specific mission may only be loosely related to a sister agency s overall mission. This study groups agencies based on their independence and examines this aspect to determine the possible effect it may have on the design and quality of Strategic Plans and performance measures. The GPRAMA requires OMB to develop a web site as a single point of entry for all Agency Strategic Plans and Agency Performance Plans. OMB s implementation at Performance.gov provides this entry point but falls short of providing a single point of access for GPRA required documents. Instead of providing all GPRA resources at a single location, Performance.gov serves as a tool that redirects users to GPRA and budget resources located on individual agency web sites. While this approach may answer the specific requirements of the GPRAMA, it falls short of the common interface envisioned by Congress. Because of this approach taken by OMB, the collection of data required for the study necessitated extensive analysis of individual agency web sites, their Agency Strategic Plans, and their Agency

29 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 27 Performance Plan. The complete list of documents and artifacts used in the qualitative analysis conducted for this study is located in Appendix A. The twelve agencies examined in this study are listed in Appendix B. All the agencies, either directly or as part of a cabinet level department, are required to report their performance based on the requirements of the CFO Act and the GPRA. Four of the agencies selected have a distributive policy focus. The first agency with a distributive policy focus is the National Aeronautics and Space Administration (NASA). NASA is an independent agency that has a broad mission to advance science and technology and to explore space (NASA, 2013). The second distributive agency is the National Science Foundation (NSF). The NSF is an independent agency that has a mission to promote the progress of science (NSF, 2014). Third in the list of agencies with a distributive policy focus is the National Park Service (NPS). The NPS, a subordinate agency within the Department of Interior (DOI), is entrusted with the mission of caring for the nation s national parks (NPS, 2014). The final agency with a distributive mission is the only cabinet level department examined in this study. The Department of Veteran s Affairs (VA) is responsible for managing the systems of service and support for the nation s veterans (VA, 2013). Four of the agencies examined as part of this study have a redistributive policy focus. The first agency with a redistributive policy focus is the Employment and Training Administration (ETA). ETA is a subordinate agency in the Department of Labor (DOL) that administers federal government job training and worker dislocation programs (ETA, 2013). The second agency with a redistributive policy focus is the Federal Housing Administration (FHA). The FHA is a subordinate agency in the Department of Housing and Urban Development that facilitates mortgages from FHA approved lenders by guaranteeing and

30 STRATEGIC GOALS AND PERFORMANCE MEASUREMENTS 28 insuring loans (HUD, 2009). Third in the list of agencies with a redistributive policy focus is the Food and Nutrition Service (FNS). FNS is a subordinate agency in the US Department of Agriculture (USDA) charged with helping to ensure the healthy nutrition of all children (USDA, 2009). The final redistributive agency in this study is the Social Security Administration (SSA). The SSA is a subordinate agency in the Department of Health and Human Services (HHS) that provides services to the nation s elderly and those with disabilities (SSA, 2013). It should also be noted that even though SSA is subordinate to HHS, it is required under the GPRA to provide its own Agency Strategic Plan and Agency Performance Plan independent of HHS. The final four of the agencies examined as part of this study have a regulatory policy focus. The first agency with a regulatory policy focus is the Occupational Safety and Health Administration (OSHA). OSHA is a subordinate agency in the Department of Labor (DOL) that enforces the safe and healthful work environment for the nation s workers (OSHA, 2014). The second agency with a regulatory policy focus is the Food and Drug Administration (FDA). The FDA is a subordinate agency in HHS that oversees the quality and integrity of the nation s food and drug programs (FDA, 2013). Third in the list of agencies with a regulatory policy focus is the US Forest Service (USFS). The USFS is a subordinate agency in the USDA that protects and manages the nation s National Forests and grasslands (USFS, 2014). It should be noted that the USFS has multiple areas of policy focus. Some areas of the agency policy focus are distributive in nature while other areas are of a regulatory focus. This study categorizes USFS as an agency with a regulatory policy focus. The final regulatory agency in this study is the Environmental Protection Agency (EPA). EPA is an independent agency that regulates the environmental quality of the nation s air, water, and land (EPA, 2010). EPA is also the only independent agency that does not have a distributive policy focus.