HPC Budget Allocation Map: Industry Averages

Transcription

1 HPC Budget Allocation Map: Industry Averages Christopher G. Willard, Ph.D. Addison Snell Laura Segervall March 2012 EXECUTIVE SUMMARY surveyed the High Performance Computing user community to complete its fourth Site Budget Allocation Map, a look at how HPC sites divide and spend their budgets. We surveyed users on their spending in seven top-level categories: hardware, software, facilities, staffing, services, utility computing, and other. Each category was further divided into constituent subcategories, resulting in 25 unique items included in the analysis. This report contains the key data, along with analysis, conclusions, and guidance. This report provides the average budget distribution for the responding sites within each category and provides a view of the entire HPC budget distribution and IT product spending (excluding facilities and staffing). Important statistical highlights from this study include: Hardware remains the largest category even as its share decreases each year. The average HPC site spent 36% of its HPC budget on hardware this year. Servers lead spending in this category, followed by storage and clients. Staffing continued as the second-largest overall expense. System management and operations combined with maintenance personnel accounted for 56% of the staffing budget. Facilities expenditures represent on average 13% of a site s HPC budget, and this is the only major category that has increased in each consecutive survey. We estimate that, on average, at least another 5% is paid for facilities items out of another budget. About 36% of respondents stated that their facilities were not in budget (NIB), and many of those who had facilities in their budget did not pay every item (power, cooling, building/floor space). Software share remains steady at about 15% of the total HPC budget. Software tools accounted for the largest share within software, with 24% share of the HPC software budget. Less than half (43%) of all respondents expect their budgets to increase over the next two years, and 44% expect little or no change in budgets. This is a shift from the last few years where over 60% expected an increase of at least 5% or more. Most of this uncertainty is at academic sites and to some degree at government sites; commercial sites are still relatively healthy. Uncertainty in funding at academic and government sites for scientific research as well as the slowness in reaching a complete recovery from the global recession may be contributing to the more conservative expectations Information from this report may not be distributed in any form without permission from.

2 TABLE OF CONTENTS EXECUTIVE SUMMARY... 1 INTRODUCTION... 3 METHODOLOGY... 3 Survey Methodology... 3 Analysis Methodology... 4 Coding Issues... 5 RESULTS AND ANALYSIS... 7 Top-Level Budget Allocation... 7 Top-Level Averages Excluding NIB Hardware Budget Allocation Estimated Additional Spending Software Budget Allocation Facilities Budget Allocation Staffing Budget Allocation Services Budget Allocation Cloud/Utility/Outsourcing Computing Budget Allocation Predicted Budget Change HPC IT Product Spending CONCLUSIONS Noteworthy Numbers Guidance to Vendors Page 2 of Information from this report may not be distributed in any form without permission from.

3 INTRODUCTION The Site Budget Allocation Map presents information on how users distribute their HPC budgets between and within various categories, such as hardware, software, and facilities. The most recent survey was conducted in September 2011 via a web-based survey tool. The survey received qualified responses from 124 sites. The information gathered allows us to make several important inferences about the HPC industry. One of the most salient pieces of information is the proportion of budget spent on servers to budget spent on all categories of IT product spending. With this ratio, we can extrapolate from the total industry spending on HPC servers to the total size of the HPC market, including other hardware, software, and services. does not consider internal costs, such as budget spent on facilities or staff, to be constituents of the HPC market size. However, these costs are important components of total cost of ownership (TCO). Tracking them provides insights into whether facilities and staffing costs are increasing over time, and helps identify potential market opportunities for products that allow organizations to reduce spending in these areas. This is the fourth year we have completed the Site Budget Allocation Map report. This report provides the average spending by category and subcategory for the overall market for all the years surveyed. METHODOLOGY Survey Methodology In September 2011, repeated its Site Budget Allocation Map survey. We surveyed members of the HPC user community utilizing a web-based survey tool. The survey spanned users throughout industry, government, and academia. The questionnaire asked respondents how their current HPC budgets are divided, on a percentage basis, between various categories: hardware, software, facilities, staff, services, cloud/utility computing, and other. We then asked respondents to further divide each category into percentage spending for various subcategories. For example, we asked how the software budget is allocated between system software, middleware, tools, storage management, third-party applications, and in-house transfer costs. The survey concluded with questions on user site demographics, total budget size, and projected growth. The complete questionnaire can be found under Research Methodology at For any item or category, the respondent had the ability to reply not in budget (NIB), meaning that payment for that item or category would not come out of the HPC budget in question. In these cases, it is possible in some cases certain that additional money is being spent out of a different budget within the organization in support of HPC. For example, several respondents marked items in the facilities category as NIB. In these cases, it is safe to assume that power consumption is paid for by someone in the organization, but that these costs do not come out of the HPC budget. In 2011, began segmenting the entire HPC market into two super-segments: High Performance Technical Computing (HPTC) and High Performance Business Computing (HPBC). HPTC (formerly referred to as Traditional HPC) is the use of HPC tools and technology by scientists and engineers in Page 3 of Information from this report may not be distributed in any form without permission from.

4 research, development, and production across industry, government, and academia. HPBC is the use of HPC tools and technology in order to make a business decision or to optimize business performance. Table 1 provides the number of respondents surveyed in each year the survey was conducted. In every survey, more than 80% of the sites were classified as HPTC sites. This report will focus on the entire HPC market and as such will include both HPTC and HPBC sites. Please note that not all questions receive the same number of responses. Table 1: Number of Respondents by Survey Year All Surveys Total Respondents % HPTC 96% 86% 82% 89% 86% % HPBC 4% 8% 9% 10% 9% Unclassified * 0% 6% 9% 2% 5% *Not enough detail provided to classify Source:, 2012 repeats the Site Budget Allocation Map annually. Analysis Methodology For each question on budget allocation, calculates the average response for each category/subcategory. uses two different methodologies to calculate the average, based on the treatment of NIB responses. One method is to treat NIB responses as zeroes, calculating an average share for each category (e.g., software) or subcategory (e.g., third-party applications) based on total number of non-blank responses (i.e. zero responses are counted into the denominator when calculating the average). The advantage to this method is that it provides a view of spending at the overall organization level. The other method is to exclude NIB responses in calculating the averages, so that only respondents who actually listed a percentage allocation (including 0%) are included in the average. The resulting calculation gives an average spending proportion on an item (or category) at the responding budget-holder s level. These two averages are useful for extrapolating the amount of unseen HPC spending that may come from different budgets in the same organization. Page 4 of Information from this report may not be distributed in any form without permission from.

5 Example: User 1 allocates 20% of total budget to facilities, and User 2 allocates 10%. User 3 marks that item as NIB. We will average these values two ways: In one method, the NIB will count as 0%, and the resulting average is 10%. This is useful in creating a total map of all budgets represented in the study. In another method, the NIB is ignored, and the resulting average is 15%. This represents the average amount spent on facilities among the users who consider it to be part of their budgets. This number is useful in modeling what additional money might come from different budgets in User 3 s organization. In this report, we provide the average spending share (NIB=0) for each category/subcategory which totals to 100%. Future reports will utilize the second method to estimate additional spending which may come from different budgets in the same organization but not the respondent s budget. Within a category or subcategory, percentages will be normalized to adjust for differences in sample size for each item. Coding Issues In calculating averages, blank and zero are interpreted differently, yielding different results. To provide consistency amongst the sites, we established the following rules to reconcile differences found between the top-level category and subcategories at a few sites: Two different scenarios existed for users who marked the top-level category as NIB. 1. If a user marked the top level as NIB and then provided percentages at the second level, their responses in this category were still used to calculate the average distribution within the subcategory. We assume that the respondent had knowledge of the budget distribution even if the costs do not fall within his or her responsibility. 2. If a user marked a top level as NIB and then the user either did not respond or provided 0% at the subcategory level, we treated the subcategory items as NIB. A similar process was used for 0% responses and varied based on two scenarios. In all cases, 0% was included in calculating the distribution of budget within that subcategory. 1. If a user marked the top level as 0% and then provided percentages at the second level, their responses at the item level were still used to calculate the average distribution within the subcategory. 2. If a user marked the top level as 0% and did not answer the second level question or provided 0%, the items in the second level were considered to be 0%. Page 5 of Information from this report may not be distributed in any form without permission from.

6 The facilities category was a unique case. We assume that everyone needs to pay for building or floor space, power consumption, and cooling. Therefore, if a respondent stated 0% was spent on facilities, we treated this as NIB. It does not have an effect on the average spending calculations; however, it does and will impact any future estimates of total expenditures. Page 6 of Information from this report may not be distributed in any form without permission from.

7 RESULTS AND ANALYSIS After four complete Budget Map Surveys, the distribution of HPC respondents budgets reveals a fairly consistent pattern (see Figure 1). The largest segment is the $2 million and higher segment, accounting for about 25% of sites. This was followed closely by the $100,000 to $499,999 segment, which accounts for about 20%. The 2007 survey utilized a different survey methodology, resulting in a higher number of large sites and therefore a higher number of large budgets. Even with the difference in methodology, the ranking of the budget segments is consistent with the other survey years. Figure 1: Respondents Current Budget at HPC Sites, Including Servers, Software, Storage, and Services Source:, 2012 Top-Level Budget Allocation The first question of the survey asked HPC users to allocate their budgets, on a percentage basis, across seven major categories: hardware, software, facilities, staffing, services, utility/outsource computing, and other. Respondents had the ability to mark a category as not-in-budget (NIB) when payments for that category would not come from the HPC budget in question. Page 7 of Information from this report may not be distributed in any form without permission from.

8 The seven top-level spending categories were defined as follows: Hardware purchases and upgrades, including servers, storage, networks, clients, and other. Software purchases and upgrades, including O/S and systems software, middleware, applications, tools/libraries/compilers, in-house developed, and other. Facilities spending, including building/floor space, power consumption, cooling, and other. Staffing, including system managers, maintenance personnel, systems programmers, application programmers, user services consultants, and others. Services purchases, including maintenance and repair, external training, programming, and other. Cloud/Utility/Outsource: Purchases of computational capacity/capability through an external utility-based service, including raw cycles, applications support, and other. Other: Anything not covered above Table 2 provides the industry averages for the responses for the top-level budget allocations for the last four surveys. Figure 2 illustrates these averages in the seven categories over the last four surveys. Table 2: Top-Level HPC Budget Distribution, NIB included Category All Surveys Hardware 44.2% 40.9% 37.3% 36.0% 38.3% Software 15.7% 14.8% 14.5% 14.9% 14.8% Facilities 8.0% 10.7% 11.0% 13.2% 11.4% Staffing 23.1% 24.1% 26.2% 23.0% 24.5% Services 8.0% 5.9% 6.7% 8.9% 7.3% Cloud/Utility/Outsourcing 0.0% 2.9% 4.1% 3.3% 3.3% Others 0.9% 0.7% 0.4% 0.5% 0.5% Source:, 2012 Page 8 of Information from this report may not be distributed in any form without permission from.

9 Figure 2: Top-Level HPC Budget Distribution by Survey Year, NIB Included Source:, 2012 Hardware continues to be the primary expenditure in HPC budgets, with a 36% share, followed by staffing, with about 23%, and software, with 15%, for 2011 budgets. These three categories have held the same ranking over the four surveys. Three trends are notable between the surveys: 1. Hardware spending decreased The share of the HPC budget spent on hardware has declined in each consecutive survey, starting at 44% in 2007 and falling to 36% in Even if we combine the share spent on utility computing with the hardware share, a decline in share still exists (44% in 2007 and 39% in 2011). 2. Facilities spending increased Facilities is the only budget segment that increased each survey year, growing from 8% in 2007 to 13% in Software expenditures are constant Figure 2 illustrates the constant nature of spending on software. Over the last four surveys, this segment has fluctuated only about 1%, with an average just shy of 15%. With four studies completed, the shift in spending appears to be a trend rather than a result of differences in sampling. We see several factors contributing to this shift. They are: Shift of expenditures within hardware The economy has been anything but certain over the last few years. Combining this with rising power costs, we believe organizations may be focused on optimizing their existing infrastructure rather than expanding, utilizing tactics such as upgrading Page 9 of Information from this report may not be distributed in any form without permission from.

10 nodes. Within the hardware segment, client share of total hardware has increased over the four years (see analysis in the Hardware section). Both of these factors would lower the total expenditures for hardware. Rising energy costs for cooling and power Power consumption and cooling in 2011 account for about three-fourths of the facilities expenditures. According to the U.S. Energy Information Administration s (EIA) Short-Term Energy Outlook, electrical rates are forecasted to increase for both the industrial and commercial sectors through We expect facilities to continue to take a greater share of the HPC budget until the installed base transitions to lower power-consuming and heatgenerating systems. Top-Level Averages Excluding NIB We allowed respondents to enter NIB (Not in Budget) for categories and subcategories which were just that not in their budget. For some categories, if it isn t in the respondent s budget then it is probably in someone else s. Excluding those who stated a category was NIB in the average, rather than setting to zero, may more accurately represent the total spending for this category. For the top-level HPC budget categories, the facilities category has and has always had the highest NIB count due to our methodology. The high percentage of NIB for facilities illustrates the unique nature for paying facilities costs. Many users do not have any facilities expenditure in their budget as these costs are borne by the larger organization. In the 2011 survey, 36% of the sites stated that facilities was not-in-budget, followed by staffing with 11%. We estimate based on the average in-budget distribution shown in Table 3 that at least another 5% is paid for facilities out of another budget. Table 3. Top-Level HPC Budget Distribution, NIB excluded Category All Surveys Hardware 42.1% 39.0% 33.6% 32.4% 35.2% Software 14.4% 14.0% 13.1% 14.5% 13.8% Facilities 10.2% 13.9% 15.8% 18.1% 15.6% Staffing 25.1% 23.9% 26.4% 22.8% 24.6% Services 7.3% 5.7% 6.6% 8.5% 7.1% Cloud/Utility/Outsourcing 0.0% 2.8% 4.1% 3.2% 3.3% Others 0.8% 0.7% 0.3% 0.5% 0.5% Source:, 2012 Hardware Budget Allocation We asked HPC users how the hardware portion of their budget is allocated, on a percentage basis, between five items: servers, storage, networks, clients, and other. Page 10 of Information from this report may not be distributed in any form without permission from.

11 The five hardware spending categories were defined as follows: Server purchases and upgrades: all sizes and types of multi-user or special-purpose systems that are purchased through the respondent s budget. Storage hardware purchases and upgrades, including direct attached storage, NAS, SAN, etc., but not including storage software, such as file systems. Network purchases and upgrades. Client computer purchases and upgrades, including workstations, personal computers, handheld devices, and any other specialized equipment or interfaces. Other hardware: Anything not covered above. The hardware budget allocation responses for the last four surveys are summarized in Table 4. Table 4: Average HPC Hardware Budget Distribution by Survey Year, NIB included Category All Years Server 65% 50% 48% 45% 49% Storage 19% 27% 27% 24% 26% Network 10% 13% 13% 14% 13% Client 6% 10% 11% 15% 12% Other 0% 1% 1% 2% 1% Source:, 2012 Several changes in spending have occurred within the hardware budget over the past four years: Server share of hardware spending decreasing Hardware spending as a percentage of the total budget has declined since our first survey in The dramatic drop between 2007 and 2008 was primarily due to a higher percentage (56%) of sites in 2007 with budgets of $2 million or greater compared to other surveys, with 21% to 29%. Large sites typically have larger server configurations which shift the mix toward servers. However, the decline continued, albeit more slowly, in 2010 and 2011, with less than 50% of the hardware budget being spent on servers. Client share of hardware spending increasing At the same time as server spending was declining, client spending was increasing. The big increase of 4% between 2007 and 2008 can be attributed, in part, to the sample population differences; however, the increase continued in later surveys where the sample population did not vary as much, suggesting a shift to client purchases. Over the years, cores on the desktop have increased, allowing smaller jobs to be completed more efficiently on the desktop. Our Technology survey revealed that while the maximum cores used for a single job was at least 160 cores for half the sites, 40% of the jobs run at these sites utilize 4 cores or less, allowing some functions to be offloaded from the servers to the workstations and bringing more control to the user. Page 11 of Information from this report may not be distributed in any form without permission from.

12 Network share increasing gradually Network spending has been slowly increasing as a share, growing from 10% in 2007 to 14% in Data explosion/big data -driven increases in data volumes within data centers generate requirements for networks that are capable of handling greater amounts of data at all stages of HPC workflows. In addition, the transition to higher-speed networking protocols, requiring new network hardware as well as wiring, may be contributing to the increase in spending. Storage spending holding steady Other Intersect360 research indicates a continual increase in demand for storage systems capacity and capability. Yet the spending on storage appears to have reached a steady state. The decline in price of both storage media and systems in the past years have aided users ability to purchase more storage at the same price but should not contribute to a decline in spending. Additionally, storage purchases was one of the more frequently mentioned categories for future purchases in the open-ended question on budget trends. We believe that storage requirements are increasing at least at the rate of pricing declines and may actually outpace drops in cost going forward. We will be following this area closely in our future research. Estimated Additional Spending The top budget category was servers, which still accounted for slightly less than half of the hardware budget (45%). However, server share is declining, leading to uplifts in the remaining segments on a yearly basis. On an uplift basis, 1 each dollar of server spending in 2011 resulted in an additional 54 cents on storage, 32 cents on networks, and 33 cents on clients (with NIB included) (see Table 5). Table 5. Estimated Additional Spending (Uplift) for Every $1 Spent on Servers Item 2007 Uplift 2008 Uplift 2010 Uplift 2011 Uplift Storage 29 cents 54 cents 57 cents 54 cents Networks 16 cents 25 cents 26 cents 32 cents Client 9 cents 19 cents 24 cents 33 cents Source:, 2012 It is important to note that with 36% of the total budget going toward hardware, and 45% of hardware budget going to servers, that servers comprise only 16% of the total HPC budget, on average. Ignoring the internal spending categories of facilities and staffing, servers comprise 25% of external (i.e. product and services) spending. That said, servers remain the largest component in the HPC budget. Software Budget Allocation Respondents were asked how they allocate their software budgets among seven items: system software, middleware, software tools, storage management, third-party (ISV) applications, transfer costs, and other. 1 That is, estimating the amount of additional spending in each category associated with a given amount of server spending. Page 12 of Information from this report may not be distributed in any form without permission from.

13 The seven software spending categories were defined as follows: System software purchases, licenses, and upgrades, including operating systems, drivers, bundled hardware management tools, etc. Middleware purchases, licenses, and upgrades, including workload managers, scheduling, load balancing, grid management tools, system monitoring and accounting packages, etc. Software tool purchases, licenses, and upgrades, including compilers, libraries, profilers, debuggers, parallel computer environments (e.g., MPI), etc. Storage management software purchases, licenses, and upgrades, including file system licenses, volume managers, etc. Third-party (ISV) applications software purchases, licenses, and upgrades. Transfer costs for software developed in-house, including software developed by dedicated staff, consultants, etc. (This item covers any transfer costs that pay for software internally, but it does not include in-house software developers salaries. Salaries are covered under staffing.) Other software: Anything not covered above. The software budget allocation responses are summarized in Table 6. Software spending was well-distributed, with all six items receiving at least 10% of spending. Table 6: Average HPC Software Budget Distribution by Survey Year, NIB included Category All Years System Software 24% 27% 24% 16% 22% Middleware 16% 15% 14% 15% 15% Software Tools 20% 23% 20% 29% 24% Storage Management 10% 13% 13% 13% 13% 3rd Party ISV 28% 15% 18% 16% 17% Transfer Costs 3% 6% 11% 11% 9% Other 0% 1% 0% 0% 1% Source:, 2012 This year s study was the first time that software tools (29%) surpassed system software (16%) as the top software expenditure. Increasing core counts on processors, coupled with a higher number of installed GPUs, has increased the complexity of programming these systems to leverage the parallelism. The increase in spending on software tools suggests that respondents have found it necessary to purchase tools to aid with this complexity. Parallel programming tools, compilers, and other software tools are necessary software components in any truly authentic pasta dish sites that develop applications in-house. The significant drop in system software spending was unexpected. Three factors could contribute to this. First, system software is often bundled with the hardware with its cost included as part of the overall system. This requires the respondent to do the difficult task of estimating the split between hardware and software. Bundling Page 13 of Information from this report may not be distributed in any form without permission from.

14 creates a slightly high bias for hardware and a somewhat larger bias for software. It has been a constant factor in every Site Census survey so most likely is not contributing significantly to the decline. The second issue is that server spending is decreasing and therefore, the operating system expenditures would decline. In addition, to the extent that sites are upgrading existing nodes rather than expanding the number of nodes, OS licenses may not need to be renewed. Third, more sites may be shifting to open-source operating systems to reduce costs. In our site census, we have seen CentOS gaining installations amongst the various operating systems. Facilities Budget Allocation The facilities budget was divided into four items: buildings and floor space, power consumption, cooling, and other. The four facilities spending categories were defined as follows: Buildings and floor space, including purchase, rental, overhead charges, and regular maintenance of the compute facility s building or floor space. Power consumption for the computer systems and associated equipment, including storage systems, networks, etc. Cooling for computer rooms and equipment. Other facilities: Anything not covered above. The facilities budget allocation responses are summarized in Table 7. Facilities had the largest number of sites stating not-in-budget for at least one sub-category 36% of the sites in the 2011 Survey. Of the remaining 78 sites in this year s survey where the facilities category was included in respondents budgets, more than 25% listed one or more items within facilities as NIB. Therefore the averages provided in Table 7 should be viewed as minimum expenditures, as additional money must be provided from other sources to meet facility spending requirements (e.g., power has to be paid for by someone). Table 7: Average HPC Facilities Budget Distribution by Survey Year, NIB included Category All Years Building/floor space 23% 32% 37% 27% 32% Power Consumption 37% 40% 37% 44% 40% Cooling 40% 28% 26% 29% 28% Other 1% 4% 8% 3% 5% Source:, 2012 Power consumption continues to be the top spending category within facilities, with a 44% share. Cooling and building/floor space are almost equivalent. As the cost of energy rises, power consumption is becoming an increasingly critical design component, especially in ultra-scalable applications where growth can be limited by power capacity and the ability to dissipate heat. As more of the HPC budget goes to power consumption and other facilities costs, less is available for spending on hardware and software products. Page 14 of Information from this report may not be distributed in any form without permission from.

15 Staffing Budget Allocation The staffing budget was allocated based on the task being performed. The six types of personnel were: systems management and operations, maintenance, system programmers, application programmers, user support and services, and other. The six staffing spending categories were defined as follows: Systems management and operations: Staff responsible for day-to-day operations, troubleshooting, system balancing and optimization, etc. Maintenance personnel, including maintenance of computer servers and clients and associated storage and networking equipment. Note: If maintenance is purchased through a services contract with an outside organization, that expense is included under services. System programmers, including those primarily responsible for developing and maintaining software to enhance the OS and/or middleware capabilities. Application programmers, including those primarily responsible for developing and maintaining programs to support end-user efforts and applications. User support and services personnel primarily responsible for such activities as: helpline, internal training, documentation, applications porting, optimization and debugging support, etc. Other staffing: Anything not covered above. The staffing budget allocation responses are summarized in Table 8. The top-level staffing share remained relatively constant, ranging between 23% and 26% over the survey years, but within the subcategories, dramatic shifts in spending occurred in Table 8: Average HPC Staffing Budget Distribution by Survey Year, NIB included Category All Years Systems Management and Operations 34% 33% 25% 33% 30% Maintenance Personnel 9% 16% 12% 24% 16% System Programmers 10% 15% 17% 21% 17% Application Programmers 16% 21% 26% 5% 18% User Support and Services 23% 14% 17% 17% 17% Other 8% 2% 3% 1% 2% Source:, 2012 Spending on application programmers declined most dramatically, going from 26% in 2010 to 5% in The same shifts occurred across each sector, with the commercial sector having the largest percentage decline. Additionally, similar declines were found in each budget group with small budgets showing the greatest change. About 45% of the 2011 respondents reported 0% spending for application programmers, compared to Page 15 of Information from this report may not be distributed in any form without permission from.

16 between 17% and 25% in the other three surveys. This dramatic shift is surprising. Our Site Census has shown that the use of open source applications is growing, even in traditionally ISV-dominated segments like CFD. However, while we were seeing these results in the Site Census, the Budget Map studies showed an increase in spending for application programming software tools. These contradictory trends make the decline even more unexpected. We believe that this shift results from the response of central IT organizations to continued pressures to reduce costs by transferring application development responsibilities to end-user organizations. In this case the IT department may mitigate the reduction in services by purchasing additional application development tools for the end-user groups. We will watch this area closely going forward to establish whether 2011 is a reset in the amount of spending on application programmers versus a short-term blip. In contrast, maintenance personnel spending doubled to 24% from its three-year average of 12%. The increase in spending on maintenance personnel can possibly be attributed to a rise in upgrades of installed systems. In the services category, maintenance and repair declined, suggesting that sites are maintaining the systems with internal staff rather than hiring external services. For some sites this growth may result from a double shift of funds and people: first a shift of spending from external support services to internal support personnel, and second a shuffling of job function from application development to maintenance programmer/staff. Such shifts would be driven by desires to maintain total staffing levels and keep experienced personnel within the organization. Services Budget Allocation Users services budgets were allocated to six categories: maintenance and repair, systems engineering support, systems integration support, training, programming services, and other. The six services spending categories were defined as follows: Maintenance and repair, including equipment repair and preventative maintenance contract services. Systems engineering support for development of site requirements and specifications, RFPs, proposal evaluations, system acceptance testing and evaluation, site planning, etc. Systems integration support for physically integrating clusters, networks, storage systems, grids, etc., as specified by the site. Training, training classes, seminars, etc. provided by external organizations. Programming: Programming services provided by external organizations. Other services: Anything not covered above. The services budget allocation responses are summarized in Table 9. Maintenance and repair was once again where most of the dollars in this category were spent, with a 29% share. However, its percentage has been Page 16 of Information from this report may not be distributed in any form without permission from.

17 decreasing with each survey. In 2011, in-house maintenance personnel share increased, suggesting that organizations are transitioning this to an internal expense rather than external. The data provided in Table 9 is the average budget for those that spend money on these items. Table 9: Average HPC Services Budget Distribution by Survey Year, NIB included Category All Surveys Maintenance & Repair 64% 44% 37% 29% 39% Systems Engineering 10% 17% 16% 22% 18% Systems Integration 10% 19% 18% 26% 20% Training 10% 12% 14% 14% 13% Programming 5% 6% 12% 7% 8% Other 0% 2% 2% 1% 2% Source:, 2012 We believe several factors contributed to the decline of maintenance and repair budgets over the years: Maintenance contracts first to go in tough economic times During the great recession, it appears sites were willing to bear the risk of potential hardware failures in order to use limited funds to retain much-needed personnel. It is easier, faster, and less expensive to cancel a maintenance contract than to lay off an employee. Now that the economy is showing some improvement, money is being shifted to internal maintenance personnel rather than external services contracts. Node replacement shifts budget to hardware With the predominance of clusters based on standardized components, replacing a node or buying extra nodes can be used as a type of warranty contract. We have heard of two different scenarios: 1) buying extra nodes at time of purchase and utilizing them if any fail and 2) purchasing new nodes if failure occurs. In these cases, the budget share would be assigned to hardware and not to external maintenance and repair. System self-monitoring and warning software Vendors have been implementing and expanding software tools that allow system management staff to monitor hardware components health and identify those that are likely to fail. These components can be swapped out during routine maintenance and sent for repairs, discarded, etc. In this case diagnostic costs are either replacing software tools, or transferred back to the vendor if components are returned. Maintenance has simplified Over the years, maintenance of computer hardware has moved to replacing larger parts rather than fixing a part and therefore, for most maintenance issues, internal personnel can handle many repairs before requiring the expertise of factory-trained personnel. Cloud/Utility/Outsourcing Computing Budget Allocation Cloud/Utility/Outsourcing computing (hereafter referred to as cloud/utility) budgets were distributed based on the type of utility being purchased. Prior to the 2011 survey, we had three subcategories: raw cycles, application support, and other. In 2011, we expanded to five categories: raw cycles, software-as-aservice/application support, cloud storage, environment hosting, and other. Page 17 of Information from this report may not be distributed in any form without permission from.

18 The five cloud/utility/outsourcing computing spending categories were defined as follows: Raw cycles: Purchase of computing cycles for which the organization supplies the applications software. Software as a Service/Applications support: Purchase of computing cycles and ISV software within the cloud. Cloud Storage: Purchase of data storage space located at a cloud provider's facility. Environment Hosting: Use of a service to support virtually all computational tasks, with servers, storage, and software all being maintained by a third party. Other utility/outsourcing costs: Anything not covered above. In the last three surveys, on average between 3 and 4% of the HPC budget was spent on cloud/utility computing. In 2011 this value was 3.3%, with less than a quarter of respondents indicating that any money was budgeted in this category. As shown in Table 10, raw cycles consumed about 42% of the cloud/utility budgets in As mentioned previously, we believe users are leveraging cloud/utility computing when needed rather than offloading major workloads. Table 10: Average HPC Cloud/Utility/Outsource Budget Distribution, NIB included Category All Years Raw Cycles 36% 41% 56% 42% 36% SAAS/Applications Support 40% 57% 35% 16% 30% Cloud Storage 12% 9% Environment Hosting 26% 20% Other 24% 2% 8% 3% 4% Source:, 2012 In this latest survey we also asked respondents to divide their cloud budgets between public and private implementations. Only 16 respondents answered this new question. For this small sample, the average budget assignment of public cloud was 41%. Five used only public cloud, four never used public cloud, and the remaining seven assigned public cloud between 10% and 40% of their budgets. Predicted Budget Change At most user organizations, HPC is tied to top-level productivity goals. For industrial users, this means revenue generation. Users cannot design a next-generation car or airplane or find the next oil reservoir or capture more subscribers in online gaming without continuing to invest in HPC technology. Intersect360 believes that the strong tie between HPC systems and buyers core R&D and business activities minimizes or delays the impact of economic downturns on HPC spending. Page 18 of Information from this report may not be distributed in any form without permission from.

19 Table 11 provides the HPC budget expectations for the last four surveys. Indeed, the budgets remained strong during the global recession, with over 60% of the sites stating they expected growth greater than 5% over the next two years. In 2011, the outlook for HPC budgets became less optimistic with only 43% expecting growth and an almost equal share expecting relatively flat budgets over the next two years. Table 11: Predicted Changes to HPC Budgets Over the Next Two Years, All Surveys Category Decline (> than 5%) 4% 10% 5% 13% Relatively flat (within +/ 5%) 25% 29% 31% 44% Growth (> than 5%) 71% 61% 64% 43% Source:, 2012 Most of the pessimism in the HPC budget outlook exists at academic and government sites, while commercial sites remain relatively healthy (see Table 12). We attribute the flattening of growth expectations in our latest survey to uncertainty in funding for scientific research at academic and government sites as well as the slowness in reaching a complete recovery from the global recession. Table 12: Predicted Changes to HPC Budgets Over the Next Two Years by Sector, 2011 Survey Category Academic Commercial Government All Sectors Decline 16% 3% 21% 13% Relatively flat (within +/ 5%) 48% 38% 42% 44% Growth 36% 59% 37% 43% Sample Size Source:, 2012 Using the midpoints of provided ranges for expectations for HPC budget over the next two years, we calculated an average 5% increase for all sites. Most of the increase came from budgets of less than $500,000 with larger HPC budgets expecting on average between 2 and 3% increases. Government and academic sites showed much lower increases (~3%) versus commercial sites (~9%). Government and academic sites have faced more uncertainty in funding as politicians argue on how to balance the US budget and Europe faces ongoing economic uncertainty. In addition to providing the predicted change to their HPC budgets, we allowed respondents to identify any areas where they expect significant changes. About 40 sites responded to this open-ended question in the 2011 survey, providing a qualitative view of the factors driving the growth in budgets. The key areas where respondents expected an increase in budget were: hardware purchases (19 sites); cloud computing (7); staffing, including programmers (5); and security (3). Within hardware, servers and storage were singled out as areas of increased spending, each mentioned by seven sites. Increased spending on security was the most unique and interesting change. HPC is not immune to security issues and breaches. In 2010 it was noted by one site, and in 2011 it was mentioned by three sites from three different vertical industries: financial services, oil/gas, and academia. Security issues can range from achieving security in a hybrid cloud environment, to protecting corporate assets and the private information of employees and clients, to maintaining an efficiently-running computer environment. Though three sites is not indicative of a trend, the fact that three sites singled out security and took the trouble to enter in the survey suggests that it may be high on the list of concerns for HPC sites. Page 19 of Information from this report may not be distributed in any form without permission from.

20 Total HPC Budget Distribution In the above sections, we examined each individual component of spending. In this section, we look at the entire HPC budget distributed across all spending components. The top-level percentages are multiplied by the appropriate subcategory item to understand the share each item represents. For example, the server share is calculated by multiplying the percentage allocated to hardware by the percentage within hardware allocated to servers. All shares add up to 100%. Items with less than 1% share were combined as other. These categories were: SaaS cloud/utility computing, programming services, cloud storage, and many of the subcategory other items. Servers, storage, and systems management/operations have been the top three spending categories in the four surveys we have completed (see Table 13 and Figure 3). The top three categories, however, have captured a smaller share in each consecutive year, declining to 33% in 2011 from 46% in 2007, driven by the decline in spending on servers. Spending on servers is still the only category with double-digit share of the HPC budget. Table 13: Total HPC Budget Distribution by Survey Year Category Server 30% 23% 22% 17% Storage 9% 10% 9% 9% Systems Mgmt and Operations 7% 7% 7% 7% Maintenance Personnel 2% 3% 3% 6% System Programmers 2% 3% 4% 5% Network 4% 4% 4% 5% Software Tools 4% 4% 2% 5% Client 3% 3% 3% 4% Power consumption 3% 4% 3% 4% User Support and Services 6% 4% 4% 4% Building/floor space 2% 3% 4% 4% System Software 2% 4% 3% 3% 3rd Party ISVs 6% 2% 2% 3% Cooling 3% 3% 2% 3% Middleware 3% 2% 2% 3% Maintenance and Repair 5% 2% 3% 2% Systems Integration 1% 1% 1% 2% Raw Cycles 0% 1% 3% 2% Storage Management 1% 1% 2% 2% Systems Engineering 1% 1% 1% 2% Training 1% 1% 1% 1% Application Programmers 4% 6% 6% 1% Transfer Costs 0% 1% 1% 1% Environment Hosting 1% Other 3% 4% 4% 4% Note: Items with < 1% share were combined into Other Source:, 2012 Page 20 of Information from this report may not be distributed in any form without permission from.

21 Figure 3: Total HPC Budget Distribution for Top Ten Items in 2011 HPC IT Product Spending Source:, 2012 IT product spending includes spending on all items within hardware, software, services and utility computing categories and accounts for 64% of the overall budgets in the survey, up slightly from the 2010 survey. IT product spending represents the budget dollars for which hardware, software, and service vendors can compete at these sites and is incorporated into our forecast model. Figure 4 provides the distribution for IT product spending. Page 21 of Information from this report may not be distributed in any form without permission from.

22 Figure 4: HPC IT Product Spending Distribution, 2011 Survey Note: Items with less than 1% share were combined into Other. Source:, 2012 Servers, storage and networks together account for almost half of the IT product spending, at 47%, and have represented the top three IT spending categories for the last three surveys (see Table 14). Spending for client hardware moved into position five in 2010 and has captured a greater share of the HPC IT budget in the 2011 survey. Table 14: Top Five Categories for HPC IT Spending Rank #1 Server Server Server Server #2 Storage Storage Storage Storage #3 3rd party ISV Network Network Network #4 Maintenance & Repair System Software System Software Software Tools #5 Network Client Client Client Source:, 2012 It is important to note that the distribution of budgets can vary widely depending on the type of user organization. These sector-level differences will be detailed in a subsequent report. Page 22 of Information from this report may not be distributed in any form without permission from.

23 CONCLUSIONS The distribution of HPC budget share reveals the priorities at the individual site: Which component of the IT infrastructure needs upgrading and improving? This is especially apparent at small organizations where, for example, hardware share may be 0% because the site bought a server last year. For larger organizations, especially in tight budget years, the budgets reveal which area is most constrained. Does the site need to buy a new system, install more storage or a higher bandwidth network, hire personnel to bring tasks back within the organization, or wait because of increasing facilities costs? Achieving an optimum balance of the HPC IT infrastructure within a finite budget can result in year-to-year fluctuations for a site that provide little insight about the industry. By aggregating all sites and averaging all budgets, budget shifts impacting the entire HPC industry can be revealed. Noteworthy Numbers identified several areas worthy of additional analysis and comment. We will track these areas closely going forward: Increased hardware spending on networks and clients Spending on client hardware and networks has increased over the last four surveys, primarily taking share away from servers. Our site census database concurs with the increase in spending on networks, as more sites have reported installations of 10 Gigabit Ethernet. The increase in clients is a balancing act between servers and workstations. For the last three years, combined share of hardware spent on servers and clients is the same, at about 60%, suggesting that compute spending is flat overall and that there has been a shift in the locale of the compute power. It appears that the pendulum may be swinging back toward the desktop, where multi-core processors can handle many of the jobs. Staffing represents 23% of top-level budgets In all four surveys, staffing continued as the second-largest major budget category item, with system management and operations and maintenance personnel accounting, on average, for 56% of the staffing budget. As computers and overall IT architectures become more complex, we expect the amount of money spent on people to manage and maintain these systems will continue to grow. However, this complexity may also create opportunities for software tool companies that should view their total market opportunity as including at least some portion of site-staffing spend. Facilities costs march onward and upward Facilities cost was the only major budget category whose share has consistently increased over the four surveys. We believe these costs are borne largely by parent organizations. This leads to two important issues: First, facilities costs are discontinuous. That is, they are stable as long as system requirements can be addressed by the current physical plant, but they can rise dramatically whenever requirements for space, power, or cooling are not met. Second, vendors with a strong facilities story should potentially look to selling both to the IT organization and to parent organizations that are paying the facilities costs. Software tools share on the rise In 2011, we saw software tools become the fourth largest IT product spending category after servers, storage, and networks. We have noted in this report and in previous reports that the complexity of the IT infrastructure is creating opportunities for software tools. The rise in share of software tools suggests demand exists. Page 23 of Information from this report may not be distributed in any form without permission from.