ITIL Practitioner Course 06 - Use Metrics & Measurement

ITIL Practitioner Course 06 - Use Metrics & Measurement

Course Slide 1 Use Metrics & Measurement CSFs, KPIs & Improvement Analyze CSFs & KPIs in context Assessments Design a Report

Slide 2 Learning Objectives o Apply Metrics & measurement from outside-in context o Define CSFs o ITIL vision to measurement o Balanced scorecard o Organizational cascade o Determine KPIs o SMART KPIs o KPIs support/underpin CSF o Analyze CSF & KPIs in given context o Technology, process & service o Progress, compliance, effectiveness & efficiency o Leading & trailing o Inside-out, outside-in o Define current state o Assessments o Assessment scope o Assessment Criteria o Assessment outputs o Design a report o Improve report value o Apply metrics knowledge to o Plan & implement improvements o Improve metrics & measurement

Slide 3 Lesson CSFs, KPIs & Improvement Start with the Vision Vision to Measurement What Gets Measured Gets Done Critical Success Factors How to Determine CSFs Determine KPIs Balanced Scorecard Organizational Cascade Continual Improvement of Metrics & Measurement

Slide 4 Start with the Vision o Vision defines high level scope o Place bounds on a mission through measurement o Define goals in business outcome-based terms o Quantifiable o Become CSFs lead to the determination of KPIs o Define objectives as targets o Required to achieve goals o Use KPIs to determine specific measures to plot progress o Start with vision to take outside-in approach o Starting with specific metrics (or measurements) is example of inside-out

Slide 5 Vision to Measurement Vision Mission 7-Step Improvement Process Goals Objectives CSF KPI Metrics Measurement

Slide 6 What Gets Measured Gets Done o Everything is subject to improvement o Service o Process o Technology o Metrics o Communication (including reports) o Organization o Nothing cast in concrete; business change IT change o New or changed business process o Resulting new or changed IT services o Regulatory requirements o Infrastructure o Applications o Organizational structure o Mergers & acquisition o Results from previous improvements or changes o Review every CSF, KPI, metric & report o Every time something above changes o At least annually (or the minimal organizational review period) o Does it still add business value?

Slide 7 Critical Success Factors o Critical is the keyword o Not all success factors meet criteria o Limit to minimum viable product (MVP) o CSF represents target goal/outcome o Limit number of CSFs o Not always directly measureable o CSF must be quantifiable o Ask: How will you know? & related questions o Key Performance Indicators (KPIs) used to determine achievement of CSF o KPI = indicator, not end or measure of achievement o Discuss KPI in context of CSF, not isolated metrics o CSFs (& KPIs) focus on business value o Limit number of KPIs (Keep it Simple) o CSFs will change with maturity o CSF change may mean change to KPIs o Capability of practice o Quality of practice o Quality of service A CSF is something that must happen if an IT service, process, plan, project or other activity is to succeed. ITIL Glossary KPI is a metric used to help manage an IT service, process, plan project or other activity. KPIs are used to measure the achievement of CSFs. ITIL Glossary

Slide 8 How to Determine CSFs o Ask questions o Determine current state o Establish how to quantify success o How will you know? o Verify business value of the new/desired state o Determine time-bound nature of resulting KPIs o Challenge & verify CSF to remove/resolve ambiguity o Sample questions: o What is the history around this CSF? o What is the specific business goal this CSF maps? o What business processes are impacted? o Is there an acceptable range for achievement? o Who uses or depends on the business process? o What would trigger negative impact if? o How is CSF achievement (or lack thereof) reported? o What KPI(s) support this CSF? Can we collect/measure them? o Tie each CSF & KPIs back to the purpose for measurement o Validate o Direct o Justify o Intervene

Slide 9 Determine KPIs o Make KPIs SMART o Specific o Measureable o Achievable o Relevant o Time-bound o KPIs are important criteria o Important to achievement of CSF o Key; ignore minor indicators o Measured current performance o Indicator of achievement; not proof of achievement o Use multiple KPIs per CSF (fewer better than many) o Beware unintended consequences o Pure inside-out approach selection bad/inappropriate KPIs o What get measured gets done

Slide 10 Balanced Scorecard o Four perspectives linked to the vision o Customer: Experience & satisfaction o Financial: Typical fiduciary responsibility o Business: process: health of internal workings, potential leading indicator o Learning & growth: linked to capability to continually improve o What will be different with vision success? o Stakeholders o Internal processes o Capability to innovate & grow o Define CSFs for each of the 4 perspectives o Establish KPIs to determine addressing if CSF achieved as planned HBR Reprint R0707M Using the Balanced Scorecard as a Strategic Management System Robert Kaplan and Daniel Norton, January 1996 Customer Financial Vision & Strategy Learning & Grown Business Processes

Slide 11 Organizational Cascade o Measure organizational performance at multiple levels o Organization o Defines overall management & leadership of change o Defines top-level objectives & goals o Cascades down through the organization o Business unit o Responsible to develop program & projects to achieve goals o Department o Team o Person o Single high level result in multiple lower level o Plans o Metrics & measurements o Metrics defined at lower level to o Support next higher tier o May impact performance evaluations & reviews

Slide 12 Continual Improvement of Metrics & Measurement o Subject everything in ITSM to improvement o Include metrics & measurement o Change that may trigger improvement o New/change business process o Change to organizational structure o New/change IT service o New/change infrastructure or application o Increased process effectiveness & efficiency (maturity) o Regulatory & related o Previous improvements o Areas of impact o What needs to be measured o Thresholds, CSFs & KPIs and other metrics o What & how for reporting

Slide 13 Lesson Analyze CSFs & KPIs in Context Metric Categories Technology, Process & Service Metrics Process Metrics Leading & Lagging Metrics Inside-out & Outside-in Metrics

Slide 14 Metric Categories o Technology o How technology adds business/customer value o Process o Compliance, efficiency, effectiveness & progress o Service o Value created & delivered o Value, outcome, cost & risk o End-to-end from an outside-in perspective o Leading & lagging o Leading metrics allow anticipation of what might happen o Lagging metrics reveal what has already happened o Inside-out & outside-in o Inside-out metrics (& thinking) represent internal/technology-first view o Outside-in metrics based on value created & delivered to the customer o Think in the larger context o Identify & select metrics that cross categories o Ensure all categories covered o Balance metrics; not too many from a few categories o Metrics identify opportunities for improvement of value o Measurement & metrics also subject to improvement

Slide 15 Technology, Process & Service Metrics o Consider all three categories o Focus on the value created & delivered (VCD) o Value, outcomes, costs & risks (VOCR) o Service metrics measure end-to-end performance o Consider value, not technology o End-to-end value of services = outside-in thinking o Process metrics measure performance o Isolated or siloed processes = inside-out o Support of services = outside-in o Technology metrics address infrastructure & applications o Measure contribution to service VCD & VOCR o Align technical with customer/user perspective o Technical availability translates to customer productivity o Customers don t care about technology o Just outcomes o Technology questions may mean service dissatisfaction

Slide 16 Process Metrics o Compliance o Meet governance or regulatory requirements o Typically pass/fail o Efficiency o How well does the process uses resources? o Effectiveness o Does the process deliver expected or required results? o Progress Metrics o Measure achievement of milestones and deliverables o New process: Process policy & documentation approved by IT & business

Slide 17 Leading & Lagging Metrics o Lagging metrics contribute to understanding history o Lagging metrics o Record past achievements o Leading metrics predict likely future o Predict & plan for future o Extrapolate trend o Possible to determine from aggregated past measures o Categorize to maintain balance o Customers expect lagging metrics o Collect & use leading metrics to learn to anticipate improvements Lagging Metric Leading

Slide 18 Inside-out & Outside-in Metrics o Inside-out is an internal IT service provider view o Some needed for internal measure o Considered in broader context: how it impacts customers o Outside-in focus is customer perspective o Represent customer view of IT services o Measure things customers actually need/want o Perspective governs how metrics are used & perceived o Service availability expressed as percentage = inside-out o Impact on productivity of service availability = outside-in

Slide 19 Lesson Assessments Assessments Introduction Assessment Scope Assessment Criteria & Outputs

Slide 20 Assessments Introduction o Analyze & understand o Behavior o Performance o Apply to o Service, process & technology o People o Anything else that supports business value o Assessments o Require clear purpose o Support creation of value for customers o One-off: current situation only o On-going: how capability evolves over time o Good assessments o Start with fully documented & understood purpose approved by stakeholders o Identify issues & gaps for improvement o identify things that are working

Slide 21 Assessment Scope o Limited & bound to a purpose o Not to broad or too narrow o Poorly defined likely to collect wrong data & wastes resources o Assess assessment o Ensure they re valid o Requirements & outcomes change over time o Identify stakeholders o Ensure their capability to influence situation is understood

Slide 22 Assessment Criteria & Outputs o Assessment success depends on o Asking right questions o Collecting right data o Defining criteria aligned to assessment purpose & scope o Assessment outputs (partial list) o Graphical models o Gap analysis o SWOT analysis o Maturity models o Process o Organization o Customer satisfaction o SLA achievement o Benchmark comparison 7-Step Improvement Process

Slide 23 Lesson Design a Report Reporting Reports Improve Report Value

Slide 24 Reporting o Identify purpose for reporting o Document service performance for stakeholders o Compare actual to target o Identify improvement opportunities o Use audience appropriate format & language o Present in document or dashboard o Business = ROI o IT Management = tactical & strategic results support business value o IT operational/technical managers = metrics to support resource o Planning o Coordination o Scheduling o Utilization o Infrastructure performance o Link reports & scorecards to o Business strategy & goals o Validate, direct, justify or intervene

Slide 25 Reports o Provide summary & analysis of metrics o Validate, direct, justify or intervene o Include balance of appropriate metrics o Technology, process & service o Leading & lagging (include relevant trends) o Inside-out & outside-in o Identify what was learned from analysis o Present information to support conclusions o Reports without critical analysis = incomplete o Allow focus on numbers not outcomes o Dashboards provide summary o Created to meet different needs of specific audiences o May be generated by ITSM tools o Facilitates rapid intervention

Slide 26 Improve Report Value o Identify purpose & stakeholders o If it doesn t add value, why do it? o Produce only reports that add or have value o Ensure appropriate level of detail o Consider infographics o Identify the o Purpose, value & audience o Action to be taken based on report (& appropriate RACI) o Data needed to produce report o Validate & verify report o Is still needed o Does is still have value

Slide 27 Summary Use Metrics & Measurement CSFs, KPIs & Improvement Analyze CSFs & KPIs in context Assessments Design a Report

Slide 28 Define CSFs & KPIs o Start with the Vision o What Gets Measured Gets Done o Critical Success Factors o How to Determine CSFs o Determine KPIs o Balanced Scorecard o Organizational Cascade

Slide 29 Analyze CSFs & KPIs in a given context o Metric categories (context) o Technology, process & service metrics o Process metrics o Leading & trailing metrics o Inside-out & outside-in metrics

Slide 30 Assessments o Assessments introduction o Assessment scope o Assessment criteria & outputs

Slide 31 Design a Report o Reporting o Reports o Improve report value

Slide 32 Additional Resources for Metrics & Measurement o ITIL Practitioner Guide o 7.1.2 Orientation worksheet o 7.2 Metrics and measurement o 7.2.6 Assessment report content examples o 7.4.14 Frameworks & criteria to plan assessments o ITIL Continual Service Improvement (CSI) o Chapter 5: CSI Improvement Methods & Techniques o 5.1 Methods & techniques o 5.2 Assessments o 5.3 Benchmarking o 5.4 Service measurement o 5.5 Metrics o 5.7 Service Reporting o 5.8 CSI & other service management processes o HBR Reprint R0707M Using the Balanced Scorecard as a Strategic Management System Robert Kaplan & Daniel Norton, January 1996

Slide 33 Checkpoint Checkpoint Introduction to Checkpoint Exercises Review Checkpoint Exercises Chapter Quiz

Slide 34 Introduction to Checkpoint Exercises

Slide 35 Review Checkpoint Exercises

Slide 36 Course Quiz

Review Questions: 1. What is the first thing Lisa should do, before meeting with the process managers? A. Examine the critical items in the CSI register B. Determine if Grab@pizza has the internal skills to carry out the assessment C. Ensure Grab@Pizza can put together an independent team (so that the people doing the assessment don t have a conflict of interest with the results) D. Define & document a clear purpose for the assessment 2. An assessment will be performed on the operational issues around incident and problem management. What should Lisa do to get this effort started? A. Establish suitable assessment criteria B. Get an external benchmark to serve as a basis for comparison C. Define the scope of the assessment D. Request a suitable budget, time and resources to perform the assessment 3. Which of the following is the best statement for the purpose for an assessment? A. Identify current state B. Identify gaps C. Identify what isn t working D. Analyze & understand performance and behavior 4. No metrics were mentioned for the processes at Grab@Pizza. What is the first process assessment that should be conducted? A. Determine process effectiveness B. Determine process compliance C. Determine process efficiency D. Determine how progress through a process is tracked 5. How should the principle to work holistically be applied to metrics? A. It serves as a reminder to apply systems thinking B. Helps ensure that only the things that are really important and add value are considered for measurement C. Help to ensure metrics are balanced D. Serves as a reminder to not only consider the process of performing the assessment, but also how the output of the assessment will be used and reported

6. A CSF mentioned in the case study was: Failures caused by changes and outages (don t have) a significant impact on business processes. Is this a good CSF? If yes, why, if no why not? A. Yes, because it describes a good future state B. Yes, because it is easily understood C. No, because it isn t clear exactly what must occur to meet the criteria D. No, because it isn t stated in terms of what must happen to be a success factor, let alone critical 7. A problem was mentioned regarding SOX compliance related to IT changes. The following CSF was created as a result of an assessment: Within 8 months (in other words, 3 months before the next SOX audit) IT changes will not impact SOX compliance. This time period was chosen to allow for any fine tuning required to ensure passing the next audit. The following interim KPIs were created to support the CSF 2-months into the endeavor to meet the CSF. 25% decrease in change related incidents 40% reduction in the time required to resolve priority 1 change related incidents 30% increase in the incidents resolved using the knowledge base 50% improvement in customer satisfaction with changes 100% changes go through change management Do these KPIs represent a balanced set? A. Yes, they are a balanced set B. No because there are no process metrics C. No because there are no inside-out metrics D. No, because there are no leading metrics 8. With regards to the inventory problem mentioned in the case study. Which of the following is the best first step to take to resolve it with the small number of suppliers involved? A. Send each supplier a survey/questionnaire to collect appropriate information B. Visit or talk with the suppliers to collect appropriate data on their interactions with Grab@Pizza C. Speak to the internal people who own the relationship with each supplier to determine why the suppliers aren t able to meet demand D. Speak to the stores experiencing the inventory shortages to determine how they report the issue

9. Which of the following statements about the CSFs and KPIs for SOX compliance is correct? A. The metrics should justify a course of action B. The metrics should validate a course of action C. The metrics should direct a course of action D. The metrics should provide a means to intervene 10. A report based on the assessment of the outages caused by changes has been prepared and includes the following: Change related outages account for 80% of all outages Change related outages caused 25% or major incidents Human error and accidents accounted for 10% or outages Partner incidents account for 5% of all outages 5% of major incidents result from unknown causes There is no KPI to assess franchisee reaction to these outages Does this report support making a decision regarding a course of action? A. Yes, because it identifies what is needed to justify a course of action B. Yes, because it includes all aspect of the outages C. No, because there s no information about causes D. No, because the report includes information that isn t needed

Answer Key: 1. D While A through C have to be performed, the first thing is determining the purpose for the assessment. Absent that, then there is no clear direction regard the other 3 answers. D is correct. 2. C What is in and out of scope will determine assessment criteria and budget. C is correct. 3. D While there are elements of truth in answers a through C, D is correct because it subsumes the other three. 4. B Without measureable compliance with process procedures and activities, the other process metrics don t mean very much. B is correct. 5. D While A is generally true about the principle to work holistically, the question asked about metrics. D is correct. 6. C The exercise reviews a way to improve this CSF. Specifically, there needs to be clarity on the meaning of significant impact. C is correct. 7. D D is correct. There are KPIs for process and inside out. There are no leading metrics. 8. B B is correct. A questionnaire isn t likely to add much information of value in a timely manner. Talking to the internal people doesn t provide information about vendor response. While talking to the stores might provide useful information, B provides the best choice.

9. C Failing the SOX audit is the only justification and validation needed. The failure requires intervention to prevent penalties. C is the correct answer. The CSFs and resulting KPIs should direct specific actions to get into compliance. 10. D There is no statement in the case study that franchisees are directly impacted by or even aware of the change-related outages. D is correct.