Outline. Introduction. Nicholas C. Romano, Jr., Ph.D. Project Management Graphics: An Experimental Comparison Nicholas C. Romano

Transcription

1 Project Management Graphics: An Experimental Comparison Nicholas C. Romano William S. Spears School of Business Management Science and Information Systems Oklahoma State University - Tulsa Nicholas.Romano@okstate.edu Garry Booker 1 Outline Introductions Project Management Graphics Left Brain vs. Right Brain Thinking Theory Research Questions Study Design and Methods Preliminary Results Implications Future Analysis and Research 2 Introduction, Ph.D. Associate Professor, MSIS, OSU-Tulsa Ph.D. from University of Arizona in MIS 20 years work experience, much of it in group support for teamwork and projects Former IBMer (Dad, Brother, Sister, Brother in law, Great Uncle, Second cousin, others I am sure) 3 1

2 Oklahoma Center for the Advancement of Science and Technology (OCAST) Title of Project: Non-collaborative productivity for Eva (Earned-value Virtual-office Animator) OARS/OCAST Grant No: AR The Grant is also privately funded by ProjectFrontier LLC (Garry Booker). 4 Non-collaborative productivity for Eva (Earned-value Virtual-office Animator) Assumptions: 1. Traditional EVM graphics require much mental effort 2. Area Graphics should require minimal mental effort Premise: If project status graphics require minimal mental effort; ; knowledge worker s s task status awareness and decision making ability can be greatly improved. Goal: Validate this theory through controlled experiments. 5 For Project Management Graphics are Important Classic Project management graphics are analytically thin, too simple, and lack substantive detail. Edward R. Tufte GANTT CHARTS PERT CHARTS (Network Charts ) Etc. 6 2

3 For Project Management Graphics are Important Earned Value Management is the most accurate method known for tracking status. However it is rarely used. A Few Surveys Reveal this 7 Survey of PM Maturity of 280+ Canadian individuals from 65 Organizations 1-55 Maturity scale in 9 PMI Knowledge areas Earned Value in Cost Management area at level 3 Results: Level 5 0 Level 4 0 Level 3 1 (1.5%) Level 2 47 (73.5%) Level 1 16 (25%) M. Mullaly, 1997 Canadian project management baseline study, in Proc. 29th Annu. Project Manage. Inst. Seminar Symp.,, Long Beach, CA, Project Management Software Functionality: usage, perceived value and potential to improve project performance 753 participants Current Primary Role: Vast majority PMPs Team Member (8%); Male (67%); Project Manager (51%); Aged (74%); Program Manager/Director (24%); Other (17%). Distribution by product type, not by industry sector (5% more in IT than in PMI membership) Engineering and Construction (12.3%) Business Services (11.8%) IT and Telecommunications (58.6%) Industrial Services (4.2%) Personal Services (0.6%) Other (12.5%) Besner,, C. and B. Hobbs (2004) "Project Management Software Functionality: ity: usage, perceived value and Potential to improve project performance."

4 Surveys of EVM Use and Adoption Besner,, C. and B. Hobbs (2004) "Project Management Software Functionality: ity: usage, perceived value and Potential to improve project performance." Nearly 70% Use Gantt Charts Extensively or very Extensively Besner,, C. and B. Hobbs (2004). An Empirical Investigation of Project Management Practice: In reality, which tools do practitioners use?. 3rd PMI Research Conference, London, England, PMI. 11 Nearly 50% Do not use EVM and Only 20% Use it Extensively or very Extensively Besner,, C. and B. Hobbs (2004). An Empirical Investigation of Project Management Practice: In reality, which tools do practitioners use?. 3rd PMI Research Conference, London, England, PMI. 12 4

5 over 60% Do not use S-Curves and Only 15% Use them Extensively or very Extensively Besner,, C. and B. Hobbs (2004). An Empirical Investigation of Project Management Practice: In reality, which tools do practitioners use?. 3rd PMI Research Conference, London, England, PMI. 13 Survey of 126 Senior practitioners who possess knowledge of the project management practices and business results within their respective organizations. 14 Grant, K. P. and J. S. Pennypacker (2006). "Project management maturity: an assessment of project management capabilities among and between selected industries." Nicholas IEEE C. Romano, Transactions Jr. on Engineering Management 53(1): Survey of 40 Small-sized to Medium-Sized Enterprises (SMEs : fewer than 250 emps., turnover less than 250 million and/or annual balance sheet less than 43 million. Only One out of 40 SMEs uses EVM Source: Murphy, A. and A. Ledwith (2007). "Project management tools and techniques in high- technology SMEs. Management Research News 30(2):

6 16 Theoretical Foundation Area Graphs ( with color and relative area ): Provide deep work status insight through right brain thinking (visualization( and Automaticity ) Prevent information overload ( Attention Load) Significantly improve task status awareness ( but not at the expense of attention ) A Few Examples Follow 17 Baby Name Wizard's NameVoyager

7 National Geographic Malaria Chart Jul2007, Vol. 212 Issue 1 page Brain Hemisphere Thinking Left Brain Analytical Linear Logical Rational Sequential Objective Looks at Parts Vertical Convergent Deductive Black/White Trial and Error Execution Right Brain Visionary/Synthesizing Spatial Analogical/Random Free Association/Holistic Intuitive/ Simultaneous Subjective Looks at Wholes Lateral Divergent Inductive Grey Reflex (Automaticity) Visualization 21 7

8 AUW OVERRUN Knowledge Presentation Comparison Line Graphics Calculation Analytical Multiple Lines Looks at parts Step by step Lines Comparison Logical Linear Sequential Trial and Error Area Graphics Recognition Visionary/Synthesizing Single Area Looks at Wholes One Visual Shapes Comparison Analogical/Random Spatial Simultaneous Reflex (Automaticity) 22 Line Graphic: Left Brain Analytical NCC TAB CBB Contract Price OTB Profit / Fees EAC Management Reserve PMB EAC TAB Management BAC Reserve EAC Undistributed Budget Management Reserve Summary Level Planning Packages Packages Planning Packages TERMINOLOGY TAB PMB Schedule Variance Cost NCC Negotiated Contract Cost EAC Contract price less profit / fee(s) Variance $ AUW BAC Management Reserve Authorized Unpriced contractually approved, but not yet negotiated / definitized ACWP PMB CBB Contract Budget Base Sum of NCC and AUW OTB Over Target Baseline Sum of CBB and recognized overrun TAB Total Allocated one Budget TAB graph Sum of all budgets for work on contract for = NCC, CBB, or OTB BCWS BAC Budget At Completion Total budget for total contract thru any given level Management Reserve PMB Performance Measurement Baseline BAC Contract time-phased budget plan Schedule BCWP Schedule Variance Variance MR Management Reserveach Budget withheld by Ktr PM for unknowns / risk management project PMB UB Undistributed Budget Broadly defined activities not yet distributed to CAs Cost CA Control Account Lowest CWBS element assigned to a single focal point to plan & control VARIANCES Favorable is Positive, Unfavorable is Negative Time Completion Variance $ time Cost Variance Now Date scope / schedule / budget Cost $ Variance CV = BCWP ACWP ACWP CV % = (CV / BCWP) *100 WP Package Near-term, task detail-planned activities within a CA Schedule Variance SV = BCWP BCWS SV % = (SV / BCWS) * 100 PP Planning Package Far-term CA activities not yet defined into WPs Variance at Completion VAC = BAC EAC BCWS Budgeted Cost for Scheduled Value of work planned to be accomplished = PLANNED VALUE Schedule Variance ACWP BCWP Budgeted Cost for Performed Value of work accomplished = EARNED VALUE OVERALL STATUS BCWS Cost Variance ACWP Actual Cost of Performed Cost of work accomplished = ACTUAL COST % Schedule = (BCWS CUM / BAC) * 100 EAC Estimate At Completion Estimate of total cost for total contract thru any given level; $ % Complete = (BCWP may be generated by Ktr, PMO, DCMA, etc. = EAC Ktr / PMO / DCMA CUM / BAC) * 100 BCWP ACWP LRE Latest Revised Estimate Ktr s EAC or EAC % Spent = (ACWP Ktr CUM / BAC) * 100 SLPP Summary Level Planning Package Far-term activities not yet defined into CAs DoD TRIPWIRE METRICS Favorable is > 1.0, Unfavorable is < 1.0 TCPI To Complete Performance Index Efficiency needed from time now to achieve an EAC BCWS Cost Efficiency CPI = BCWP / ACWP time Time Completion EVM POLICY: DoDI , Table E3.T2. EVMS in accordance with ANSI/EIA-748 is required for cost or incentive BCWS Now Date Schedule Efficiency SPI = BCWP / BCWS contracts, subcontracts, intra-government work agreements, & other agreements valued > $20M (Then-Yr $). EVMS contracts > $50M (TY $) require that the EVM system be formally validated by the cognizant contracting officer. Additional BASELINE EXECUTION INDEX (BEI) (Schedule Metric) Guidance in Defense Acquisition Guidebook and the Earned Value Management Implementation Guide (EVMIG). EVMS is BEI = # of Baseline Tasks Actually Completed / # of Baseline Tasks Scheduled BCWP for Completion discouraged on Firm-Fixed Price, Level of Effort, & Time & Material efforts regardless of cost. BCWP CRITICAL PATH LENGTH INDEX (CPLI) (Schedule Metric) EVM CONTRACTING REQUIREMENTS: CPLI = (Critical Path Baseline Duration + Float Duration) / Critical Path Baseline Duration Non-DoD FAR Clauses Solicitation (Pre-Award IBR) or (Post Award IBR) TO COMPLETE PERFORMANCE INDEX (TCPI) # Solicitation & Contract TCPI EAC = Remaining / Cost Remaining = (BAC BCWP time CUM) / (EAC ACWP Time DoD( Completion CUM) $20M) DFAR Clauses for solicitations and for solicitations & contracts Now Date Contract Performance Report DI-MGMT-81466A * 5 Formats (WBS, Organization, Baseline, Staffing & Explanation) ESTIMATE AT COMPLETION # Integrated Master Schedule DI-MGMT * (Mandatory for DoD EVMS contracts) EAC = Actuals to Date + [(Remaining ) / (Efficiency Factor)] time Integrated Time Baseline Review (IBR) - Mandatory for all EVMS contracts Completion Now* See the EVMIG for CPR and IMS tailoring guidance. Date EAC CPI = ACWP CUM + [(BAC BCWP CUM) / CPI CUM ] = BAC / CPI CUM EAC Composite = ACWP CUM + [(BAC BCWP CUM) / (CPI CUM * SPI CUM)] # To Determine a Contract Level TCPI or EAC; You May Replace BAC with TAB To Determine the TCPI BAC,LRE Replace EAC with either BAC or LRE PMB Control Accounts There will be TAB BAC 23 Slightly Behind Schedule Area Graphic: Right Brain Intuitive Slightly Over Budget Whole project and All task graphs can be shown at the same time More than half Completed Less than half Remaining 24 8

9 Research Questions Overarching Research Question: Can EVM-Area Graphs enhance project team member efficiency, effectiveness and satisfaction in terms of the process and outcomes for project status determination tasks? Specific Research Questions: 1. Can EVM-Area Graphs enhance the efficiency of Status Determination? 2. Can EVM-Area Graphs enhance the effectiveness of Status Determination? 3. Can EVM-Area Graphs enhance the Quality of Decisions? 4. Can EVM-Area Graphs enhance the Satisfaction of users that perform Status Determination? 25 Theoretical Causal Process Model P1 (+/-) P2 (+/-) Cognitive Load Information Load P5 (+) P6 (+) P9 (-) Task Performance Knowledge Presentation P3 (+/-) Physical Load P7 (+) Attention Load P10 (-) P11 (-) Decision Quality P4 (+/-) Satisfaction Task Automaticity P8 (-) 26 Operationalized Research Model Dependent Variables Task Completion Time Independent Variable Graphic Type: 2 Levels (+) Task Completeness EVM Line Graphics (S Curves) (+) Effort Over Time (-) Task Correctness Decision Quality EVM Area Graphics (-) Perceived Ease-of-Use Perceived Usefulness 27 9

10 in Figure 6 this peak goes almost unnoticed unless considerable cognitive mental effort is expended. 28 Cleveland, W. S. and R. McGill (1984). "Graphical perception: Theory, experimentation, and application to the development of graphical methods." Journal Nicholas of the American C. Romano, Statistical Jr. Association 79(387): Experimental Design and Predictions Treatment Classic EVM Line Graphics (S S Curves) EVM Area Graphics Attention Load (Effort Over Time) High Low Task Completion time Task Correctness Score Higher (Inefficient Lower Slow) (more errors) Lower Higher (Efficient, (Fewer Fast) Errors) 29 Experimental Control Procedures Same Room, computers and Facilitator Computers Locked down Identical Script except for LINE/AREA Identical Training Same Data and Examples Identical Status Determination Task 10 Identical Questions random question presentation 30 10

11 Participants 42 Students taking university business classes Average Age 30.5 ( Line 30.3; Area 30.7) 57% Male (Line 52% male; Area 62% Male) Only 3 (0.0071%) reported at least moderate Project Management experience Only 7 (16%) reported slight or some EVM experience ( this could be a reference to the training session) 35 (83%) reported NO EVM experience 31 Short Training example Comparison 32 How to Determine Percent Complete $1,800k $1,600k $1,400k $1,200k $1,000k Saturn2 Project All 10 Tasks Combined 45% Remaining $800k $600k $400k $200k 55% Completed $0k 12/29/2006 1/5/2007 1/12/2007 1/19/2007 1/26/2007 2/2/2007 2/9/2007 2/16/2007 2/23/2007 3/2/2007 3/9/2007 3/16/2007 3/23/2007 3/30/2007 4/6/2007 4/13/2007 4/20/2007 4/27/2007 5/4/2007 5/11/2007 5/18/2007 5/25/2007 6/1/2007 6/8/2007 6/15/2007 6/22/2007 6/29/

12 How to Determine Percent Complete Saturn2 Project All 10 Tasks Individually $250k $200k $150k $100k $50k Task 1 45% Remaining 55% Completed $0k 12/29/2006 1/5/2007 1/12/2007 1/19/2007 1/26/2007 2/2/2007 2/9/2007 2/16/2007 2/23/2007 3/2/2007 3/9/2007 3/16/2007 3/23/2007 3/30/2007 4/6/2007 4/13/2007 4/20/2007 4/27/2007 5/4/2007 5/11/2007 5/18/2007 5/25/2007 6/1/2007 6/8/2007 6/15/2007 6/22/2007 6/29/ How to Determine Percent Complete Saturn2 Project All 10 Tasks Individually $120k $100k $80k Task 2 0% Remaining $60k $40k $20k 100% Completed $0k 1/12/2007 1/19/2007 1/26/2007 2/2/2007 2/9/2007 2/16/2007 2/23/2007 3/2/ How to Determine Percent Complete Saturn2 Project All 10 Tasks Individually $160k $140k $120k $100k $80k $60k $40k Task 3 10% Remaining 90% Completed $20k $0k 2/2/2007 2/9/2007 2/16/2007 2/23/2007 3/2/2007 3/9/2007 3/16/2007 3/23/2007 3/30/2007 4/6/2007 4/13/2007 4/20/2007 4/27/

13 How to Determine Percent Complete Saturn2 Project All 10 Tasks Combined 55% Completed 45% remaining 37 Task 1 Task 2 Task 3 How to Determine Percent Complete Saturn2 Project All 10 Tasks Individually 100% Completed 55% Completed 90% Completed 100% 0% 90% remaining 10% remaining 45% remaining Estimated percent complete 55% 38 Task: Answer 10 Project Status Questions Progress Questions: Which task is 100% complete? Which task is less than 25% complete? Schedule Questions: Is the Jupiter3 Project ahead of schedule, on schedule, or behind schedule? Which task is exactly on schedule? Cost Questions: Is the Jupiter3 Project (Overall) under, on, or over budget? Which task has the largest budget? Which tasks are more than 10% under budget? Which tasks are more than 10% over budget? Which task is approximately 5% under budget? Which task is approximately 7% over budget? 39 13

14 40 41 Preliminary Results Efficiency and Effectiveness 42 14

15 Time (minutes) All 10 Questions Efficiency Line Charts Area Charts Mean Min Max 6.3 Participants that used Area charts completed the task 3.4 times faster than those that used Line Charts. Statistical Results All 10 N=42 Time ANOVA F r (effect size) sig (p) Very Large Significant 43 Time (minutes) Progress-Related Questions Efficiency Line Charts Area Charts Mean Min Max 3.0 Participants that used Area charts completed the task 2.4 times faster than those that used Line Charts. Statistical Results Progress N=42 Time ANOVA F r (effect size) sig (p) Large Significant 44 Time (minutes) Schedule-Related Questions Efficiency Line Charts Area Charts Mean Min Max Participants that used Area charts completed the task 2.9 times faster than those that used Line Charts. Statistical Results Schedule Time ANOVA F r (effect size) sig (p) N=42 Large Significant 45 15

16 Time (minutes) Cost-Related Questions Efficiency 8.01 Line Charts Area Charts Mean Min Max Participants that used Area charts completed the task greater than 4 times faster than those that used Line Charts. Statistical Results Cost N=42 Time ANOVA F r (effect size) 0.79 Very large sig (p) Significant 46 Score (Points) All 10 Questions Effectiveness 12.9 Line Charts Area Charts Mean Min Max Participants that used Area charts scored 48% higher in terms of correct answers than those that used Line Charts. Statistical Results All 10 N=42 Score ANOVA F r (effect size) sig (p) Large Significant 47 Score (Points) Progress-Related Questions Effectiveness Line Charts Area Charts Mean Min Max Participants that used Area charts scored only slightly higher in terms of correct answers than those that used Line Charts. Statistical Results Progress N=42 Time ANOVA F r (effect size) 0.04 Very small sig (p) 0.82 NOT Significant 48 16

17 Schedule-Related Questions Effectiveness Score (Points) Line Charts Area Charts Mean Min Max Participants that used Area charts scored 26% higher in terms of correct answers than those that used Line Charts. Statistical Results Schedule Score ANOVA F r (effect size) sig (p) N=42 Small Significant 49 Score (Points) Cost-Related Questions Effectiveness Line Charts Area Charts Mean Min Max Participants that used Area charts scored 59% higher in terms of correct answers than those that used Line Charts. Statistical Results Cost N=42 Score ANOVA F r (effect size) sig (p) Large Significant Initial Answers to Questions 1 and 2 1. Can EVM-Area Graphs enhance the efficiency of Status Determination? YES for all three types of questions Both Statistically and Practically Significant 2. Can EVM-Area Graphs enhance the effectiveness of Status Determination? YES for all three types of questions Both Statistically and Practically Significant 51 17

18 Take Home Message EVM is the most accurate method known for tracking status However if status is not communicated: efficiently (quickly and automatically) effectively (with fewer errors) Accuracy of the EVM data is not the real issue. The real issue is EVM data presentation for communication of status 52 Implications The usability and usefulness of classic project status charts is called into question. Area charts show promise as a method of improving communication regarding project status. EVM status determination can be learned by Novices 53 Further Research Can area charts and line charts be combined to draw from the strengths of each? Can area charts be used for more than passive communication of project status? Can area charts be used as a medium for project team collaboration? Can area charts be used in all phases of the project life cycle, including planning and estimating. Mapping Brain Activity to verify Right/Left 54 18

19 Questions? 55 19