Key Reliability Standard Spot-Check Report

Transcription

1 Key Reliability Standard Spt-Check Reprt CIP-001-2a, EOP-004-1, EOP December f Peachtree Rad NE Suite 600, Nrth Twer Atlanta, GA

2 Table f Cntents Preface...3 Executive Summary...4 KRSSC Intrductin...6 KRSSC Objectives...7 KRSSC Scpe...8 Reliability Standard Selectin...8 Audit Selectin...9 Scpe Expansin KRSSC Methdlgy KRSSC Risk KRSSC Team KRSSC Recmmendatins Recmmendatin: Attestatins Recmmendatin: Event Identificatin Recmmendatin: Evidence Validatin KRSSC Best Practices Best Practice: Training Evidence Review Best Practice: Attestatin Supprting Evidence Best Practice: Audit Recrds Best Practice: Dcument Management Review EOP Cnclusin and Next Steps Cnfidential (Nn-Public) Appendices f 23

3 Preface The Nrth American Electric Reliability Crpratin ( NERC ) is a nt-fr-prfit electric reliability rganizatin (ERO) whse missin is t ensure the reliability f the Bulk-Pwer System ( BPS ) in Nrth America. NERC develps and enfrces Reliability Standards; annually assesses seasnal and lng term reliability; mnitrs the BPS thrugh system awareness; and educates, trains, and certifies industry persnnel. NERC is subject t versight by the Federal Energy Regulatry Cmmissin (FERC) and is under similar bligatins in Canada, as well as a prtin f Baja Califrnia Nrte, Mexic. NERC s jurisdictin includes users, wners, and peratrs f the BPS, which serves mre than 334 millin peple. In 2007, FERC apprved agreements by which NERC delegates its authrity t mnitr and enfrce cmpliance t eight Reginal Entities (REs), as shwn in the map and crrespnding table belw. The members f the REs cme frm all segments f the electric industry: investr-wned utilities; federal pwer agencies; rural electric cperatives; state, municipal, and prvincial utilities; independent pwer prducers; pwer marketers; and end-use custmers. These entities accunt fr virtually all the electricity supplied in the United States, Canada, and a prtin f Baja Califrnia Nrte, Mexic. FRCC MRO NPCC RFC SERC SPP-RE TRE WECC Flrida Reliability Crdinating Cuncil Midwest Reliability Organizatin Nrtheast Pwer Crdinating Cuncil ReliabilityFirst Crpratin SERC Reliability Crpratin Suthwest Pwer Pl Reginal Entity Texas Reliability Entity Western Electricity Crdinating Cuncil 3 f 23

4 Executive Summary The Key Reliability Standard Spt Check (KRSSC) prgram is a NERC versight prject that is designed t enhance reliability thrugh effective and rigrus cmpliance audits thrughut Nrth America. This is accmplished by capturing a snapsht f the prcedures and prcesses being used by the eight REs t audit registered entities fr cmpliance t the Reliability Standards. NERC Reliability Standard CIP-001-2a Sabtage Reprting was selected fr this KRSSC. CIP-001-2a states that disturbances r unusual ccurrences suspected r determined t be caused by sabtage shall be reprted t the apprpriate systems, gvernmental agencies, and regulatry bdies. During the CIP-001-2a evidence review, the KRSSC team decided t expand the scpe f the KRSSC t include EOP Disturbance Reprting because f the verlap with EOP-004-1, which addresses disturbances r unusual ccurrences that jepardize the peratin f the Bulk Electric System (BES). Anther interrelated Reliability Standard identified by the KRSSC team is EOP-004-2, which has a January 1, 2014 mandatry enfrcement date. FERC simultaneusly apprved EOP Event Reprting and the retirement f EOP and CIP-001-2a. Accrdingly, this KRSSC reprt will prvide a histrical review f EOP and CIP-001-2a as well as a frward lk at EOP The inclusin f EOP prir t the January 1, 2014 mandatry enfrcement date prvides Reginal Entities and registered entities with further guidance t aid cmpliance effrts. The fllwing criteria were used t select the Reliability Standard fr inclusin in the 2013 KRSSC prgram: Recently enfrceable Reliability Standard; Inclusin in Tier 1 f the 2012 Actively Mnitred List ( AML ); Histrical ranking f mst vilated Reliability Standards; 2012 ranking f mst vilated Reliability Standards; and Histry f knwn cncerns. Tw audits perfrmed in 2012 frm each f the REs were selected fr review. The team perfrming the KRSSC determined and dcumented its cnclusins as Recmmendatins and Best Practices. Recmmendatins identify areas f imprvement and prvide specific recmmendatins t be utilized by all REs t remediate the issues nted. Best Practices identify certain practices r elements f RE audit prcesses that enhance auditing practices and appraches. Fr this KRSSC, three items were identified as Recmmendatins and fur items were identified as Best Practices. The subject matter f these items relates t varius aspects f RE audit practices, including: Recmmendatins Attestatins: With respect t Attestatins, NERC recmmends the fllwing: Crrbrating evidence shuld be btained: sample perating lgs, SME interviews, review f available event infrmatin; All attestatins shuld be signed by an entity subject matter expert (SME) with intimate knwledge f the event being attested; If Reliability Standard Audit Wrksheets (RSAWs) are accepted as attestatins, REs must ensure that the entity statement is clearly defined, cvers all elements f the requirement, and is signed by an entity SME with intimate knwledge f the situatin being attested; and REs frmalize written prcedures regarding the acceptance f attestatins. 4 f 23

5 Executive Summary Event Identificatin: When validating reprtable event infrmatin, NERC recmmends the fllwing: At a minimum, all REs review the U.S. Department f Energy s (DOE) annual OE-417 summaries lcated at t validate reprtable event infrmatin prvided by the audited entity; REs utilize their EA department r staff, as applicable; and REs develp and frmalize written prcedures regarding the validatin f reprtable event infrmatin prvided by the registered entity. Evidence Validatin: when assessing an entity s prcedures as a means f validating a specific cnclusin regarding that entity s cmpliance, NERC recmmends the fllwing: REs fllw Generally Accepted Gvernment Auditing Standards (GAGAS) Chapter 6, 1 which cvers evidence validatin, whenever accepting entity prcedures as evidence f cmpliance. Best Practices Training Evidence Review NERC ntes that ne RE assigns the audit subteam perfrming the PER- 005 detailed training recrd t review all Reliability Standards where training-related dcuments have typically been prvided as evidence by registered entities. This ensures n gaps in awareness regarding varius Reliability Standards and entity persnnel. Attestatin Supprting Evidence NERC ntes that anther RE des nt cnsider the presence f electrnic dcuments/prcedures t be sufficient alne as evidence f being prvided guidance. The RE always requires supprting evidence, such as the presence f hard cpy prcedures and SME interviews. The RE requires registered entity persnnel t shw the audit team where the prcedure is lcated and walk the audit team thrugh the prcedure t demnstrate they understand it. Audit Recrds NERC ntes that ne Reginal Entity uses phtgraphs t dcument the evidence reviewed during cntrl center turs. Observed evidence that is difficult fr the audit team t dcument is phtgraphed t be included in the permanent audit file. Examples f bserved evidence include: phne turrets, prcedure lcatins (electrnic and paper), Physical Security Perimeter, etc. This practice is applicable t multiple Reliability Standards where bservatin is a methd f evidence gathering. Dcument Management Review NERC ntes that ne Reginal Entity perfrms a dcument management review during the exit briefing f all audits. This review pints t ptential weaknesses and areas f imprvement in the registered entities dcument management prcess. 1 See Gvernment Auditing Standards 2011 Internet Versin 5 f 23

6 KRSSC Intrductin KRSSC Intrductin NERC has the respnsibility t assess the REs implementatin f the NERC Cmpliance Mnitring and Enfrcement Prgram (CMEP) and determine whether the prgram, as implemented by the REs, effectively meets the requirements under the NERC Rules f Prcedure (ROP), the CMEP and the crrespnding annual implementatin plan, 2 as well as the Reginal Delegatin Agreements (RDAs). 3 The KRSSC prgram is ne element f the RE Audit Prgram 4 that NERC uses t perfrm RE versight. The KRSSC prgram is based n the GAGAS fr perfrmance audits. 5 NERC staff develped this GAGAS-based KRSSC audit reprt 6 t cmmunicate the findings and recmmendatins that address areas needing imprvement and t enhance cnsistency acrss all REs. A summary reprt that addresses crss-reginal issues identified during the KRSSC is prvided t all REs. Each RE is prvided with a cnfidential appendix that illustrates its reginal-specific Recmmendatins and Best Practices. The cnfidential appendices are prvided nly t the crrespnding RE and are nt shared with ther REs. The KRSSC Prgram is designed t imprve cnsistency f cmpliance auditing implementatin by capturing a snapsht f the prcedures and prcesses being used by the eight REs t audit registered entities fr cmpliance with the Reliability Standards. Emphasis is placed n selecting Reliability Standards that have been effective fr an entire CMEP year in rder t fairly evaluate the cnsistency f RE auditing practices. NERC staff reviews the results f each Key Reliability Standards Spt Check fr ways t imprve the assciated RSAW. The KRSSC prgram highlights variatins in RE prcesses and prcedures that require additinal guidance t imprve perfrmance and cnsistency. 2 See NERC Rules f Prcedure at Sectin 400 and Appendix 4A, Audit f Reginal Entity Cmpliance Prgrams, 3 See Reginal Delegatin Agreements, psted n the NERC website at: 4 See Rules Cncerning Certificatin f the Electric Reliability Organizatin; and Prcedures fr the Establishment, Apprval and Enfrcement f Electric Reliability Standards, Order N. 672 at P 773, FERC Stats. & Regs. 31,204, rder n reh g, Order N. 672-A, FERC Stats. & Regs. 31,212 (2006) ( We cntemplate that a cmpliance audit f the ERO wuld typically invlve an examinatin f the ERO s nging cmpliance with statutry and regulatry criteria fr certificatin and its perfrmance in carrying ut its respnsibility t versee the cmpliance with and enfrcement f Reliability Standards. The Cmmissin, hwever, maintains the flexibility t determine the applicable scpe f a particular audit. The Final Rule eliminates the prpsed peridic Cmmissin cmpliance audit f each Reginal Entity. Instead, we require the ERO peridically t audit each Reginal Entity s nging cmpliance with relevant statutry and regulatry criteria and perfrmance in enfrcing Reliability Standards and reprt the results t the Cmmissin. A Cmmissin audit f the ERO may include a review f the adequacy f the ERO s audits f Reginal Entities. Mrever, the Cmmissin retains the authrity t participate in any ERO cmpliance audit f a Reginal Entity r cnduct its wn cmpliance audit in respnse t particular circumstances that may warrant Cmmissin participatin r interventin. ) 5 See United States Gvernment Accuntability Office Gvernment Auditing Standards at: 6 See Chapter 7 f United States Gvernment Accuntability Office Gvernment Auditing Standards, 6 f 23

7 KRSSC Objectives KRSSC Objectives The KRSSC seeks t enhance BPS reliability thrugh a number f bjectives. 7 The bjectives f the KRSSC are t: Identify issues relating t the RE s cmpliance determinatins with the selected Reliability Standard as detailed within the audit reprt and as supprted by the cllected evidence, RSAWs, and NERC staff cmmunicatins with the audit team; Identify issues RE audit teams experience when evaluating cmpliance with the selected Reliability Standard; Identify areas in RE audit appraches that may require additinal guidance t imprve prcesses r t prmte cnsistency in audit appraches; Determine whether the evidence prvided substantiates the findings f the RE audit team; Ensure selected RE audits adhere t the ROP, CMEP, and RDAs; 8 Ntify REs f the results and recmmendatins f the KRSSC in a written reprt; and Prvide transparency f NERC s review f RE prcesses and applicable NERC guidance. 7 See Sectin f United States Gvernment Accuntability Office Gvernment Auditing Standards, 8 See Sectin 6.15 f United States Gvernment Accuntability Office Gvernment Auditing Standards, 7 f 23

8 KRSSC Scpe KRSSC Scpe Reliability Standard Selectin The criteria used t select the Reliability Standard fr inclusin in the 2013 KRSSC prgram are as fllws: Recently enfrceable Reliability Standard; Inclusin in Tier 1 f the 2012 AML; Histrical ranking f mst vilated Reliability Standards; 2012 ranking f mst vilated Reliability Standards; and Histry f knwn cncerns. NERC guidance, FERC interpretatin, Reliability Standards Issues database. The KRSSC Team selected CIP-001-2a frm a ppulatin f Reliability Standards cnsisting f all Tier 1 standards that became effective, r had a new versin becme effective, within 12 mnths f January 1, 2012 that were included in the 2012 AML. T make a final determinatin n the Reliability Standard t use fr the KRSSC, the KRSSC Team examined each standard s ptential risk r ability t prduce severe and negative impacts n reliability f the BPS if vilated. Additinally, the Team reviewed vilatin histry: Table 1: Ptential KRSSC Standards Standard Effective Date Histrical ( ) Vilatins CMEP Year Vilatins CIP-001-2a Sabtage Reprting 10/1/ FAC Crdinatin f Plans Fr New Generatin, Transmissin, and End-User Facilities IRO Reliability Crdinatin - Facilities IRO Reliability Crdinatin - Operatins Planning MOD-001-1a Available Transmissin System Capability 10/1/ /1/ /1/ /1/ MOD Capacity Benefit Margin 4/1/ MOD Transmissin Reliability Margin Calculatin Methdlgy 4/1/ In additin t the afrementined vilatin histry, the KRSSC Team cnsidered whether NERC guidance was issued fr the ptential Reliability Standards. 9 The vilatins frm may relate t a retired versin f the Reliability Standard and nt necessarily t the current versin. Fr instance, the 494 vilatins fr CIP-001 include vilatins frm CIP-001-1, CIP-001-2, and CIP-001-2a. 8 f 23

9 KRSSC Scpe CIP-001-2a was the Reliability Standard recmmended fr the 2013 KRSSC. This recmmendatin was based n the selectin criteria and the list f ptential Reliability Standards prvided abve. CIP-001 is the nly Reliability Standard that appeared n the histrical list f mst vilated Reliability Standards and n the 2012 CMEP list f mst vilated Reliability Standards. CIP-001 has 494 vilatins frm June 18, 2007, thrugh December 31, In additin, CIP-001-2a has 42 vilatins frm January 1, 2012, thrugh December 31, 2012, and was included n the histrical list f mst vilated standards fr the 2012 CMEP year. Nne f the ther ptential Reliability Standards were n either list f mst vilated Reliability Standards. NERC prvided guidance and utreach n CIP-001 and several ther Reliability Standards; fr instance, CAN related t CIP-001 R1 Sabtage Reprting Prcedure, thugh it was later retired. 10 Additinally, based n industry feedback, CIP-001-2a has five issues identified fr reslutin in the frm f guidance r future versins f the Reliability Standard. The Reliability Standard serving as the initial pint f fcus fr this KRSSC was selected based upn a number f cnsideratins. Fllwing is a list f sme f the factrs affecting the decisin t select CIP-001-2a Sabtage Reprting: CIP-001 is included in Tier 1 f the 2012 AML, indicating there is a gd ppulatin f audits fr selecting samples; CIP-001 was the sixth 11 mst vilated Reliability Standard frm ; and CIP-001-2a was the tenth 12 mst vilated Reliability Standard in Audit Selectin NERC selected the audits based n the functins being audited that are applicable t CIP-001-2a and the audit having been perfrmed between January 1, 2012, and December 31, The audit selectins were als based n the fllwing cnsideratins: The KRSSC Team selected tw audits frm each RE t assess cnsistency. The KRSSC Team selected ne audit perfrmed tward the beginning f the year and ne audit perfrmed tward the end f the year, t assess ptential changes in apprach ver time. The preference is t have each audit perfrmed six mnths apart, when pssible. The KRSSC Team selected ne large entity and ne small entity frm each RE, t assess ptential scalability challenges. Fr the purpse f this KRSSC, entity size is relative t the ptential effect n the BPS and/r respnsibility fr the BPS as measured by generating capacity. The KRSSC Team evaluated size as fllws: Small is cnsidered t be <3000 MW f generating capacity; and Large is cnsidered t be >3000 MW f generating capacity. 10 See als high-level review f CAN-016. Available at: Level%20Review.pdf 11 See page 14 f 12 See page 13 f 9 f 23

10 KRSSC Scpe The KRSSC Team als evaluated the ptential and actual risk f the audited registered entity in part as determined by a review f key characteristics f the entity: Organizatin and structure; Registered functins; System size; Neighbring entities; Transmissin circuit miles; Number f intercnnectins; Generatin prtfli; and Peak demand. Scpe Expansin The scpe f the KRSSC was expanded t include EOP Disturbance Reprting based n KRSSC Team bservatins during the CIP-001-2a evidence review. The KRSSC Team determined that the prcedures fr CIP- 001 and EOP-004 were ften included in the same registered entity prcedures. On June 20, 2013, FERC apprved EOP Event Reprting, alng with the retirement f existing Reliability Standards EOP and CIP-001-2a. Accrdingly, the KRSSC Team expanded the scpe t als include this new Reliability Standard. The KRSSC Team perfrmed a traditinal backward-lking KRSSC n CIP-001-2a and EOP , and a frward-lking review f EOP prir t its January 1, 2014 mandatry enfrcement date. NERC believes that this expanded review prvided additinal value t REs and registered entities by freshadwing EOP Industry may be able t efficiently and effectively implement peratins and planning business practices t address EOP due t the infrmatin prduced by this freshadwing. 10 f 23

11 KRSSC Methdlgy KRSSC Methdlgy The KRSSC methdlgy is designed t supprt the KRSSC bjectives. 13 The methdlgy includes the steps that are perfrmed t address the KRSSC bjectives, reduce audit risk t an acceptable level, and prvide reasnable assurance that the evidence requested is sufficient and apprpriate. It shuld be nted that the methdlgy specific t CIP-001-2a is addressed in a fllwing sectin. The KRSSC Prgram prcess fr reviewing the RE audits f the selected Reliability Standard is as fllws: Prvide REs ntificatin packages annuncing the cmmencement f the KRSSC fr the selected Reliability Standard. The ntificatin package cmmunicates an verview f the bjectives, scpe, methdlgy, and timing f the perfrmance audit and planned reprting. 14 The ntificatin package is prvided t RE Executive Management and Cmpliance Management. Select audits recently perfrmed by each RE that invlved the selected Reliability Standard. The audits chsen fr review are selected by NERC staff based upn varius criteria including, but nt limited t, the registered functins being audited that are applicable t the selected Reliability Standard, having been cmpleted in calendar year 2012, and perfrmed six mnths apart, when pssible. The NERC KRSSC Team is led by a senir Audit Assurance and Oversight (AAO) staff member wh prvides regular prgress updates t the Manager f AAO. 15 NERC staff perfrmed a detailed review f the selected Reliability Standard and related dcumentatin including, but nt limited t: Case Ntes, CANs, Cmpliance Analysis Reprts (CARs), dismissal analysis, Reliability Standards issues database, Standard and Prject Crss Reference, and future versins f the selected Reliability Standard. The KRSSC team wrks tgether t review each selected audit t assess the use f tls, prcesses and prcedures and indentify any issues f incnsistency acrss the RE s ftprint. NERC staff review the 16 RE s audit files based n identified criteria that is specific t the selected Reliability Standard. The KRSSC review includes, but is nt limited t: The Nn-Public Audit Reprt; Cmpleted RSAWs; Auditr ntes and wrking papers; RE evidence tracking recrds; Registered entity evidence files; SME respnses; and Mitigatin plans pen during the audit. NERC staff develps a set f cmmn crss-reginal questins based n issues identified during the review. Telecnferences are held with cmpliance persnnel frm each RE t discuss remaining 13 See Sectin 6.03 f United States Gvernment Accuntability Office Gvernment Auditing Standards, 14 See Sectin 6.47 f United States Gvernment Accuntability Office Gvernment Auditing Standards, 15 See Sectin 6.47c f United States Gvernment Accuntability Office Gvernment Auditing Standards, 16 See Sectin 6.57 f United States Gvernment Accuntability Office Gvernment Auditing Standards, 11 f 23

12 KRSSC Methdlgy questins regarding the selected audit, respnd t a set f crss-reginal questins, and prvide input n issues related t incnsistency in auditing the selected Reliability Standard. An audit reprt is develped t cmmunicate the findings and recmmendatins that address areas needing imprvement and t enhance cnsistency acrss all REs. 17 A summary reprt that addresses multi-reginal issues discvered during the KRSSC is prvided t all REs. Cnfidential appendices that address reginal-specific audit prcesses and issues are prvided t each crrespnding RE. Prir t the issuance f the KRSSC audit reprt, the manager f AAO and the directr f Cmpliance Operatins perfrm a supervisry review t ensure the wrk supprts findings, cnclusins, and 18 recmmendatins cntained in the reprt. 17 See Chapter 7 f United States Gvernment Accuntability Office Gvernment Auditing Standards, 18 See Sectin 6.54 f United States Gvernment Accuntability Office Gvernment Auditing Standards, 12 f 23

13 KRSSC Risk KRSSC Risk The audit risk 19 f the KRSSC is the pssibility that the NERC audit findings may nt prduce the same results as the RE audit teams findings. T reduce the audit risk, the audit plan includes a methdlgy that ensures the NERC audit team has access t all evidence reviewed and files used by the RE audit team. It shuld be nted that NERC staff base their findings n a review f the audit material prvided by the registered entities alng with insight gained thrugh verbal and written cmmunicatins with the REs. NERC staff d nt knw the cntent r nature f the RE audit teams cnversatins with the registered entity SMEs. In the event the KRSSC review des nt prduce the same results as the RE audit teams findings, NERC will ntify the RE f the discrepancy in rder t receive additinal infrmatin r request an RE review. As applicable, NERC will wrk cllabratively t reslve any identified discrepancies. 19 See Sectin 6.05 f United States Gvernment Accuntability Office Gvernment Auditing Standards, 13 f 23

14 KRSSC Team KRSSC Team The KRSSC Team is cmprised f NERC Cmpliance Operatins staff. This KRSSC als included bservers frm FERC and the Cmpliance and Certificatin Cmmittee (CCC). The KRSSC Team and bservers included: Table 4: KRSSC Team Name Craig Struck Denise Hunter Gilbert Lwe Terry Bilke Jasn Marshall Jennifer Flandermeyer Title Senir Cmpliance Auditr, KRSSC Team Lead Cmpliance Auditr FERC Observer CCC Observer (CCC Chair) CCC Observer CCC Observer 14 f 23

15 KRSSC Recmmendatins KRSSC Recmmendatins The KRSSC team identified thirteen findings based n the detailed review f RE wrking papers and RE respnses t the KRSSC questinnaires. These findings have been identified as ptential surces f incnsistency thrughut the ERO in terms f audit appraches. T remediate these incnsistencies, the KRSSC team has develped eight specific recmmendatins. Recmmendatin: Attestatins The KRSSC team identified that the REs used different criteria fr acceptance and dependence f attestatins. The KRSSC team asked the REs t explain the criteria used when accepting attestatins. Is crrbrating evidence required; hw d REs determine the signatry f an attestatin is in a psitin t have knwledge f the event; and are RSAW statements accepted as attestatins and/r evidence? Based n KRSSC team bservatins and RE questinnaire respnses, NERC recmmends the fllwing: Crrbrating evidence shuld be btained: sample perating lgs, SME interviews, review f available event infrmatin, etc. All attestatins shuld be signed by an entity SME with intimate knwledge f the event being attested. If RSAWs are accepted as attestatins, REs must ensure that the entity statement is clearly defined, cvers all elements f the requirement, and is signed by an entity SME with intimate knwledge f the situatin being attested. REs shuld frmalize written prcedures regarding the acceptance f attestatins. Recmmendatin: Event Identificatin The KRSSC team s review f audit wrk papers identified incnsistencies within REs regarding their prcess fr validating reprtable event infrmatin. The KRSSC team asked each RE t explain their prcedure, and what resurces were used t determine whether a registered entity experienced any reprtable events during the audit perid. RE respnses indicated that nt all REs have frmalized written prcedures. Based n KRSSC team bservatins and RE questinnaire respnses, NERC recmmends the fllwing: At a minimum, all REs shuld review the DOE s annual OE-417 summaries lcated t validate reprtable event infrmatin prvided by the audited entity. Additinally, if applicable, REs shuld utilize their EA departments. Finally, REs shuld develp and frmalize written prcedures regarding the validatin f reprtable event infrmatin prvided by the registered entity. Recmmendatin: Evidence Validatin The KRSSC team identified that prcedure dcuments cited as evidence t validate a cnclusin were nt sufficient and apprpriate evidence. The cited evidence did nt reflect authrizatin, revisin histry, effective date, review date, r any ther attributes that wuld signify apprpriate evidence. The KRSSC team asked each RE t describe the prcess used t determine entity prcedures were cmplete, cncise, authrized, and reviewed. Based n KRSSC team bservatins and RE questinnaire respnses, NERC recmmends the fllwing: REs fllw GAGAS Chapter 6 20 which cvers evidence validatin, whenever accepting entity prcedures as evidence f cmpliance. 20 See Gvernment Auditing Standards 2011 Internet Versin 15 f 23

16 KRSSC Best Practices KRSSC Best Practices The KRSSC Team identified certain practices, r elements f RE audit appraches, that represent Best Practices. These Best Practices were based n the detailed review f RE audit wrking papers and respnses t the KRSSC questinnaires. The Best Practices shuld be used by all REs within the ERO. Assciated details fllw. Best Practice: Training Evidence Review One RE has established the practice whereby the audit subteam perfrming the persnnel recrd review required under PER-005 reviews all Reliability Standards where training-related dcuments are prvided as evidence by registered entities. This practice ensures n gaps in awareness regarding the evidence prvided t dcument the training being dne by persnnel t cmply with Reliability Standards. Tasking the PER-005 audit subteam with keeping training recrds as evidence means that the recrds are available fr any Reliability Standard being audited. The RE nted that as its auditrs gained experience perfrming audits, they bserved that entities were prviding the same training-related evidence fr varius Reliability Standards. Fr instance, the RE uses audit subteams based n Reliability Standard family, such as Persnnel Perfrmance Training and Qualificatins (PER), Emergency Operatins (EOP), Prtectin and Cntrl (PRC), etc. These separate subteams may als request the same discrete training-related evidence that the PER-005 audit subteam tracks. The RE s auditrs nticed that the centralized requirement is fr the entity t have a prcess t address the lifecycle f training (i.e., requirements, audience, bjectives, frequency, initial, and/r cntinuing). It was a natural prgressin fr auditrs f the discrete training requirement t discuss with the PER-005 subteam during nsite audit caucuses that the training is part f the prgram. This is a lgical apprach, especially as Reliability Standards get rlled int thers and the specific training requirement may disappear (i.e., CIP-001 int EOP-004). This practice cannt be used fr registered entities wh are nt BAs/RCs/TOPs because PER-005 is nt applicable t them. Hwever, this practice is used by the RE fr all audits f BAs/RCs/TOPs that are perfrmed nsite. This practice may be necessary as mre standards are rlled int ther standards (as CIP-001-2a and EOP are being rlled int EOP-004-2). This practice may prve mre efficient and rigrus as the RSAW audit appraches fr discrete requirements are changed in level r expectatin. As an example, the PRC-001 R1 RSAW is being revised t remve the peratr as the sle persn required t have familiarity and knwledge f prtectin systems. The prpsed RSAW changes will allw the entity t identify any persnnel f its chice, including smene ther than the peratr, t be respnsible fr the requirement. Under this scenari, the PER- 005 subteam wuld audit this specific nuance, as well as all required training, which shuld be knwn t the BA/RC/TOP entity s Training Crdinatr in determining required training. This is a test f hw well the Training Crdinatr and the SMEs fr their respective NERC standards cmmunicate training needs. Best Practice: Attestatin Supprting Evidence One f the REs des nt cnsider the presence f electrnic dcuments/prcedures t be sufficient n their wn as evidence f being prvided guidance. The RE always requires supprting evidence, such as the presence f hard cpy prcedures and SME interviews. The RE requires registered entity persnnel t shw the audit team where the prcedure is lcated and walk the audit team thrugh the prcedure t demnstrate an understanding f the prcedure. The RE utilizes multiple methds t cnfirm evidence. These methds are usually a cmbinatin f inquiry, bservatin, walk-thrugh, and re-perfrmance. This apprach is utilized by auditrs and is cnsistent with generally accepted audit practices. 16 f 23

17 KRSSC Best Practices If a registered entity prvides a prcedure nly (whether electrnic r hard cpy) as evidence f prviding guidance t their persnnel, the RE always requests crrbrating evidence. Examples f what the RE might request are: 1) s addressed t the apprpriate persnnel making them aware f the lcatin and nature f the prcedure; 2) recrds f persnnel training n the prcedure; r 3) a dcumented respnse frm the registered entity s persnnel they have read and understand the prcedure. The RE might als request that apprpriate persnnel demnstrate t the audit team where the prcedure is lcated (electrnically r physically) and guide the audit team thrugh the prcedure t demnstrate the emplyee understands the prcedure. In the past, there have been instances in which prcedures merely existed and apprpriate persnnel were nt aware f them, did nt understand them, r did nt knw hw t find them. Where a registered entity nly prvides its persnnel with electrnic versins f a prcedure (e.g., cntrl rm envirnment prcedures), the RE, when apprpriate, recmmends that hard cpies are als available in the event f the lss f access t the electrnic dcumentatin. The RE has fund that this practice has resulted in imprved awareness by entities and has encuraged entities t prvide multiple mediums/platfrms fr guidance material t help ensure access availability and persnal cmprehensin methds. In several cases, an entity s internal cntrls regarding dcument management have been imprved because f the RE s auditing practice. Best Practice: Audit Recrds One f the REs uses phtgraphs t dcument the evidence reviewed during cntrl center turs. Observed evidence that is difficult fr the audit team t dcument is phtgraphed t be included in the permanent audit file. Examples f bserved evidence include: phne turrets, prcedure lcatins (electrnic and paper), Physical Security Perimeter, etc. This practice is applicable t multiple Reliability Standards where bservatin is a methd f evidence gathering. The RE auditrs d nt take phtgraphs during the walk-thrugh exercises. In rder t minimize the effect f cnducting an audit fr a real-time peratins center, the RE has instituted the practice f requesting screenshts f SCADA availability, alarms, and ther pertinent tls peratrs use, as well as phtgraphs f cmmunicatin equipment, prcedures, emergency plans, and ther types f bservable evidence in preparatin fr n-site interviews. This prvides RE auditrs with first glance assurance f devices and prcedures that will be later cnfirmed during the walk-thrugh. This practice allws auditrs t spend mre time and fcus n the peratr interviews. Althugh prviding screenshts and phtgraphs is nt required, registered entities have generally cperated with this request. Starting in 2014, initial audit packages will include a request fr screenshts and phtgraphs as described abve. It shuld be nted that the RE des nt request phtgraphs f Physical Security Perimeters. Best Practice: Dcument Management Review Anther RE perfrms a dcument management review during the exit briefing f all audits. This dcument management review pints t ptential weaknesses and areas f imprvement in the registered entity s dcument management prcess. The RE perfrms a review f hw the entity manages its audit cmpliance effrts and dcumentatin. The RE des nt cmment n the technlgy the registered entity uses fr dcument management. The auditrs d cmment n reliability, accuracy, validity, and sufficiency f the dcumentatin that supprts an entity s cmpliance activities. During the curse f an audit, the RE audit team will bserve and take nte f an entity s dcumentatin and the methdlgies it uses t track changes, versins, management review, and apprval, f the varius 17 f 23

18 KRSSC Best Practices dcuments. During the summary r exit briefing, the audit team will acknwledge an entity s effrt in dcumenting its evidence r make recmmendatins fr imprvement. This always serves as an pprtunity t remind entities that their evidence is nly as gd as their dcumentatin prcess. In several cases, internal cntrls regarding dcument management have been imprved because f this practice. 18 f 23

19 EOP EOP The scpe f the KRSSC was expanded t include EOP Disturbance Reprting based n KRSSC Team bservatins during the CIP-001-2a evidence review. The KRSSC was als expanded t include a frward-lking review f EOP Event Reprting prir t its January 1, 2014 mandatry enfrcement date. NERC believes that this expanded review prvided additinal value t Reginal Entities and registered entities. The majr differences between the new EOP and the sn-t-be-retired CIP-001-2a and EOP-004-1, as well as ther key pints industry shuld be aware f, are as fllws: The term Sabtage will be eliminated frm EOP Terrrism and vandalism that meets the threshlds fr reprting in Attachment 1 will be reprted t law enfrcement agencies pursuant t the Respnsible Entity s Operating Plan (Requirement R1). Respnsible Entities shuld review the EOP flwchart titled Example f Reprting Prcess Including Law Enfrcement. EOP will nt cver real-time reprting; it will be strictly fr after-the-fact reprting. Real-time reprting is accmplished thrugh the Reliability Crdinatr Infrmatin System (RCIS) and ther means f cmmunicatin and is cvered in ther standards and the Event Analysis Prgram. Respnsible Entities will be required t have a dated Operating Plan fr reprting events. The plan is t include cmmunicatin prtcls based n EOP-004 Attachment 1: Reprtable Events. Respnsible Entities are t validate all cntact infrmatin in the Operating Plan each calendar year. Respnsible Entities shuld retain their current Operating Plans and all versins issued since the last audit. Respnsible Entities are t reprt events per their Operating Plans within prescribed time frames. Respnsible Entities will be required t submit ne reprt fr each individual event. There may be reginal differences in reprting requirements as sme Regins may g beynd what is required in EOP-004. Respnsible Entities shuld retain a cpy f the cmpleted EOP-004 Attachment 2: Event Reprting Frms r DOE OE-417 frms as evidence f reprting an event and shuld als retain cpies f the cmmunicatins medium used as evidence f submitting the reprt. Duplicate reprting t the DOE and NERC may nt be required. Alternate reprts will be accepted by NERC if they include all infrmatin required by EOP Fr example, DOE will receive the DOE OE- 417 reprt; the same reprt can be submitted t NERC rather than creating a new ne based n Attachment 2. The requirement t analyze events will be remved. Hwever, further evaluatin as part f the ERO Event Analysis Prgram may be undertaken. Distributin Prviders (DP) will have special cnsideratins: DPs that d nt meet the Threshld fr Reprting, as identified in EOP Attachment 1, may nly have t cmply with R1 and R3. DPs will be required t have a simple Operating Plan t address all Event Types listed in EOP Attachment f 23

20 EOP The Operating Plan must state which Event Types are nt applicable. DPs are required t review EOP-004 Attachment 1: Reprtable Events annually t ensure applicability. DPs will be required t develp a detailed Operating Plan if applicable events are identified during the annual review f Attachment 1. DPs wh meet the threshld fr reprting are nly respnsible fr reprting events n BES facilities that they wn; they are nt respnsible fr reprting events they bserve n anther entity s facilities. The respnsibility fr reprting rests with the wner f the facility. Fllwing is a list f sme ptential challenges that Respnsible Entities may face in implementatin: Cnslidating and revising current CIP-001 and EOP-004 prcedures t align with EOP-004-2; Training apprpriate persnnel n the new prcedures; and Identifying actins fr each f the 18 new Event Types included in EOP-004 Attachment 1: Reprtable Events. Fllwing is additinal infrmatin t assist Respnsible Entities with identifying actins fr the new Event Types included in EOP-004: All Event Types must be addressed regardless f applicability t the Respnsible Entity. EOP-004 Attachment 1: Reprtable Events required t be reprted are: Damage r destructin f a Facility that results: in actins t avid a BES Emergency frm actual r suspected intentinal human actin frm physical threats t a Facility r BES cntrl center, excluding weather r natural disaster BES Emergency requiring public appeal fr lad reductin requiring system-wide vltage reductin f 3% r mre requiring manual firm lad shedding greater than r equal t 100 MW resulting in autmatic firm lad shedding greater than r equal t 100 MW Vltage deviatin f +/- 10% f nminal vltage n a Facility fr greater than r equal t 15 cntinuus minutes. IROL vilatin (all Intercnnectins) r SOL vilatin fr majr paths (WECC) based n time perating utside the limit. Lss f firm lad fr 15 minutes r mre based n previus year s demand. System separatin resulting in an island f 100 MW r mre. Generatin lss within 1 minute greater than r equal t 2000 MW in Eastern r Western Intercnnectin, r 1000 MW in ERCOT r Quebec Intercnnectin. Cmplete lss f ff-site pwer t a nuclear generating plant per the Nuclear Plant Interface Requirement (NIPR). Transmissin lss f 3 r mre BES Elements caused by a cmmn disturbance 20 f 23

21 EOP Unplanned BES cntrl center evacuatin fr 30 cntinuus minutes r mre. Cmplete lss f vice cmmunicatin capability affecting a BES cntrl center fr 30 cntinuus minutes r mre. Cmplete lss f mnitring capability affecting a cntrl center fr 30 cntinuus minutes r mre. 21 f 23

22 Cnclusin and Next Steps Cnclusin and Next Steps The KRSSC prgram ffers a unique and valuable perspective n the CMEP and its implementatin by REs with respect t audit appraches. Fr the KRSSC f CIP-001-2a and EOP-004-1, a number f areas have been identified where cnsistency amng the eight REs can be imprved. The recmmendatins and Best Practices fund in this reprt are related t six majr areas: Attestatins (and supprting evidence); Event identificatin; Evidence validatin; Training evidence review; Audit recrds; and Dcument management review. The KRSSC s gal is t bring t light differing interpretatins and applicatins regarding the subject Reliability Standards and RE audit practices and ensure that guidance and directin is develped that ensures cllabrative effrts tward ERO-wide bulk pwer reliability. Thus, t cntinue driving tward a mre cnsistent and selfaware ERO, the KRSSC prgram shuld be a nearly cnstant effrt, fcusing n thse Reliability Standards that are mst integral t mitigating risk. 22 f 23

23 Cnfidential (Nn-Public) Appendices Cnfidential (Nn-Public) Appendices Cnfidential Nn-Public Appendices address RE-specific tpics and have been redacted frm the public reprt; the Cnfidential Appendices are prvided nly t the apprpriate RE. 23 f 23