Episode 11 - NERC Lessons Learned Podcast

July 21, 2022, midnight by Chris Sakr | Last modified Jan. 5, 2023, 11:39 a.m.







PANDEMIC RESPONSE PART 1: EARTH STOOD STILL

Below is the full transcript for this episode. If you'd like to review or follow along with the original .pdf version of this NERC Lesson Learned, visit: 

https://www.nerc.com/pa/rrm/ea/Lessons%20Learned%20Document%20Library/LL20211001_Pandemic_Response.pdf

Chris Sakr: For most customers, electricity's just a bill we pay so our stuff keeps working. I've rarely heard anyone outside the power industry highlight how critical its people are. But without them, critical industries like healthcare, public safety, and almost any conceivable critical resource would vanish. As Tacoma Power System Operations Supervisor Cullen Ritchie put it...

Cullen Ritchie: In the case of control room operations for power, water, those different utilities, a lot of people don't know we exist. And so we had to get their attention to start with so that we could get their engagement and assistance.

Chris Sakr: COVID19 means a lot of things to a lot of people, not the least of which, it's been a global volume knob that turned up fast and sustained longer than almost anybody would've anticipated. Little problems got bigger and big unforeseen hardships—the kind you'd never planned for—well, they got bigger too. Here's Debra Smith, CEO of Seattle City Light, flashing back to early March, 2020.

Debra Smith: We were in incident command mode at the City of Seattle. And my mom passed, unexpectedly, which at the time when she passed, it was just, what do I need to do? It was pure logistics, and there was no opportunity to process that. Because at that point in the pandemic, we were in the starting blocks for what we thought was going to be a sprint.

Chris Sakr: Of course, that sprint became a marathon. And through hardships, personal and professional, the North American power industry faced unprecedented risks to keep society turning. On October 25th, 2021, NERC published a Lesson Learned entitled Pandemic Response. They focused primarily on utilities Pandemic Preparedness (or Business Continuity) plans, but when our team at WPP read it, we saw an opportunity, not just to look closely at these plans, leveraging our extensive pool of industry experts, but to use the document as a springboard. Over the next three episodes, we'll investigate how for over two uncertain years, the power industry bravely persevered through COVID19. This episode takes us from January through April of 2020, wherein the industry used their own continuity plans as springboards to keep their people safe. The two episodes after will carry us all the way up to 2022, as the industry adapted, evolved, and continued keeping the lights on. As a consumer, we hope you see what it took. If you're a power professional, we hope at every step you feel seen and, perhaps more importantly, proud. I'm your host, Chris Sakr, this is Pandemic Response Part One: Earth Stood Still.

News Anchor: On this Monday night, coronavirus confirmed in Canada. One official case, another presumptive case, their connection... This new case isn't surprising. Plus, new questions over how the virus spreads…

Brett Hallborg: Sometime in January, we had the first positive case of a person in the province being infected by the virus who had not traveled abroad or overseas or outside the country.

Chris Sakr: That's Brett Hallborg, BC Hydro's Senior System Control Manager. At the end of 2019, I personally remember lots of people saying this coronavirus was just another media circus, but by January, 2020, it started to feel like maybe there was some fire under all that smoke. Now, most utilities had and periodically tested pandemic or business continuity plans, but there's no stress test like the real thing, and it's nearly impossible to account for every potential hurdle. Here's Cullen again.

Cullen Ritchie: When I first came to work here about 13 years ago, we already had in existence what we called a "Pandemic Illness Response Plan". That plan had kind of fallen to the wayside. It was very dusty, it was very out of date, so to speak, but it did provide some guidance. It provided a little bit of a framework and, most importantly, some ideas and some options and some precedents.

Chris Sakr: PSCO's training coordinator, Tony Martinez.

Tony Martinez: If you don't use it, you lose it. I think when you create it, it was good for that time. But if there's a gap in between that you didn't have to use it, again, some of the things are not going to be beneficial. Had to go back in, make certain that all procedures were up to date, maybe some of the tasks that were a little dated, if you will, were kind of in our old pandemic routine or process. Definitely, some updating was required on our side as well.

Chris Sakr: Jason Bucholtz, AESO's Real Time Operations Manager.

Jason Bucholtz: Our BCP is regularly reviewed on a yearly basis, so I think that the only thing that was missing is kind of some of the details that we learned through this process, right? So now that we're going to update these documents to include that stuff, it will be part of our yearly review moving forward, I would assume.

Chris Sakr: You can test and retest your plan, but no one's got a crystal ball, and the pandemic you plan for may not be the one you get; you'll have to adapt. That seems important to clarify, because over at BC Hydro...

Brett Hallborg: ... And these plans were unfortunately very difficult to pull together. It wasn't like we could pull our pandemic response plan down off the shelf and blow the dust off it, refresh it, and roll it out, so we had to create something from scratch.

Chris Sakr: Unlike everyone else I spoke with, BC didn't have a pandemic specific plan, but they did have other continuity plans to draw from, so they spent the first two months of 2020 doing the same thing everyone else was doing: collecting and updating pieces of similar plans, making additions with available guidance in the few specifics anyone knew. By March, it started getting real.

News Anchor: Just in, breaking news here in Canada, public health officials say four new COVID19 cases have been confirmed in Ontario. That brings the total number in Canada to 24. South of the border, an outbreak is growing in Washington State. Jennifer Johnson has that angle.

Chris Sakr: Debra wasn't in Seattle. She was in Portland for some meetings, but stuck in her hotel room on a call. An emergency cabinet meeting for Seattle officials had been convened.

Debra Smith: And it was in this meeting they laid down the first set of things like, "Here are the stats, here's what's happening. Here are the things that we're tracking. No new contracts," basically, "Hiring freeze, no travel going forward," and all of these things were going to be effective like the following week. It was almost like, "Wow, is this really happening? I can't believe this is happening."

Chris Sakr: I think we all said something like that at one or maybe many points. Seattle and King County were becoming the US's COVID epicenter. Debra flew back, and when she returned to the office...

Debra Smith: And I think it was on a Monday, and I believe that my mom died on a Tuesday, and the city went into ICS mode. And I'm in the city's Emergency Operations Center, and literally we're all sitting there, we're thinking about everything. I was not only dealing with my utility, but we were making decisions about how to shut the city down. And I'm there thinking that this is a good place for me to be, except that, of course, I keep breaking into tears. And finally, it's clear to me that I do need to go back home, I need to be with my brother, etc. So I did go back, I flew back Friday night. And my grandson who was, I think, four at the time, broke his leg in three places jumping off of his bunk bed. So that happened literally while I was on the plane back to Eugene, and he was in Eugene. So then that was clear to me that that really was a good place for me to be.

Chris Sakr: The big hardships, those we never plan for, got bigger. Now, with the coasts getting hit hardest, Colorado had been getting a head start.

Tony Martinez: We were preparing for it. We had heard some of the news that was coming pretty much from the West Coast and then moving inland, if you will. What we did is we tried taking some precautions of ordering extra laptops, extra monitors, just in case our operators had to work from home or were given the option to work from home. So we were able to get those materials ordered, had them on site. We were right in the middle of our Winter/Spring training; this is called REMTAK. We meet offsite anywhere from 35 to 40 operators all at one time. Unfortunately, when we did finally cancel all that as well, we were right in the middle of the training. And so all of the Colorado entities are pretty close. We do get some folks from the South Dakota area. They unfortunately had to turn back around or catch flights back home. Our reliability coordinator is based out of Arkansas, so unfortunately they were in the air, had to turn back as well. We had a few of those little mishaps, but I think we caught it early enough, couple of flights and a couple gas tanks of gas getting back home and everything, I think everybody was happy with it.

Chris Sakr: A potential crisis averted. But no matter what part of North America you were in, the wolf was at the door. And while providing reliable electricity may one day be a remote endeavor, today it still requires at least a few people to share one room.

Cullen Ritchie: One of the very, very first things that we did was sit down and inventory our assets, such as our primary and backup control centers, our abilities for working remotely, the tools that we had, exactly what staffing we had available, and how that staff could be engaged, and then also the different rules that were in place. Being part of the City of Tacoma, being a municipal utility, we also had to consider the city's COVID response and the requirements that they were putting in place for all city employees, as well as our county health department recommendations and our labor partners with IBEW, and their concerns and requirements.

Brett Hallborg: We went through a major change in our control centers in 2008, wherein we consolidated five control centers. Now, we're in two large control centers. So we have 29 operating desks. They're not all staffed at all times, but during the day we'll have 22 people on shift.

Chris Sakr: BC had a unique obstacle to consider, their sizable and centralized operation staff. But no matter the size of any company’s shop, the job was the same: Figure out who needed to be in the control rooms and how to keep them safe.

Jason Bucholtz: Right out of the gate, we realized the importance of keeping controllers healthy was our number one priority, and how could we reduce the risk of exposure to a COVID case in our control room?

Brett Hallborg: The key things in there were staffing models, cleaning protocols, communication strategies, and a various number of other facets. But all this prefaced on the virus itself, what we knew about the virus, what we knew about its transmissibility, what we knew about the staffing levels that we were going to see or could see, and then we would have various triggers in our pandemic response plan based on staff illness. So should the situation gets worse and we start to lose a high number of staff, we would trigger various levels of this pandemic response plan.

Cullen Ritchie: In system operations, we created what we called a PACE plan, being “primary, alternate, contingency”, and “emergency”. And those basically set our different levels of staffing and how we could adjust and flex that staffing and flex those resources to go from a no impact, everybody's able to come to work and do their job and everybody's healthy, to a most extreme, in this case a outbreak situation, where we had very, very limited staff and we would have to use basically every tool in our toolkit to keep the lights on.

Jason Bucholtz: We started to do a more thorough, thought through process of, okay, what's going to happen here if controllers do start getting sick? So we built out a staged pandemic plan to deal with the different scenarios of controllers getting sick. The plan was used to help guide the team when operators started getting sick and how we would reduce to different levels of staffing if required.

Cullen Ritchie: We were able to get underway within just a few days.

News Anchor: Seemingly overnight, much of our daily lives have been put on hold. This morning, the list of closings and activities being suspended is growing from coast to coast. All Americans now trying to navigate what we're calling a "new normal"…

Brett Hallborg: On March 13th, 2020, we made the executive decision across the company to send all non-essential staff home.

Tony Martinez: And then of course, the lovely day, March 16th. All support personnel, managers, assistant managers, real time engineers, outage coordinators, everybody out of the control center.

Jason Bucholtz: The only staff that were actually in the control center were our operators and some facility folks, and then my leader who's the Director of Grid Market Operations, was in there as well to support the teams.

Cullen Ritchie: Myself and my manager in fact, we moved into a model of only one of us was allowed to be on site at a time and the other one would be working remotely. And in fact, during the initial response, we intentionally only had one of us here and kept the other person working from home. In case we did have an outbreak, we would have a supervisory person available who should be protected.

Chris Sakr: Now prior to COVID, the power industry was largely understood to be an onsite workforce with very few exceptions. Sending all non-essential staff home was unprecedented and extremely demanding.

Debra Smith: Employees were all shell shocked and they were just stunned as well, and they were suddenly dealing with their kids, and they didn't have technology at home, and you know all the stories. Everything that we've heard everywhere was true for us and, I'm sure, for every utility. It's like, "Oh, I don't have the technology and we're all sharing one laptop. And how do we get work done?" And then we started really focusing on, internally, how do we roll technology out to people? How do we deliver computers to people who need them? How do we get video cameras to people who are now all working remotely? So tons of super logistical work.

Jason Bucholtz: The organization actually did an amazing job of responding quickly, delivering computers and monitors to folks who didn't have them and didn't have that quick access, to get them basically to their home office, right? So amazing support. The IT team at ISO was actually awesome when it came to making sure your tools were working.

Chris Sakr: And while remote staff got set up, shop leaders continued doing their best to keep system operators who had to stay behind COVID-free.

Tony Martinez: Our leaders did a really good job just with that unknown. Tried to be as transparent as possible, again, taking guidelines from the CDC, from our local governments, our state, you name it, news media. We were trying to get as much information as we could. However, as an operator, unfortunately, you have to work long, ugly shifts. You have to work holidays, weekends, nights, you name it. So again, the operators are used to that. With this, we just wanted to make certain, "Okay, hey, is our food fresh? Let's overstock more than what we need." And of course, everybody remembers that was... We wanted to make sure that we had enough toilet paper to survive.

Brett Hallborg: You can challenge yourself and say, "Maybe we went a little too far," but I'll say this, we had no idea what was going to happen. We had no idea about the transmissibility of the virus. We had no idea how many people were going to be infected.

Chris Sakr: Through the great toilet paper rush of 2020, most utilities decided it was safe enough for their operators to travel between work and home and that sequestering them on site wasn't necessary. But BC and a handful of other North American utilities determined that wasn't a risk worth taking.

Brett Hallborg: We quickly shifted from hotels to RVs and now had to work through the logistics of sourcing the RVs, which the good news for us in the province of British Columbia is there's not a lot of heavy RV usage in the February/March timeline. But then we had to provision all the other services, so the black water, the gray water services for cleaning the vehicles out, hooking them up to the grid because we are not a campground. One of the reasons our shelter in place kind of ended a little bit earlier than normal, or we would've liked, is we did not have 30 amp or 40 amp services for these RVs to supply things like air conditioning. Based on what we were hearing from not only the World Health Organization, the Canadian Center for Disease Control, but also our local provincial health officers, the incubation period and other things and the transmissibility of the virus, we were planning anywhere in the area of 10 to 14 days of saying, "We need staff to be at work. We need a core crew of people we'll refer to as our skeleton crew, whose sole purpose is reliability." Again, with most of those field crews working from home, we would have a little bit of capacity to take on that, but it was only for emergency response. So the skeleton crew would've been fine to handle that, and this is not dissimilar to how we staff ourselves at nights.

Chris Sakr: Adapting from existing BCPs, they asked their unionized workforce for volunteers. Who was willing to join a rotating shelter in place A or B team working 12-hour shifts?

Brett Hallborg: And the only cross-pollination of an A to a B team was for shift change, and that was done respecting all our physical distancing and other protocols like mask guidance. Some people worked 10 straight days, some people worked 10 straight nights, other people, and we utilized our other control center for this flexibility, worked a five-day and then a 24-hour period off and then a five-night rotation in that 10 days. I think it was five days and four nights, but the idea here was some roles, we didn't want them working 10 straight nights. We have onsite laundry facilities. We brought in three meal a day catering plus snacks and other sundries that we needed to support that. So they worked their shift and then go home and sleep in the RV and take advantage of some of the other things we have in our facility, such as our gyms and our walking track that circles the facility and other things to keep themselves busy. And at the end of 10 days, we swapped out one rotation for a fresh batch of people.

Chris Sakr: BC's shop was up and running with skeleton crews, ready to run the grid. And other non-sequestered shops also had to consider coverage, physical arrangements, and recommended health precautions.

Cullen Ritchie: We worked very closely with our emergency management group, and they helped us out, with also our human resources group's assistance, to evaluate our control room and determine what type of distancing and what barriers we could put in place so that there was no real danger of transmission.

Tony Martinez: We wanted to just limit exposure and limit everybody, wanted to make certain that the three operators that we have on shift all the time still were in one place, we didn't want to separate them.

Jason Bucholtz: We very quickly turned our DTS training room into a third control room that was used for all our night shifts. So we had operators separated in our main control room during day shift, and then a new crew on nights in a different third control room. The operators were isolated to their crews. There was no face-to-face turnovers, no cross-crew coverage. If coverage was needed, the operator that was coming to cover shift would go to our backup location to stay isolated from the crew. So, we were trying to do the best job we could to keep the operators basically isolated to their immediate crew.

Tony Martinez: Hand sanitizer, thermometers, making certain that nobody had symptoms. I don't think we had tests at the time. Are the operators socially distanced far enough? In our shop, they are a little over six feet away from each other, so they're not as crammed.

Jason Bucholtz: Both of our control rooms were given a deep clean after every shift. We quickly moved to masks were required to be worn whenever operators were in close distance to each other, and we did encourage good social distancing right off the bat.

Chris Sakr: Social distancing and masks were the two mainstay precautions the world took in the face of this virus. Distancing presented obvious challenges with fairly straightforward solutions, but masks, well, you probably remember what that was like at first.

Cullen Ritchie: Our control room operators who spend a lot of time on the phone and on the radio, it was very fatiguing for them to have to be having those communications with a mask on when they weren't used to wearing a mask. But then also muffled communication. We were able to notice really quickly that at the end of a shift, our employees just felt exhausted.

Chris Sakr: Whether it was masks or any other obstacle COVID presented, the interconnected grid depended on balancing safety with making employees' jobs as frictionless as possible. Cullen ramped up team check-ins, opening the floor to feedback as frequently as he could. That feedback was combined with public health guidance to make small tweaks along the way. Incremental changes that led to positive results became new pandemic protocols.

Cullen Ritchie: Getting a solid, comprehensive plan that we could take forward, not just for this particular COVID situation but 10 years down the road if we saw something similar, that took about six months. There was a lot of trust and a lot of delegation of authority coming from our senior management down to the managers and supervisors at my level to be able to react more quickly and implement safe planning. We always looked at what was coming out from different agencies as a minimum guidance, a minimum level of guidance for employees' safety.

Chris Sakr: That trust and the agility that came from it would become finely tuned muscles for Tacoma's and everyone else's shops; it would serve them well down the line. Between January and April of 2020, for the first time in my life, the whole world seemed to grind to a halt. Days felt like weeks, weeks like months, and months felt like centuries. Everything was foreign. By April 16th, Seattle had gone from the pandemic's epicenter to some semblance of stability.

News Anchor: ... Crossover now to Seattle, Washington, bring in Erin McLaughlin. Where what a return to normal may look like if in fact things do get underway there. Erin, good morning, give us the perspective of how things might be in a place that has at least believed to have turned the corner. Good morning, Eamon. Well, there's some good news here…

Chris Sakr: However silly this seems now, a lot of us felt like an end was in sight. In those uncertain early months from boardrooms to breaker units, the North American power industry, one that prides itself on models, plans, and consistency, set aside business as usual to keep us from the brink. Electricity isn't just some jobs, and it's certainly not just a bill we pay to keep lights on, make coffee, or watch Netflix, it's fire stations, food storage, school, work, federal buildings, birthing units, and funeral homes. It's how we say hello and goodbye.

Debra Smith: At the time when she passed, it's like, "I'll deal with this later." Well, two years later, I never have. And I think almost without exception, my colleagues have gone through similar experiences, of one sort or another, that affected them that way.

Chris Sakr: No matter the circumstances, saying goodbye will always be a long, painful process. But with the global volume turned up, staring down the barrel of an even less certain future, Debra and so many others had to set that aside too. Because as we all now know all too well, this was just the beginning. Next time on Pandemic Response Part Two, The Lights Stayed On.

Brett Hallborg: One of the biggest things changing throughout the life of the pandemic has been our provincial health orders.

Cullen Ritchie: And almost immediately, I had more than one person out with COVID exposure.

Debra Smith: I can tell when I've landed badly on someone, and my goal is always to fix it as soon as possible. Completely missed it.

Tony Martinez: It'd be 7:00, 8:00 at night, and I'm still watching the system with the guys and still paying attention to everything instead of unplugging.

Phil Mason: COVID messed things up pretty quickly for us. We had a lot of continuing education hours to make up and plan for, and we had no idea how we were going to do that because we didn't have an infrastructure in place.

Chris Sakr: That's it for this episode of NERC Lessons Learned powered by Source.training and produced by Western Power Pool. If you're listening on iTunes, SoundCloud or somewhere else, be sure and visit WesternPowerPool.org for the full Lesson Learned text and a transcript of this episode; the link's in the show notes. If you like the show, share it with others in the industry who you think would feel the same way. Subscribe on your platform of choice and leave us a review and a rating if you have a second, we'd love your feedback. Thanks so much for listening. We hope you got a lot out of it and we'll see you next time.