PANDEMIC RESPONSE PART 2: LIGHTS STAYED ON
Below is the full transcript for this episode. If you'd like to review or follow along with the original .pdf version of this NERC Lesson Learned, visit:
https://www.nerc.com/pa/rrm/ea/Lessons%20Learned%20Document%20Library/LL20211001_Pandemic_Response.pdf
Chris Sakr: Welcome to part two in our three part series on pandemic response. Now, if you missed part one, that's okay. You should be able to follow along just fine. Feel free to check it out before or after this. And if you haven't heard, it took us from January to April, 2020, where utilities put their pandemic plans to work, and it seemed like the world might start turning normally again, but instead it just turned differently. We conducted interviews for this series in March, 2022, and one person not featured in part one was Phil Mason, AESO's manager of training and compliance delivery.
Phil Mason: You could characterize this pandemic in a lot of ways, but this was a complex, and still continues to be a very complex, event or situation we find ourselves in. And several years ago, I heard somebody say that the remedy for complexity is communication.
Chris Sakr: On this episode, we pick up where we left off, April 2020. Pressing to the end of 2021, we'll see what was happening in the field and back on site, where those in charge of staff safety managed ever evolving health guidance. But I think we can all agree the real tectonic shift COVID brought on didn't necessarily happen outdoors, or even inside facilities. By late April, 2020, poll showed 70% of us workers had gone remote at least some of the time, and that massive thrust into the future far sooner than anybody expected challenged, collaboration, security, employee training, and it even further complexified one of the hardest tasks in any job. Here's Debra Smith, CEO of Seattle City Light.
Debra Smith: I generally am someone who can read the room. I can read body language. I can tell when I've landed badly on someone, and my goal is always to fix it as soon as possible. Completely missed it.
Chris Sakr: The pandemic changed how we communicate, yanking back the curtain on a new way of working for the better, and in some cases, for the worst. But through trial, error, a lot of learning, and a few existing mechanisms, the power industry, again, took challenges head on and settled in for a very long ride. I'm your host, Chris Sakr, and this is Pandemic Response Part Two: Lights Stayed On. The power grids got seemingly endless moving parts. They don't really care if there's a pandemic. When they work, they work. And when they don't, the consequences can be pretty dire. Here's Tony Martinez, [inaudible] training coordinator,
Tony Martinez: Unfortunately, breakers still malfunction, breakers still have issues that they have to respond to. Business as usual, they would go out. But now instead of two substation electricians in one truck, they're taking two.
Chris Sakr: Cullen Richie, Tacoma Power system operations supervisor.
Cullen Ritchie: We went to what we called a crew intact structure. You would establish a crew and those crew members, and you didn't move those members between crews and make up different crews as you needed
Chris Sakr: BC Hydro's senior system control manager, Brett Hallborg.
Brett Hallborg: If you had a headquarter, for example, that had 20 lineman in it, or something along that line, you would work in a pod of no more than three or four. And so you would have multiple pods, and there would be no cross pollination of pods, unless there was a timeout period for the incubation of the pandemic.
Cullen Ritchie: We looked across our topology at our different facilities, our different substations, our service centers, and we established new home bases for a lot of those crews so that they weren't coming onto the campus every morning and every evening. They would only come here to resupply.
Tony Martinez: Once they get to the substation, of course, they got to work a little closer, hand in hand, but again, hopefully, proper PPE, masks, everything like that.
Chris Sakr: Back in control rooms, managers had to tackle the endless multi-layered process of adjusting safety protocols based on emerging science and government health orders, which was sometimes a one step forward, two steps back sort of situation.
Brett Hallborg: The provincial health officer would roll out a new set of regulations/protocols on a quarterly, or if not shorter than that, basis, all the way through the pandemic. So, this is stuff that we are regulated to follow as a business operating in British Columbia.
Cullen Ritchie: We had some of the different people that had the knowledge, so to speak, health department staff, the HR staff, because I'm not a doctor, I'm not a medical expert, I'm not an infectious disease expert by any means. They are. So, leveraging those people to help us answer those questions, really understanding not only what we have available, but what we needed to do, they were invaluable.
Brett Hallborg: We then had what we refer to as our COVID executive team who took the provincial health orders, pushed that down through certain aspects of the business, specifically very key players, such as myself, where we were able to comment on what that meant for us, whether we could follow it, how we would follow it, or in the case of my control centers, where we would go over and above any of those protocols. So, every time a change came down, we'd reestablish a new set of corporate protocols, and those corporate protocols could be also enhanced with some local protocols. So, my pandemic response plan would be lined up with both the provincial health order, the corporate plan, but then would have aspects of it that I thought were important. For example, if one of them, and this is just a hypothetical, came down and said, "Oh, you don't have to wear a mask anymore," we might say, "No, we're still going to have masks wearing in our control centers." Those conditions were ever changing, and we just had to adapt with ever changing protocols.
Cullen Ritchie: There was a period of time in 2020, where we started seeing some relaxation of some of the requirements, and almost immediately I had more than one person out with COVID exposure. So, we really, we tightened back up to our previous posturing a little bit, and that really kind of drove home the fact that while there is a lot of science, a lot of fact, and a lot of planning that goes around it, this was a very dynamic situation. We had to just continue to be diligent and maintain safety.
Brett Hallborg: By the summer of 2020, we were much back into, not normal by any stretch of the imagination, but back into business as usual, at least from a control center field worker plan, build, operate type model.
Chris Sakr: This was the foreseeable future. Companies had to find their own equilibrium when it came to following guidelines, and not all employees had the same risk factors.
Debra Smith: Even after the very, very initial stay home, stay healthy term ended, what we did is, we had a thing that said all you had to do as a city employee was to say that I'm high risk. And there was no proof required of that. It was just, "Hey, I am high risk." And it kind of keyed off that the CDCs definition of high risk. So, it was kind of anybody that was, I think it was 62 and older, so you had a lot of our senior folks. And they were the people that had the most experience. It was crew chiefs, a lot of folks like that who self-identified identified as high risk and stayed home.
Tony Martinez: We were lucky enough not to have any operators in that category, but you have operators that have children or a spouse that is a little bit more in jeopardy, if you will. So again, we had a couple operators that were like, "Well, hey, I don't want to cover my shift, and then bring something home to my son or daughter or to my wife that could be compromised." So, we made certain, "Hey, okay, well, what can we do?" The control center is off limits, just limited to the operators, and that was it. I think everybody's fears were calmed a little bit more, but we definitely had to address that with two of our operators.
Chris Sakr: Meanwhile, everyone from onsite staff to those working from home were learning how to collaborate remotely. Even with technology like Microsoft Teams in hand, it proved challenging.
Debra Smith: I'm a finance person, so I can still build a reasonably okay spreadsheet, but Teams is different because in order to learn new things about how to use it, I have to be on a call with people. And none of us have time to sit and watch Debra learn how to use Teams, and none of us want to watch Deborah learn how to use Teams. So, guess what Deborah has never learned? I have a CFO who is quite a bit younger than me, and she is the one who knows how to use use Teams. The rest of us often will be like, "Well, where's that file? Where's that file?" And she's like, "It's in the chat," or "It's in the channel." And I'm like, "What's the channel? I don't understand."
Brett Hallborg: Even for people who are probably very adept at doing this, when it's one or two people, it's probably fairly straightforward. When it's a group of 10 or 15 people, it's difficult to control the conversation. It's difficult to collaborate properties. And we're probably lucky that things like Microsoft Teams and WebEx were there and ready to go for us.
Phil Mason: If you wanted to ask somebody a question, you have to make a call in Teams or pick up your phone. And if you're going to do that, once you're in Teams or whatever environment you're working in, what's their status? Are they free? Are they in a meeting? Okay, they're in a meeting. I guess I'll plan something for later on this afternoon.
Chris Sakr: Aside from the tech learning curve and time management, this way of working raised another red flag. With new outside software suddenly in common use, one of the world's most understandably guarded industries had new security concerns.
News Anchor: It is the hottest way to teach a class right now, but tonight the FBI is sending out a warning about Zoom, specifically about trolls hijacking online Zoom classes. Seven Eyewitness News is helping you protect your-
Phil Mason: We talked to our communications folks, our IT folks, and we came up with a plan in terms of who we were going to allow access to the room. And we had pretty much a zero tolerance kind of policy. And unfortunately, in the early days, as everybody was learning how this was going to work, one of our ISO system controllers connected to one of our classes using a family member's PC that we could not identify, so we kicked them out of the meeting. And the way the software works is they can't get back at all.
Chris Sakr: Zoom hacks were folded in with too much bigger problems facing Phil and his training team. First, AESO was scheduled to host annual in person power system restoration training for operators from the entire province at the University of Calgary in May of 2020. And second to plan against an outbreak, AESO had turned their training room into a backup control center.
Phil Mason: COVID messed things up pretty quickly for us. We had a lot of continuing education hours to make up and plan for, and we had no idea how we were going to do that because we didn't have an infrastructure in place. We quickly scrambled to get a plan together that allowed us to do a couple of things, one of which was deliver restoration training to operators in the province before the end of the year. And the other thing, which was much more important and affected all of our training, was determining how we were going to have synchronous training with our own operators, plus the transmission operators, in a remote world,. We had April and May to kind of figure things out.
Chris Sakr: One early consideration was how to monitor trainees' realtime activities or questions through software like Zoom, Teams, or Adobe Connect.
Phil Mason: It's not as easy as getting up and walking over to a small group sitting at a desk or a table. We are now going to go into a virtual breakout room and listen in on them, but we can't see what they're doing and saying, unless they have the ability to share their screen. And with that, there are limitations. If you're sitting at an EMS console in a classroom, you might have three monitors. When you go to share your screen in a virtual environment, which monitor are you sharing? All of these little things add up and they made the first few attempts at remote training a little bit of a challenge. We made plans to offer some training in a very limited format so that we could test out some of the equipment and software we were planning to use to go forward. That initial round of training in June of 2020 included six NERC continuing education hours. It was all delivered synchronously using Adobe Connect and went off kind of without a hitch.
Chris Sakr: And by that, he means it worked, but they found a few more items to address. There was internet service, participants didn't all have the same connection quality. Well, not a lot AESO could do about that, but they also saw a need to tighten up instructions, like how to mute and unmute, starting and stopping video and document sharing. Basically, they needed training for the training. The more detailed, the better. In fall of 2020, they honed in even further after performing a small version of restoration training remotely through Zoom. And what came of that? Well, first, reenvisioning how to authentically replicate communication between control rooms.
Phil Mason: So, the idea was to mimic exactly the environment that would occur in our real time event, where everybody has a phone on their desk and a console on their desk, and they can phone whoever they need to phone whenever they need to phone them. If an AESO system controller wanted to talk to a transmission operator from a different entity, they could make a person to person phone call that only they would be a part of. And at the same time they were doing that, there could be other phone calls going on.
Chris Sakr: The temporary fix they landed on were those virtual breakout rooms where trainees could branch off into their own separate private calls. They're still looking for a better long term solution. And with these breakout rooms came new issues with monitoring trainee progress and receiving questions.
Phil Mason: If you've got a group of five in a breakout room and they have a question for the instructor, how do they post that? Will the instructor see it if they're not in the breakout room? We started putting all its check questions and survey questions into our classes. And the goal was always to do that once every four or five minutes. In practice, we didn't actually achieve that, and we're still not achieving it because it's hard to go back to the learners that frequently, but it does give you a gauge on who's paying attention. So, there's always those tweaks going on.
Chris Sakr: With tweak after tweak and test after test, Phil's team found a training system that worked, but the key is they didn't wait. They made moves.
Phil Mason: We do not sit back and decide to do nothing. We're always learning from what we see, what feedback we get. We adapt something, and then we try something new. We now know that Zoom can work using breakout rooms, and we now know Zoom can be used to share screens as part of a simulation, and we also know that teams can do the same thing. But in terms of achieving some level of comfort, I would say it was last fall, because that's when we delivered eventual restoration training in Zoom for the province, with everybody connected to our simulation environment. And that was the last big hurdle.
Chris Sakr: By fall of 2021. They'd crack the code. But as anyone in this industry can attest, training comes in different forms with different applications, from regular protocol updates to new initiatives and NERC continuing education. And most of this needed an overhaul.
Tony Martinez: I established a daily call on Tuesday and a daily call on Thursday. Both crews, so six operators would be on. Our outage coordinators, some of our realtime planning engineers, they'd be on as well. We would also include some of our sub entities as well, because again, we have a few other transmission operators and everything within our BA that work hand in hand with us. And so again, we want to make certain that they were kept up to speed and in the loop, if you will. So, we include them. And what we would do is just, again, discuss anything that's going on in the control center. We'd share it. Of course, one crew that was on during their relief week, the previous week, they were on night shifts. So, anything that they had endured on those night shifts the week before, they can share with the group. And again, it gave us an opportunity for the management to give any kind of updates to the operators as well. If it wasn't something that was already pre-recorded, it was something that maybe I had put together, a counterpart with another op-co that I've got.
Brett Hallborg: The immediate need stuff and the future need stuff was all handled basically through vendor provided virtual training. We did that both for the workers who are working shelter in place, because they weren't as busy as they would normally be. All the other people who were not part of the shelter in place rotation now had ample opportunity to do that virtual training from home. So, we took advantage of as much of the virtual training as we could.
Cullen Ritchie: For several years prior to COVID, we had already started leveraging remote learning and computer based training. We were not using it exclusively by any means, but we already had a lot of the infrastructure in place, anything from teams meetings to computer based trainings. What I really loved is that we were able to take those tools that we were using for system operations and expand them out to the high entire utility. And I don't think we would've been able to get there. Had we not had those virtual tools in place.
Brett Hallborg: And most specifically, the Western power pool training.
Tony Martinez: We took full advantage of the source.training platform.
Chris Sakr: That's right. We at Western Power Pool develop and deliver industry leading training for system operators through our online platform source.training. In fact, that's the biggest part of your humble host's job. We regularly upload new original content and source also allows members to securely host their own utility specific training.
Cullen Ritchie: You could look at the number of hours we delivered in 2019 versus the hours we delivered in 2020 and 2021 and just see the exponential increase of the number of hours we delivered on that platform. And that was just to get some essential things, some essential safety trainings, and our switching and tagging procedures training just deployed. But that was something I was kind of proud of, because luck, skill, whatever you want to call it, since we already had those pieces and that knowledge base in place, it was just simply a matter of expanding it and making it available to other teams. And so I feel like that piece was very successful for us.
Chris Sakr: In our first year 2014, source delivered 543 NERC continuing education hours. And by 2019, we were delivering 7,739 per year. But in 2020, that number skyrocketed to 17,624. Needless to say, we're all super proud. But even with our tool or thoughtful substitutes, like AESO's virtual simulations, for certain material, it can be nearly impossible to viably replace in-person training.
Brett Hallborg: We still take advantage of a vendor provided training, but once we started to get familiar with the pandemic, we did return very quickly to in-person or virtual training being developed and delivered by our own staff, because we are a NERC approved continuing education provider. So, we got back into that into the fall of 2020, at least virtually where we needed to, or in attendance with limited occupancy for in-person training.
Chris Sakr: And even with the slew of remote tools available throughout the pandemic, lots of folks feel it's nearly impossible to replace the in-person experience at all. And like with COVID risk factors, no two people are the same.
Cullen Ritchie: We had some staff members who just blossomed working remotely. They were able to work very well, their quality of life seemed to improve. For example, one of our employees, it was able to alleviate some childcare challenges, where in other cases, I've got some people we've had some people that aren't doing well, or didn't do well, who were feeling very negatively impacted, very cut off, if you will.
Tony Martinez: I found myself, when I was working from home, that it was very hard to unplug. It'd be seven, eight o'clock at night, and I'm still watching the system with the guys and still paying attention to everything instead of unplugging.
Chris Sakr: Jason Bucholtz, a real time operations manager at AESO.
Jason Bucholtz: The days actually became longer, because I wasn't willing to shut it off, where I normally would just leave work, until about June. And then I kind of decided that I was going to go back into the office, and I've been mostly in the office since June of 2021. I found that personally more rewarding, just being able to kind of shut the day down and maybe think about how things went on the way home type thing and be able to kind of get home and be, okay, this is family time, and just really disconnect. The calendar used to never be as full as it is now, because every single person that has a question for you has got a 30 minute meeting booked, right? You're online all the time, and I certainly don't like working in this environment one bit.
Chris Sakr: The psychological aspects may be hard to quantify, but they're there, and there's no silver bullet. We face each other in new ways now. We've all had to adjust, maybe incrementally, or maybe we learned the hard way.
Debra Smith: A colleague of mine at the city, who I thought I had a good relationship with, there was no body language, there was no anything, and we were having conversations. And we weren't even using Zoom or Teams in the same way we do today. I've learned now to do a Teams call so that at least I'm looking at you. But back then, we're on the phone... Bottom line was I really hurt this person, or if I didn't hurt them, my actions contributed to what was already a very deep and an understandable hurt. And I like to think of myself as someone... I generally am someone who can read the room. I can read body language. I can tell when I've landed badly on someone, and my goal is always to fix it as soon as possible. Completely missed it, and it changed the relationship. And that was, I think, the moment when I realized that, oh my goodness, we've got to do things really differently.
Chris Sakr: Every company and individual had to meet this new world their own way in their own time. But whether in the field or at a desk, in Zoom or through a mask, from health orders to collaboration, time management, security, or training, for the lights to stay on, work still had to get done. While some things had to change, they also had to stay the same, because until they're over, dynamic situations stay dynamic. And if we learned anything from the beginning of the pandemic or these touch and go past few years, just when you thought COVID was letting go, it pulled you back in.
Brett Hallborg: In the fall of 2021, we were very much of the mindset that we'd be going back to our new sense of normal. And then that's kind of when we hit the hard U-turn, is when the omicron variant showed up.