PREVENTING EEAS
Below is the full transcript for this episode. If you'd like to review or follow along with the original .pdf version of this NERC Lesson Learned, visit: https://www.nerc.com/pa/rrm/ea/Lessons%20Learned%20Document%20Library/LL20200602_Preventing_EEAs.pdf
EPISODE TRANSCRIPT
Chris Sakr: My Uncle Kenny has hobbies. A retired surgeon, he drives fast, fishes, hunts, tends his property, works on cars, like his World War II Combat Jeep equipped with a plugged 50 caliber machine gun, which he drove with 10 year old me through the McDonald's drive-thru. If he hears a turn of phrase that jumps out at him, he wants to know where it came from. Like this quote from a 1987 issue of Musician Magazine, "He told me he couldn't play reggae. Of course he could, but it wasn't in his wheelhouse."
Before that, wheelhouse had described the striking zone for a baseball in a 1959 issue of the San Francisco Chronicle, borrowed from the airline pilots' cockpit before that, lifted from the ship captain's steering quarters, all the way back to 1808, when it was the room on a farm where wagon wheels were stored. In the September 1944 issue of Plain Talk Magazine, the word checkout was first printed to mean check-out: appraise, investigate, look closer. Which is exactly what we're going to do with this NERC lesson learned. Preventing Energy Emergency Alerts, published June 11th, 2020. As I reviewed the document, a phrase kept popping into my mind. As you may have figured out by now, researching figures of speech, that's my wheelhouse.
It seems the first use of the word pickle relevant to this discussion appeared in the diary of English Parliamentarian, Samuel Pepys, Wednesday, December 16th, 1660. "At home with the workmen all afternoon, our house being in the most sad pickle." In a pickle, meaning something is very seriously wrong.
An energy emergency alert is a pickle. We cover EEAs in greater detail in our qualifying events course on source.training, so check that out too. EOP-011 basically defines them as emergency procedures initiated by an RC. How big of a pickle depends on which of the three EEA levels you find yourself in. NERC defines an EEA1, as all available generation resource in use. An EEA2, load management procedures are in effect, and an EEA3, firm load interruption is imminent or in progress, or that you don't have adequate contingency reserves.
As Scottish poet Robert Burns said in 1786, "The best laid plans of mice and men often go awry." Translation, plan all you want, you'll still end up in a pickle.
Tim Reynolds: I'm Tim Reynolds, I'm the manager of the event analysis and situational awareness department here at WECC. As mentioned in the title, part of our primary role for our department is to be aware about events that are happening on the interconnection and perform analysis on them, and to participate where we can within the NERC event analysis process.
CS: Tim and WECC generously helped with this episode, which makes us lucky. Tim co-authored the document. The problem statement goes as follows, "As several energy emergency alerts (EEAs) were issued over the course of several months, key items were identified that could have prevented the EEAs from being issued, and in some cases prevented the BAs from unnecessarily shedding firm load to maintain system reliability."
TR: There's a deficiency that's happening. There's something that's going on where a BA cannot meet their obligations. So if we're continuing to see higher and higher numbers of EEAs that are happening within the interconnection, it can be pointing to a potential deficiency that is happening within the interconnection.
CS: Going back to 2017, 11 EEAs were issued in the Western interconnection. 2018, 21. By 2019, the year this document pulls from, a record 33 EEAs occurred. In the document, Tim focused on five, and for this show we honed in on three. The document's linked on the episodes page at nwpp.org, and you should definitely take a further look. So without further ado... What hold up, hold up, wait a second. That's one. We need more ado for a second. Hold up. So, Norse invaders to Northern England used the word "at" the way we use "to". So when they said, "At do," like, "To do," it sounded to English ears like "ado", which means fuss.
Okay. So anyway, without further ado, case one. It happened in a different reserve sharing group than the Northwest Power Pools, beginning with two BAs in a heat wave BA one didn't forecast. Out of nowhere, two of their generating units trip offline. With them went generation to meet real-time demand and reserves. Definitely a pickle. So to make up the necessary reserves for their reserve sharing group, BAs one and two, reach out to the RC requesting to be put into an EEA state. Then the fun starts.
TR: There was an error within the program that the Reserve Sharing Group uses, that was unknown at the time, where if one BA submitted information while another BA was in the process of doing the exact same thing, that other BA, unless they closed everything out and got back into it to submit their request, would never acknowledge anything that they did to help that first BA that submitted anything.
CS: Because of the error, BA two provides more assistance than needed and the pickle sours.
TR: Unfortunately for them, a second issue happened where they weren't able to provide additional reserves because they wouldn't start up. Some sort of error was happening and the generators couldn't start up. So they were short about an additional 60 megawatts that was no longer available for them.
CS: Another error. So BA two has bad news for the RC. "We're shedding 150 megawatts of load. We need to meet our disturbance recovery per BAL-002 within the required 15 minute timeframe. Between providing too much support to BA one and their offline reserves, they weren't going to make it. Unfortunately...
TR: There are certain thresholds to make a balancing authority and reserve sharing group have to recover within that 15 minute timeframe. In this particular instance though, the two units that tripped actually were further apart where it didn't qualify for it. Unfortunately it was one minute and six seconds apart. And the qualification is the two units would have to trip within one minute. So the operator not knowing that, still did what he thought was right and shed the 150 megawatts of load.
CS: Operators may be prepared 350 days out of the year. Problems pop up, but they're mostly predictable. The other 15 days? Well, that's plenty of time for things you've never seen to completely upend your expectations. On November 6th, 1914, Morton Salt debuted their new slogan, a phrase you probably know well, especially if you live in the Northwest: "When it rains, it pours." Many of these issues arise when no one's expecting two sets of problems at once. So says the old familiar voice to any listener of this show, Power Pool's own Greg Park.
Greg Park: When things go sideways for an operator, they go sideways very quickly. The requirements are normally looking at a single contingency. And then the expectation is that situations returned to normal. But a lot of times what we see is the non-normal, the non-single contingency things happen. You start with a highly stressed system and maybe you lose two plants, one right after the other. And then all of a sudden, you've never planned for that. And so I think there's going to be a greater emphasis on doing better planning for our operators. Having a better game plan for game day than we have today.
CS: There's the game plan for non-normal moments, but this case also shows the importance of really knowing what you need to recover in 15 minutes, and how membership in reserve sharing group may change that. Case three, which did occur in the power pool RSG, ends in a very similar place. But first it begins with a wildfire getting, as Sammy Davis would put it circa 1956, "too close for comfort."
TR: In this case, it was a WECC path, a very important one. And as the fire got closer to it, they had to de-rate the line just in case the fire was going to happen to trip it out. This caused other BA's that were part of the reserve sharing group to be a little more isolated from the overall reserve sharing group, which also increased how much contingency reserve they had to carry.
CS: One BA in the group determined they were too short and asked for an EEA3, but according to later RSG analysis, they'd made a crucial miscalculation.
TR: The RSG treats certain areas as kind of sub-regions within their overall footprint. So this sub-region as a whole was actually still complete and still had all the reserves they needed to. So if this particular BA looked at what they needed as a reserve sharing group as a whole, they would have had everything they needed.
CS: So, there are two clear issues here. To address the first, we point to the 16th century English proverb by John Heywood, "Ye cannot see the wood for the trees." Or as we know it "seeing the forest through the trees." Meaning when you're inundated with the details, it's hard to see the whole problem.
TR: Also, look for ways to improve the operator screen, making sure that the information that is being procured is making its way into that, but also simplifying some of the screens to be able to show, especially if you're part of an RSG, having that there in one screen, to help you to identify different things in it, really helps out.
CS: All operators can make screen suggestions. They're weighed and voted on. Your shop may be different, but if you see a potential gap or even too much confusing information, make it known, prevent larger problems when you can't afford them. But then there's the second issue. Had they understood what was available to them as group members, they would have avoided the EEA together. Now finally, case five differs in some particulars, but the bottom line, it's about the same.
TR: There was a BA that was a member of an RSG that had a generating unit that tripped. As they asked for assistance from the rest of the RSG, the members provided assistance. However, this particular BA was unable to meet their share of what was obligated for them. When the operator was looking at this and trying to get things online to help to recover and provide what their obligation was on that, they were looking towards BAL-002 for the DCS recovery. However, it was getting close to that 15 minute mark. And so they decided to shed about a hundred megawatts of load, contacted the RC. They were then placed into an EEA3, so they can be able to shed the appropriate load.
According to BAL-002 there's certain qualifications that need to happen. As I mentioned before, there's one that has to deal with a one minute timeframe with how much you lose. And then part of it is too, is how much you lose within that one minute timeframe. When this generator tripped, the amount that went offline did not meet those thresholds as far as the total amount that was lost within a minute timeframe. And so it was not a qualifying event.
CS: The phrase "bottom line" seems to go all the way back to accounting in 1832. And as we account for these three cases, the bottom line should seem pretty clear.
TR: To understand how BAL-002 applies to you as a member of an RSG or as a BA only. Kind of learning when it is, when you're in a part of an RSG, how does that apply? What are you looking for? Your MSSC as a whole, with that reserve sharing group and the thresholds with it.
GP: We expect our operators to step back and understand when you read BAL-002, the responsible entity for you, if you're a reserve sharing group member is not on you as a BA, but your part of that RSGC. That's the fundamental that we've tried to express to our balancing authorities when we train them. And also the fundamental that, when we train the RC's, when it comes to contingency reserves, it's really not a reliability problem unless the RSG can't deliver it.
CS: Many operators need increasing training and awareness surrounding RSG rules to prevent EEAs, especially given the harsh reality we're all now pretty aware of.
TR: NERC puts out a state of reliability report every year, and in their 2020 state of the reliability report, it showed the Western Interconnection as a red. The only interconnection as a red for energy emergency alerts, meaning that there needs to be some actionable things that happen. So we had a record number in 2019, and 2020 we're actually on pace to possibly exceed it.
GP: Which is unprecedented in our industry for any time that we've been doing EEAs, which is since 2006 or 2007. So this isn't a problem that's getting less complicated. It's a problem that's getting probably more and more complicated. And honestly, with the changing resource mix that we're seeing, lots of retirements of a conventional generation and not just coal: California's retired tens of gigawatts of natural gas generation in their state. Anytime you change the resource mix as drastically and as quickly as you have in the Western Interconnection, it's going to bring problems around delivery of energy and probably our operators aren't as prepared as they need to be. And I'm saying this holistically, not looking at one single entity, but I think there might be some improvements to how we do business, to making sure we're reliable and we can meet the needs of our customers
CS: By August, 2020, 39 EEAs has occurred in WECC, eclipsing 2019. With 27 issued just between August 14th and 19th. "Full circle", as Shakespeare coined. In some ways, this increase is oddly a mark of the system's evolution. For his telling of the Spider-Man story, Stan Lee authored what's known as the Peter Parker principle, adapting a saying from the French national convention way back in 1793. Lee's version: "With great power comes great responsibility." As the system evolves, that power partially belongs to you, the operator, and there are tools within reach.
TR: If there's anything that I can impart the most, it's make sure that you are looking at the lessons learned documents, because the main purpose of the lessons learned documents is to prevent future reoccurrence. And so if there's any questions, anything that needs to be known about this, please reach out to the event analysis and situational department here at WECC. And as you can see, with all the lessons learned, there is contact information at the bottom to be able to get additional information if needed.
CS: Of course, we'll keep doing our part with these podcasts and other training, and there's no downside to reaching out to Tim or WECC with questions or concerns. So we again thank them for their help here and ongoing efforts. EEAs are here to stay. It could someday not long in the future, be your shop, on your screens, during your shift.
TR: And what was interesting during the 2020 heat wave that happened last month in August; it's public information: we know that CAISO, their BA shed load, but what others may not know is there was more than CAISO that was in an EEA state during that timeframe.
CS: Between 599 and 661AD, Iman Ali wrote the earliest known version of "knowledge is power." Keep that in mind at your desk. There will always be some new thing to learn and understand to prevent unforeseen emergencies. Change is inevitable. For those difficult days of growing pains, remember the wisest of phrases, for all anyone knows, first printed in the Sydney Morning Herald, May, 1847: "He again, begged it to be simply understood that he had stated he merely threw out a suggestion considering that it was always better to be safe than sorry."