Tuesday, October 30, 2012

Fission Stories #116: My 5 Scariest Nuclear Plant Moments #nukefreecal #nirs #nukenews

Fission Stories #116: My Five Scariest Nuclear Plant Moments:
During the Q&A following panel presentations during the Base-load Electricity from Natural Gas and Nuclear Power symposium at the Howard H. Baker Jr. Center for Public Policy last month, I described an incident at the Browns Ferry nuclear plant and mentioned it was the third scariest moment I had experienced during my career in nuclear plants. At the next break, someone asked about my scariest moments. Here are my five scariest moments:

No. 5 – Draining down the Unit 1 reactor vessel at Hatch

My first job out of college was as a Junior Engineer at the Hatch nuclear plant near Baxley, Georgia. The Unit 1 reactor was shut down for a refueling outage when I reported for work on June 18, 1979. Prior to restarting the reactor, many tests had to be successfully completed to confirm that emergency systems were operational. Not long after my arrival, my supervisor handed me a test procedure and told me to go out and run it.
The testing involved timing how long it took for motor-operated valves in safety systems to move from the fully closed to fully opened positions and vice-versa. The plant’s safety studies assumed that these valves opened within a specified time frame to admit makeup water to the reactor vessel or closed within so many seconds to prevent the loss of water inventory. Each valve had a switch in the main control room and two lights: only the green bulb was lit when the valve was fully closed and only the red light was lit when it was fully opened. My task was simple: ask the control room operator to turn the switch to cause a closed valve to travel to the opened position. When the red bulb lit up, I’d click a stopwatch. When the green bulb went out, I’d click the stopwatch and write down on the test procedure how long it took the valve to stroke open. This process was reversed to test a valve’s closing time. Several dozen valves were to be tested and the only real challenge was getting an operator – busy supporting many other ongoing tests – to manipulate the switches.
Things went smoothly until it came to testing the motor-operated valve circled in red. It was within the residual heat removal (RHR) system, a vital and multi-purpose emergency system that can be used to cool the reactor core, the suppression pool water, and the containment. The RHR system has two fully redundant sets of pumps, pipes, valves, and other equipment. One set (outlined in dark blue) was being used to cool the water inside the reactor vessel. I was testing a valve in the other set. This set was idle at the time.

Fig. 1
At my request, an operator flipped the switch to open the valve. I clicked my stopwatch and started intently at the valve’s lights. This size valve took nearly 30 seconds to travel to the fully opened position and I dared not miss its green bulb going out.
But as this valve stroked open, the activity level in the control room increased. Alarms sounded and warning lights flashed. I stole a quick glance at the digital readout on the main control panel showing the water level inside the reactor vessel. It had been +33 when I started the test. It was +25 and dropping rapidly. Seconds later, even more alarms and warning lights went off as the lowering water level triggered an automatic signal to shut down the reactor core.
The green bulb finally went out. I clicked the stopwatch, jotted down the elapsed time, and quickly moved to a remote part of the control room to get out of the way of the operators as they responded to the situation. The digital readout now read -8 and continued decreasing.
The Three Mile Island Unit 2 accident had occurred just a few weeks earlier. I knew it was caused when the water level in the reactor vessel dropped below the top of the reactor core. I had not yet learned how water level in the Hatch reactor vessel was measured, but assumed that negative levels were bad. So, the good news about the valve stroking closed to open within the specified time limit was tempered by the concern that the reactor core may have been uncovered and damaged along the way.
Things weren’t quite as bad as I’d imagined. Zero for the digital readout of reactor vessel water level was an arbitrary point nearly twenty feet above the top of the reactor core. Adequate core cooling had not been seriously challenged and the operators were able to quickly restore level to its normal range.
I had caused it. Earlier during the refueling outage, the suction piping for the RHR pumps in this idle set had been drained of water to allow maintenance to be performed. The maintenance had been completed but the piping not re-filled. At least not re-filled until the valve’s opening allowed water from the reactor vessel to drain into it. One inch of level in the reactor vessel corresponds to about 200 gallons and there was a long stretch of empty large-diameter piping to re-fill.
As the reactor vessel’s water level dropped about five inches below normal, alarms had sounded to warn operators about the problem. As the level dropped about two feet below normal, a protective system triggered the automatic shut down of the reactor core. All of the control rods were fully inserted at the time, but they received a signal to fully insert anyway.
The reactor core survived and the test was completed successfully. It turned out to be a great way for a new Junior Engineer to meet, and get yelled out, by most of the plant’s senior managers.

No. 4 – Assuming my way to unconsciousness at Hatch Unit 1

Due to my stellar performance testing valve stroke times, or perhaps because of a staffing shortage, I was then assigned to test the logic circuit for the core spray system, another emergency system for Unit 1 at Hatch. The core spray system is designed to provide large (more than 5,000 gallons per minute) makeup flow to the reactor vessel to mitigate the rupture of the largest pipe connected to it.
The logic test involved simulating all the conditions necessary for the core spray system to automatically start, but not actually resulting in the system’s pumps starting. The prerequisite conditions were simulated using electrical jumpers – insulated wires with alligator clips on either end. Installing the jumpers around relays in the control circuit would make the circuit “believe” that the necessary conditions were met. The jumpers are installed to connections along terminal strips inside control panels.

Fig. 2
The terminal strips inside control panels at Hatch were labeled with numbers. The test procedure directed me to install jumpers to connections by those numbers. The test progressed smoothly until I got to a control panel without labels on its terminal strips. All the panels I’d tested thus far looked the same with four vertical rows of terminal strips. So, I opened the adjacent control panel that had labeled terminal strips and counted how many connections down from the top was labeled with the number I needed for my jumper’s location. I fastened the alligator clip on one end of the jumper to a connection and squatted down to attach the alligator clip on the other end. I saw a bluish flash.

The next thing I saw was a circle of workers looking down at me like in the opening credits of “Eight is Enough,” a TV sitcom popular at the time. I was lying on the tiled floor of the control room. Instead of jumpering across low voltage direct current power of a logic circuit, I’d tried installing the jumper across a higher voltage alternating current power circuit.
Electrical current arced to the end of the alligator clip I was holding as it neared the high voltage connection, blackening the metal and melting enough that the clip is soldered closed. The other end of the alligator clip remains workable.

Fig. 4
Turns out that terminal strips in all control panels are not configured the same. That’s likely why layout drawings for every terminal strip in every control panel are developed. I learned about the existence of these layout drawings after my close call from some of the senior managers I got to meet (or re-meet) and listen to yell at me.
The yelling was pro-forma. Thinking I’d melted a reactor core and actually melting a clip I was holding was more than enough to focus my efforts on being way more careful in the future.

No. 3 – Losing control of Browns Ferry Unit 1

On December 6, 1982, I was the Shift Technical Advisor and the on-duty reactor engineer as the Unit 1 reactor was being restarted from a short maintenance outage. The reactor had been shut down two days earlier to fix a small leak from the piping near one of the three feedwater pumps.
I worked a 12-hour shift. The operators worked 8-hour shifts. With the end of their shift approaching, the crew of operators wanted to continue withdrawing control rods until the reactor core achieved criticality (i.e., a self-sustaining nuclear chain reaction) and turn over the unit to the oncoming shift at this low power configuration.
But as an operators withdrew control rod 18-51 six inches (the smallest movement interval allowable for the core’s 185 control rods), the reactor power level rose rapidly. Instead of waiting nearly ten seconds for the control rod to settle out, the operator used the emergency rod in switch to quickly return the control rod to its original position.
After a caucus, we opted for Plan B. We’d leave the control rods alone and allow the reactor water temperature to increase. As the water temperature rose, the reactor power increase resulting from control rod’s 18-51 withdrawal would be diminished.
The water temperature steadily increased as the three control room operators turned their duties over to three oncoming operators. The operator who had been handling control rod withdrawals briefed the operator who would be assuming this task. The operator who had been preparing to place the turbine/generator in service briefed his replacement. And the operator who had been placing auxiliary plant equipment in service during the startup explained where he’d gotten in those duties to his replacement.
I used the moment to catch up the reactor engineering logbook at our desk in the back of the control room. I noticed the operator who’d taken over the turbine/generator job move to the main control board where the control rod control switches were located, but assumed he was only monitoring the reactor core’s status. I was wrong. He withdrew control rod 18-51.

Fig. 5
The withdrawal of control rod 18-51 caused the reactor core’s power level to increase far more than it had earlier. Although the reactor water temperature had significantly increased since the first attempt, control rod 18-51 malfunctioned this time. Instead of being withdrawn six inches, the control rod double-notched and came out of the reactor core an entire foot. The reactor core achieved criticality and began doubling its power level very quickly. The preferred rate of power increase is doubling every 30 to 60 seconds. The actual rate doubled the power level about every five seconds. The soaring power level caused the protective system to automatically cause all the control rods to fully insert into the reactor core to terminate the nuclear chain reaction and shut down the reactor. Because of this automatic protective response, no nuclear fuel was damaged. But having a 100-ton reactor core double its power output every five seconds was extremely discomforting.

No. 2 – Damaging fuel rods on Browns Ferry Unit 1

On one midnight shift at Browns Ferry, I supported the efforts to bring the Unit 1 reactor to full power following its refueling outage. During the outage, about a quarter of the fuel assemblies had been offloaded from the reactor core into the spent fuel pool and replaced with fresh fuel assemblies. The reactor core’s power level was approximately 70% when I gave the operator a list of four control rods to withdraw a foot each. At this power level and configuration, this was a very conservative, vanilla move. It would not increase the reactor core’s power level much, but would re-distribute power within the reactor core. That at least was my plan.

Fig. 6
The operator had not finished moving all four control rods when many white, several red, and some purple alarm windows flashed. Browns Ferry’s control rooms featured alarm panels with white, red, and purple windows. White alarms alerted the operators to off-normal conditions. Red alarms informed the operators about off-normal conditions that had, or could, trigger the automatic shut down of the reactor. And purple alarms warned the operators about off-normal conditions involving high radiation levels.
The operator quickly re-inserted the four control rods to their original positions. It did not take long for us to determine that the alarms were caused by radioactivity most likely released from damaged fuel rods in the reactor core. In other words, one or more fuel rods in the vicinity of those four control rods had been damaged. Radioactivity escaped from the damaged fuel rods into the reactor water. In a boiling water reactor like Browns Ferry, the radioactivity was then carried away in steam to the main turbine. Much of the plant had been contaminated.
Because it happened early on midnight shift, I had several hours to pack up personal belongings from my desk to be ready when I got fired in the morning. But I got a reprieve – the order from the manager’s morning meeting was to send data off to the fuel vendor, General Electric, for analysis of what caused the damage. This provided enough time to update my resume (leaving off the breaking fuel experience) before the final verdict came back.
I got an unexpected response from GE. Their analysis concluded that the fuel rods had been damaged during manufacturing, most likely from a bad weld where the end caps were fastened to the fuel rods. GE reported that pre-existing holes or defects in the fuel rods provided a pathway for radioactivity to leak into the cooling water. When the four control rods were withdrawn to increase the power level in that region of the reactor core, the higher power level served as the catalyst to propel radioactivity out these pathways. GE indicated that if the damage had been caused by improper withdrawals (i.e., my mistake), it would have taken at least several hours for the excessive stress to damage the fuel rods and allow radioactivity to escape.
So, I had not damaged fuel rods with my control rod movements. Instead, those movements had only exposed pre-existing fuel rod failures. GE’s analysis took me off the hook.

No. 1 – Stranded inside the reactor vessel at Grand Gulf

In 1981, I was among the large contingent of General Electric workers assisting the initial startup of the Grand Gulf Nuclear Station near Port Gibson, Mississippi. Mike Haben, another GE worker, and I would often walk around the plant during our lunch hours. Because the plant had not yet operated, there were no radioactively contaminated areas. We could access equipment and rooms that were not readily accessible at an operating plant.

Fig. 7
During one lunch hour, we went to the refueling floor. We found it was possible to climb a ladder down into the reactor vessel and walk on plywood boards placed on the top guide. The reactor head was removed. The steam dryer and steam separator had not been installed. And nuclear fuel had not yet been loaded into the reactor core. We jumped at the opportunity to examine first-hand equipment we’d only seen in pictures and drawings.
For housekeeping reasons (i.e., to prevent stuff in our pockets or other debris from getting into the reactor vessel), we had to put on coveralls, gloves, and rubber shoe covers before going down the ladder. We then climbed down one ladder into the refueling cavity above the reactor vessel and a second ladder down onto top guide of the reactor vessel.
It was like a trip to nuclear nirvana for geeks like us. We saw things like the inside surfaces of the main steam nozzles, the top guide, and the core spray rings. Then we spotted the yellow brick road – a rope ladder between the core shroud and the reactor vessel between jet pumps. We could get to the domed lower portion of the reactor vessel below the core plate to look at the control rod guide tubes. I quickly got onto the rope ladder and started down.
I didn’t get very far. I had climbed down five or six feet until my shoulders were about level with the tops of the jet pumps (called the rams heads due to their shape) when I made an important albeit belated discovery – the rope ladder didn’t extend all the way to the bottom of the jet pumps. My foot searched for, but didn’t find, the next rung of the ladder.
I called up to Mike, who was peering over the side of the core shroud down to me, and explained the situation. He pointed out that there was a huge pile of rope ladder lying on the top guide that we could lower the rest of the way. He told me to climb back up.
I tried climbing back up. But because of my weight on the bottom of the ladder, climbing up was different than climbing down. The rungs of the rope ladder were pressed solidly against the outer surface of the core shroud. And the two inches or so of empty space in the toes of my rubber shoe covers prevented my feet from getting firm toeholds. I could not climb back up the ladder.
Panic, anxiety, impatience or some combination thereof set in. Images of being stuck on the end of that ladder inside the reactor vessel until I starved to death flashed through my mind. I yelled to Mike for him to pull me back up. To his credit, Mike tried to do so. But I outweighed him by at least 40 pounds. Mike looked back over the edge at me and informed me, “you’re screwed.”
I am not claustrophobic, but I felt a real urgency to get out of there. I pulled myself up far enough to sit on top of a jet pump and ponder my options. I could not reach the core shroud above me. I thought I might be able to reach it if I stood on the jet pump. But the rams head had a curved surface and my rubber shoe covers had already caused problems. The starving images were replaced by images of my slipping and falling down between the jet pumps, and then starving.
Finally, some sense returned. I got back onto the rope ladder. Using one foot to press against the core shroud and swing the rope ladder away from it, I was able to move a foot up a rung. Repeating that slow process, I rejoined Mike on the top guide.
We fed more rope ladder over the side and tied it off. Then Mike went down the ladder and through the access hole into the lower plenum of the reactor vessel. I followed him down. We saw components close up that made it well worth the effort.
I was first back up the rope ladder. I put a foot on the first rung and stepped down. My weight only moved the rung down against the metal surface of the lower reactor vessel dome. My next step brought the second rung down to the metal. I wasn’t climbing up the rope ladder, I was stretching the rope ladder down to me. Finally, the third step moved me up instead of the ladder down. We made it back to the refueling floor.

Our Takeaway

The common theme among my five scariest moments is that they were almost all self-inflicted. Training classes at the nuclear plants repeatedly told me to take personal responsibility for my safety. Maybe I should buy and wear one ear muff to keep such sage advice from just passing in one ear and out the other.
I was probably in greatest danger when I nearly electrocuted myself at Hatch. But the jolt of electricity was so fast that it gave me little time to panic or worry. The moment of helplessness on the short ladder inside the reactor vessel at Grand Gulf gave me ample time for fear to develop. And I wasted little of that time developing plenty of fear. Fortunately, the combination of not really being in any danger and having a short attention span got me through my scariest moment in a nuclear plant.

“Fission Stories” is a weekly feature by Dave Lochbaum. For more information on nuclear power safety, see the nuclear safety section of UCS’s website and our interactive map, the Nuclear Power Information Tracker.