For years, federal regulators urged state officials in charge of the Oroville Dam to consider installing cameras, lights and more sensors and monitors to help alert managers to potential structural problems and, in a crisis, ensure there would be time to evacuate towns downstream. But on Feb. 12, as murky floodwaters washed away the second of two failing spillways, nightfall left officials struggling to figure out what was happening.
"Can't confirm. Dark," wrote a state water agency engineer, who was serving as note-taker during an emergency meeting of top officials in the first hours after an evacuation order.
"Don't have eyes. Until we see it, assume it is a threat," said another unidentified official at the meeting, according to the 35 pages of handwritten and typed notes obtained by The Associated Press in response to a public records request. It was one of a series of calls and meetings that at various times included Gov. Jerry Brown, his Cabinet secretary and others struggling to assess the risk from the badly eroding emergency spillway.
The documents reveal for the first time the difficulty Oroville's managers had gauging the evolving dangers for 188,000 people who were ordered to evacuate below the huge dam: State water officials used drones and scrambled to borrow cameras and helicopter rides from other agencies to inspect their own dam and its spillways.
At least since 2011, the Federal Energy Regulatory Commission, which oversees hydropower dams, had expressed concern that Oroville Dam and adjoining structures along Northern California's Feather River needed cameras and other monitors to spot dangerous structural failures, according to documents obtained by AP.
In 2011, federal regulators noted communities immediately downstream likely would not have time to evacuate if the dam abruptly failed. They urged California's Department of Water Resources to install more technology to detect signs of sudden structural breakdowns, including remote cameras, nighttime lighting, and reservoir elevation sensors.
During the February crisis, water managers borrowed a camera from the state Transportation Department to provide a live video feed and tried to find out during the first hours after the evacuation of that camera operated in the dark so they could see the disintegrating emergency spillway.
They borrowed 10 cameras in all, training most on the other, main spillway. They pointed one toward the emergency spillway, but it was too far away to capture the portion that was falling apart.
"Even with all of this technology in place, continuous monitoring at night was difficult as the drones were not equipped with infrared cameras and the helicopters are limited on the number of hours they can be in the air," water agency spokeswoman Erin Mellon said in an email.
The federal energy commission, which is investigating the near-catastrophe, will probe the issue of cameras and other structural monitors, spokeswoman Celeste Miller told AP.
The state water agency, which oversees the dam, rejected installing cameras, deeming them impractical because of the size of the dam complex, which includes 6 miles (10 kilometers) of the embankment and the 770-foot-tall (235-meter) dam, Mellon, the agency spokeswoman, said.
Water agency officials told federal regulators they would use roving patrols instead, although regulators noted it could take patrols 20 minutes or more to reach and verify a trouble spot. The agency also later installed sensors to detect if water started dropping in California's second-largest reservoir, signaling a breach.
In annual reports since 2013, meanwhile, federal and state dam inspectors noted only three of the 56 devices installed inside the dam when it was built in the 1960s to detect leaks and other problems still worked. Annually, inspectors told California to devise a long-term plan to improve structural monitoring at Oroville Dam. Each year, the state water agency told regulators its engineers were studying that.
The crisis at Lake Oroville, a main water source for the 39 million Californians, began Feb. 7 when the dam's main spillway began breaking apart. Fearing the complete loss of that spillway if they released large amounts of water, dam managers allowed winter runoff to fill the reservoir to the top.
By Feb. 12, the reservoir began flowing down Oroville's never-used second, emergency spillway, which began collapsing. The near-disaster has cost a half-billion dollars so far and prompted more than $1 billion more in claims from people downstream.
The notes provided by the state give only a rough account of a series of meetings and phone calls with senior officials over the first days of the crisis. The remarks, jotted down by Alyssa Stutz, a water agency engineer, are frequently in incomplete sentences - sometimes single words - and often do not make clear who was speaking.
Brown's office and the state water agency declined to make the principals in the meetings available for interviews, or identify some of the participants. Mellon also refused to talk directly to reporters, responding only to emailed questions over several weeks.
"A set of handwritten notes does not accurately capture the whole process or comprehensive emergency response," Mellon wrote.
Still, the notes reveal the uncertainty faced by top officials as they struggled to assess the threat from the spillway.
"It's an ugly, sh---y mess, and we are trying to make the best of it," one official declared as the evacuation played out.
RELATED: Oroville Dam managers made missteps in handling crisis, AP reports
During an early phone call marked in the notes as "governor call-in," an unidentified official apparently broached the possibility of calling off the evacuation: "Tell everyone to come back."
Another official, identified only as Mark, responded that the order would remain in place until the agency learned more about what was happening.
"Need inspections," the notes continued. "Unsure of current state."
Brown spokesman Evan Westrup said the governor was in close contact with aides managing the crisis. Westrup said Brown's office did not ask to lift the evacuation order.
The notes also reveal an unidentified female geologist played a critical role in persuading state and local officials that the failed emergency spillway posed an immediate threat. On the afternoon of Feb. 12, the geologist urgently brought new drone photos to officials overseeing the crisis.
Butte County Sheriff Kory Honea saw the photos when he stopped by and heard the worried discussion of the damage shown. It was Honea, not a state official, who finally ordered the evacuation.
"If it wasn't for one geologist ... he is afraid we wouldn't have even caught the problem," the notes quote Whitlock, a dam manager, as saying.