No injuries were reported.
More than 30 additional earthquakes with magnitudes of at least 3.5 jiggled the same area near the southern end of the Salton Sea, Graves said.
"The type of activity that we're seeing could possibly continue for several hours or even days," Graves said.
The quakes pushed 20 mobile homes at a trailer park off their foundations, displacing the families that lived in them, said Maria Peinado, a spokeswoman for the Imperial County Emergency Operations Center.
Sporadic power outages affecting 2,500 Imperial Irrigation District customers also prompted authorities to evacuate some patients from one of the county's two hospitals.
At the El Sol Market in Brawley, food packages fell from shelves and littered the aisles.
Several glasses and a bottle of wine crashed to the floor and shattered at Assaggio, an Italian restaurant in Brawley, said owner Jerry Ma. The shaking was short-lived but intense, he said.
"It felt like there was quake every 15 minutes. One after another. My kids are small and they're scared and don't want to come back inside," said Mike Patel, who manages Townhouse Inn & Suites in Brawley.
A TV came crashing down and a few light fixtures broke inside the motel, Patel said.
A Brawley police dispatcher said several downtown buildings sustained minor damage.
The first quake, with a magnitude of 3.9, occurred at 10:02 a.m. The USGS said more than 100 aftershocks struck the same approximate epicenter, about 16 miles north of El Centro.
Some shaking was felt along the San Diego County coast in Del Mar, some 120 miles from the epicenter, as well as in the Coachella Valley, southern Orange County and parts of northern Mexico.
USGS seismologist Lucy Jones said earthquake swarms are characteristic of the region, known as the Brawley Seismic Zone.
"The area sees lots of events at once, with many close to the largest magnitude, rather than one main shock with several much smaller aftershocks," Jones said.
The last major swarm was in 2005, following a magnitude-5.1 quake, she said.
Sunday's quake cluster occurred in what scientists call a transition zone between the Imperial and San Andreas faults, so they weren't assigning the earthquakes to either fault, Graves said.