The blaze, dubbed the Warm Fire, was reported to be near Lake Hughes and Warm Springs roads. About 75 percent of the structures at Warm Springs Rehabilitation Facility were burned.
PHOTOS: Acres burned, buildings destroyed in Castaic brush fire
Firefighters believe the fire, which is moving rapidly through dry brush, has the potential to grow to 2,000 acres. Fire personnel are dealing with triple digit temperatures, steep terrain and heavy, dry vegetation that can add extra fuel to the blaze.
"In this specific area, it's been about 30 years since it burned. This is very heavy brush and some very steep terrain," said Battalion Chief Pat Rohaley with the Los Angeles County Fire Department.
Robert Garcia, fire chief with the Angeles National Forest Service, said the fire started in the area of the rehabilitation facility and quickly grew.
Edison crews are also at the scene as the blaze threatens nearby power lines.
No evacuations have been ordered. Lake Hughes Road at Elizabeth Lake Road was shut down.
The cause of the fire is under investigation. An area of interest was tapped off at the rehabilitation center.