According to a report in the San Francisco Chronicle, they are concerned the 88-year-old democrat may not be fit for office.
The newspaper spoke to her colleagues, including four U.S. senators and three former Feinstein staffers.
Not wanting to be identified, they claim she is showing clear signs of cognitive decline, including forgetting who she is meeting with and repeating the same information multiple times throughout a conversation.
Feinstein's term expires in 2024.
She provided this statement to The Chronicle on March 28.
"The last year has been extremely painful and distracting for me, flying back and forth to visit my dying husband who passed just a few weeks ago. But there's no question I'm still serving and delivering for the people of California."
Feinstein says she'll put her record up against anyone's.
Watch the full interview with San Francisco Chronicle's Washington correspondent Tal Kopan in the media player above.