Northam said that while he previously apologized, after further reflection he does not believe he is in the photo. Northam also said he had previously worn blackface for a dance contest but later learned the racist connotations of such costumes and understands it was wrong.
WATCH: Gov. Ralph Northam's full statement
The Democratic governor said Saturday that he, in fact, was not in the photo and had never even seen the yearbook until Friday.
His refusal to resign signals a potential bruising fight between Northam and his former supporters. Leaders in both parties have repeatedly urged Northam to resign, saying he's lost the public's trust.
Northam is calling Virginia Democrats to try and gain support. Nearly his entire political base has called for him to resign.
On Saturday, former Vice President Joe Biden tweeted a call for Northam's resignation.
There is no place for racism in America. Governor Northam has lost all moral authority and should resign immediately, Justin Fairfax is the leader Virginia needs now.— Joe Biden (@JoeBiden) February 2, 2019
Northam's 1984 yearbook page shows a picture of a person in blackface and another wearing a Ku Klux Klan hood next to different pictures of the governor.
After the picture went public Friday, Northam issued a statement saying he was "sorry for the decision I made to appear as I did in this photo."
But the Democrat who spoke to Northam said that after further reflection the governor said he has no memory of wearing either of the racist costumes and doesn't not believe he is in the picture.
The Virginia Legislative Black Caucus, the state House Democratic Caucus and the state Senate Democratic Caucus all called on Northam to resign late Friday, along with several key progressive groups that have been some of the governor's closest political allies.
The yearbook images were first published Friday afternoon by the conservative news outlet Big League Politics. The Virginian-Pilot later obtained a copy from Eastern Virginia Medical School, which Northam attended. The photo shows two people looking at the camera - one in blackface wearing a hat, bow tie and plaid pants; the other in a full Ku Klux Klan robe.
An Associated Press reporter saw the yearbook page and confirmed its authenticity at the medical school.
In his first apology, issued in a written statement, Northam called the costume he wore "clearly racist and offensive," but he didn't say which one he had worn.
He later issued a video statement saying he was "deeply sorry" but still committed to serving the "remainder of my term."
"I accept responsibility for my past actions and I am ready to do the hard work of regaining your trust," Northam said.
A small number of protesters stood outside the governor's mansion Saturday to demand his resignation.
Northam's departure would mean current Lt. Gov. Justin Fairfax, a Democrat who is only the second African-American to win statewide office in Virginia, would be the next governor. Northam's term was set to end in 2022.
Black lawmakers said they met with Northam Friday evening, and said in a statement they appreciate his service.
"But given what was revealed today, it is clear that he can no longer effectively serve as governor," the Virginia Legislative Black Caucus said.
State Sen. Louise Lucas of Portsmouth, a close ally of Northam and longtime African-American lawmaker, said black leaders felt "mortally wounded" by the photo and that he has no option but to resign.
"He betrayed us," she told The Associated Press on Saturday.
Others said there was no question he should step down. Among them: Democratic presidential hopefuls Cory Booker, Julian Castro, Kirsten Gillibrand, Kamala Harris and Elizabeth Warren; newly elected Democratic U.S. Reps. Elaine Luria and Abigail Spanberger of Virginia; the NAACP and Planned Parenthood.
Northam spent years actively courting the black community in the lead up to his 2017 gubernatorial run, building relationships that helped him win both the primary and the general election. He's a member of a predominantly black church on Virginia's Eastern Shore, where he grew up.
"It's a matter of relationships and trust. That's not something that you build overnight," Northam told the AP during a 2017 campaign stop while describing his relationship with the black community. Northam, a folksy pediatric neurologist who is personal friends with many GOP lawmakers, has recently come under fire from Republicans who have accused him of backing infanticide after he said he supported a bill loosening restrictions on late-term abortions.
Last week, Florida's secretary of state resigned after photos from a 2005 Halloween party showed him in blackface while dressed as a Hurricane Katrina victim.
WLS-TV contributed to this report.