The massive shakeup Wednesday night came hours after Paterno announced that he planned to retire at the end of his 46th season.
But the outcry following the arrest of former assistant coach Jerry Sandusky on molestation charges proved too much for the board to ignore.
One key question has been why Paterno and other top school officials didn't go to police in 2002 after being told a graduate assistant saw Sandusky assaulting a boy in a school shower.
Paterno says he should have done more. Spanier has said he was not told the details of the attack.
Sandusky has denied the charges.
Defensive coordinator Tom Bradley will serve as interim coach while Rodney Erickson will serve as interim school president.
Earlier in the day, Paterno said in a statement he was "absolutely devastated" by the case, in which Sandusky, his onetime heir apparent was charged with molesting eight boys in 15 years, with some of the alleged abuse taking place at the Penn State football complex.
"This is a tragedy," Paterno said. "It is one of the great sorrows of my life. With the benefit of hindsight, I wish I had done more."
John Surma, the vice chair of the board of trustees said, "these decisions were made after careful deliberations and in the best interests of the university as a whole."
"The past several days have been absolutely terrible for the entire Penn State community. But the outrage that we feel is nothing compared to the physical and psychological suffering that allegedly took place," he added.
Sue Paterno opened the door of the Paterno home briefly when a reporter knocked, then closed it and turned off the light.