That was one fight Frazier could never win.
He was once a heavyweight champion, and a great one at that. Ali would say as much after Frazier knocked him down in the 15th round en route to becoming the first man to beat Ali at Madison Square Garden in March 1971.
But he bore the burden of being Ali's foil, and he paid the price. Bitter for years about the taunts his former nemesis once threw his way, Frazier only in recent times came to terms with what happened in the past and said he had forgiven Ali for everything he said.
Frazier, who died Monday night after a brief battle with liver cancer at the age of 67, will forever be linked to Ali. But no one in boxing would ever dream of anointing Ali as The Greatest unless he, too, was linked to Smokin' Joe.
"You can't mention Ali without mentioning Joe Frazier," said former AP boxing writer Ed Schuyler Jr. "He beat Ali, don't forget that."
They fought three times, twice in the heart of New York City and once in the morning in a steamy arena in the Philippines. They went 41 rounds together, with neither giving an inch and both giving it their all.
In their last fight in Manila in 1975, they traded punches with a fervor that seemed unimaginable among heavyweights. Frazier gave almost as good as he got for 14 rounds, then had to be held back by trainer Eddie Futch as he tried to go out for the final round, unable to see.
"Closest thing to dying that I know of," Ali said afterward.
Ali was as merciless with Frazier out of the ring as he was inside it. He called him a gorilla, and mocked him as an Uncle Tom. But he respected him as a fighter, especially after Frazier won a decision to defend his heavyweight title against the then-unbeaten Ali in a fight that was so big Frank Sinatra was shooting pictures at ringside and both fighters earned an astonishing $2.5 million.
The night at the Garden 40 years ago remained fresh in Frazier's mind as he talked about his life, career and relationship with Ali a few months before he died.
"I can't go nowhere where it's not mentioned," he told The Associated Press. "That was the greatest thing that ever happened in my life."
Though slowed in his later years and his speech slurred by the toll of punches taken in the ring, Frazier was still active on the autograph circuit in the months before he died. In September he went to Las Vegas, where he signed autographs in the lobby of the MGM Grand hotel-casino shortly before Floyd Mayweather Jr.'s fight against Victor Ortiz.
An old friend, Gene Kilroy, visited with him and watched Frazier work the crowd.
"He was so nice to everybody," Kilroy said. "He would say to each of them, `Joe Frazier, sharp as a razor, what's your name?"'
Frazier was small for a heavyweight, weighing just 205 pounds when he won the title by stopping Jimmy Ellis in the fifth round of their 1970 fight at Madison Square Garden. But he fought every minute of every round going forward behind a vicious left hook, and there were few fighters who could withstand his constant pressure.
His reign as heavyweight champion lasted only four fights -- including the win over Ali -- before he ran into an even more fearsome slugger than himself. George Foreman responded to Frazier's constant attack by dropping him three times in the first round and three more in the second before their 1973 fight in Jamaica was waved to a close and the world had a new heavyweight champion.
Two fights later, he met Ali in a rematch of their first fight, only this time the outcome was different. Ali won a 12-round decision, and later that year stopped George Foreman in the Rumble in the Jungle in Zaire.
There had to be a third fight, though, and what a fight it was. With Ali's heavyweight title at stake, the two met in Manila in a fight that will long be seared in boxing history.
Frazier went after Ali round after round, landing his left hook with regularity as he made Ali backpedal around the ring. But Ali responded with left jabs and right hands that found their mark again and again. Even the intense heat inside the arena couldn't stop the two as they fought every minute of every round with neither willing to concede the other one second of the round.
"They told me Joe Frazier was through," Ali told Frazier at one point during the fight.
"They lied," Frazier said, before hitting Ali with a left hook.
Finally, though, Frazier simply couldn't see and Futch would not let him go out for the 15th round. Ali won the fight while on his stool, exhausted and contemplating himself whether to go on.
It was one of the greatest fights ever, but it took a toll. Frazier would fight only two more times, getting knocked out in a rematch with Foreman eight months later before coming back in 1981 for an ill advised fight with Jumbo Cummings.
"They should have both retired after the Manila fight," Schuyler said. "They left every bit of talent they had in the ring that day."
Born in Beaufort, S.C., on Jan 12, 1944, Frazier took up boxing early after watching weekly fights on the black and white television on his family's small farm. He was a top amateur for several years, and became the only American fighter to win a gold medal in the 1964 Olympics in Tokyo despite fighting in the final bout with an injured left thumb.
"Joe Frazier should be remembered as one of the greatest fighters of all time and a real man," promoter Bob Arum told the AP in a telephone interview on Tuesday night. "He's a guy that stood up for himself. He didn't compromise and always gave 100 percent in the ring. There was never a fight in the ring where Joe didn't give 100 percent."
After turning pro in 1965, Frazier quickly became known for his punching power, stopping his first 11 opponents. Within three years he was fighting world-class opposition and, in 1970, beat Ellis to win the heavyweight title that he would hold for more than two years.
It was his fights with Ali, though, that would define Frazier. Though Ali was gracious in defeat in the first fight, he was as vicious with his words as he was with his punches in promoting all three fights -- and he never missed a chance to get a jab in at Frazier.
Frazier, who in his later years would have financial trouble and end up running a gym in his adopted hometown of Philadelphia, took the jabs personally. He felt Ali made fun of him by calling him names and said things that were not true just to get under his skin. Those feelings were only magnified as Ali went from being an icon in the ring to one of the most beloved people in the world.
After a trembling Ali it the Olympic torch in 1996 in Atlanta, Frazier was asked by a reporter what he thought about it.
"They should have thrown him in," Frazier responded.
He mellowed, though, in recent years, preferring to remember the good from his fights with Ali rather than the bad. Just before the 40th anniversary of his win over Ali earlier this year -- a day Frazier celebrated with parties in New York -- he said he no longer felt any bitterness toward Ali.
"I forgive him," Frazier said. "He's in a bad way."
Dahlberg reported from Las Vegas.