Eliminated from the race were Doha, Qatar; Prague, Czech Republic, and Baku, Azerbaijan.
The final field was selected by the International Olympic Committee executive board.
The big four - Chicago, Madrid, Tokyo and Rio - all virtually were assured of advancing to the final stage. The main issue had been whether Doha also would make the cut.
"It's a very competitive field," Chicago Mayor Richard Daley said. "The competition begins now."
In the end, Doha was left out even though it tied for third with Chicago and was ahead of Rio in an IOC evaluation report assessing the overall technical merits of the bids.
The IOC said Doha's proposal to hold the Olympics in October, rather than July or August, to avoid the searing summer heat was the decisive factor.
The board "unanimously decided not to grant this exception as it conflicts with the international sporting calendar and would therefore be bad for the athletes and for sports fans," IOC spokeswoman Giselle Davies said. "It would disrupt athletes' training programs and overload the sports schedule at an already busy time of the year.
"Today the executive board felt it would be a disservice to Doha to take them into the next round."
There were signs the IOC board had been locked in debate about whether to include the Qatari bid. The announcement of the shortlist was delayed by nearly half an hour, and IOC president Jacques Rogge did not appear, leaving the duties to Davies.
"All the bids were of a very high standard," Rogge said later in a separate IOC statement. "It is a tribute to the health of the Olympic Movement that the field was so strong. I congratulate the candidate cities and I hope that those cities which were unsuccessful this time have benefited from the process."
The finalists now advance to a 16-month race that will culminate on Oct. 2, 2009, with a secret ballot by the full IOC at its session in Copenhagen, Denmark.
The evaluation report, which assessed sports venues, government support, security and other matters, put Tokyo in front, followed closely by Madrid. In a surprise, Chicago and Doha were tied for third, with Rio fifth. Prague and Baku were sixth and seventh.
"I feel relieved, exhilarated and very much honored," Chicago bid leader Patrick Ryan said. "We are going to study the report and we're going to learn from that and correct all the deficiencies."
The early ranking is not crucial. When the IOC picked the finalists for the 2012 Olympics in 2004, London was ranked third behind first-place Paris and Madrid in the technical report. London went on to win the games in 2005.
"This is the first hurdle," U.S. IOC member Bob Ctvrtlik said. "This is Chicago's first time in this process. Our goal was to make it to this phase. We'll keep improving the bid until we submit our bid book later this year."
Chicago is a contender to take the Summer Games back to the U.S. for the first time since the 1996 Atlanta Games.
Madrid is back again after a third-place finish in the vote for the 2012 Olympics, which went to London.
"I think there are about 40 million Spaniards that want us to bring the games home," bid leader Mercedes Coghen said.
Tokyo, which held the games in 1964, hopes to bring the Olympics to Asia eight years after Beijing.
"Now it's full steam ahead for my team and the Japanese people," Tokyo bid chief Ichiro Kono said. "We have no time to lose in preparing the best possible candidature file for the IOC.
Rio would become the first South American city to host the Olympics.
"We're very happy - the emotion to have a city of a country that never organized the Olympic Games among the others that have already organized. It's a great challenge but I think we are ready," Rio bid president Carlos Arthur Nuzman said.
Doha, capital of a tiny but wealthy Arab Gulf country of about 1 million people, had loomed as the wild card as it sought to bring the games to the Middle East for the first time. It cited its hosting of the 2006 Asian Games as evidence that it can handle the Olympics. But the IOC rejected the proposed games dates of Oct. 15-30.
Davies said Wednesday's decision did not rule out Doha or another Middle Eastern country ever hosting the Olympics - "especially given the IOC's desire to see the games held around the world."
Some rival bid officials had been worried that if Doha made the shortlist, the city - while a long shot to win - would have the capacity to take away crucial votes in the early rounds of voting.
Doha bid chairman Hassan Ali Bin Ali congratulated the finalists.
"While we are of course disappointed, we remain extremely proud to have been given the opportunity over the past 12 months to demonstrate that Doha is a truly modern and developing city, able to compete with some of the world's largest capitals and able to host international sporting events of the highest standard," he said.
Cities that made the cut will have to submit their detailed bid files to the IOC by Feb. 12, 2009. After that, a panel of IOC experts will visit each of the cities, tour the proposed sites and meet with bid and government leaders. The panel will release an evaluation report to the IOC members a month before the October 2009 vote.
The 2016 decision was the centerpiece of a three-day IOC board meeting in Athens, the last before the committee gathers in Beijing on the eve of the Aug. 8-24 Olympics.