The accident could delay the delivery of the stadium in Sao Paulo by FIFA's December deadline to have all 12 venues ready. Work was immediately halted at the Itaquerao Stadium, which was practically completed before the collapse.
Television images showed a huge metal structure buckled atop the stadium, destroying part of the stands in the east side of the venue. A LED panel installed outside the venue also was hit.
Firefighter official Mauro Lopes said in a radio interview that at least three people died in the accident, which apparently happened when a crane collapsed on top of the metal structure. The crane was installing what was the last part of the structure.
The accident happened at lunchtime, so not a lot of workers were on site at the time.
The stadium will seat nearly 70,000 people for the opener on June 12. The venue is scheduled to host other five matches, including a semifinal. It was initially expected to be built for the Confederations Cup this year, but delays with financing for the venue prompted authorities to scrape the stadium from the World Cup warm-up tournament.
Brazilian club Corinthians, which owns the stadium, released a brief statement "lamenting" the accident.
Constructor Odebrecht said it was already investigating what happened.
Brazil is running against time to deliver the last six World Cup stadiums by the end of the year, although work at the Itaquerao was advanced compared to the other venues.
"FIFA and the LOC have learnt of the death of workers at the Corinthian's Arena site in Sao Paulo with great sadness," FIFA said in a statement. "We wish to send our heartfelt condolences to the family of the workers who tragically died today."
Football's governing body said the "safety of workers is the top priority" to World Cup organizers.
"We know the safety of all workers has always been paramount for all the construction companies contracted to build the 12 FIFA World Cup stadiums," it said in a statement. "The local authorities will fully investigate the reasons behind such a tragic accident."
FIFA President Sepp Blatter said he was "deeply saddened by the tragic death of workers" at the Corinthians stadium. "Our heartfealt condolences are with the families," he said.
FIFA Secretary General Jerome Valcke said he was "extremely shocked by the news from Sao Paulo."
"Our thoughts are with the families of the victims of this accident," he said.