Among the missing were two workers and a woman who was staying in an apartment at a town house flattened by the crane.
"Each passing hour, things get a little more grim," Fire Commissioner Nicholas Scoppetta said. Twenty-four others were injured, including 11 first responders, said Mayor Michael Bloomberg. Eight remained hospitalized Sunday, officials said.
The crane rose 19 stories and was attached to an apartment tower under construction when it broke away Saturday and toppled like a tree onto buildings as far as a block away.
City officials said the crane was inspected Friday. At day later, it was being lengthened with a new section, a process known as "jumping," when it fell.
Bloomberg said mechanical failure or human error may have caused the accident. "As far as we can tell, all procedures that were called for were being followed," he said.
Bloomberg said that about 250 cranes are operating in the city on any given day, and the accident should not alarm New Yorkers living near high-rise construction sites.
"Do I think that you should worry if there's a crane across the street? No," Bloomberg said. "This is such a rare thing that I don't think we should worry about it."
The city had issued 13 violations in the past 27 months to the construction site where a 43-story high-rise condominium was going up. "Every large construction site has violations," Bloomberg said.
Manhattan Borough President Scott Stringer said Bloomberg should form a multi-agency task force to inspect major construction projects.
"It is unacceptable for the Department of Buildings to say yesterday that the 13 open violations on this construction site were ... business as usual," Stringer said. "We can't keep going on like this."
The massive crane collapsed around 2:20 p.m. Saturday afternoon, smashing into a high-rise building on 51st Street and Second Avenue. It then continued falling striking on top of another smaller brownstone, with a bar on the first floor.
"That happened and here's my thought," said eyewitness, Sean Watts. "Oh my god, it's coming to me in the next second and a half, I'm gone."
The company that manages the crane says a piece of steel fell off and sliced the tie that was holding the crane to the building it was constructing.
It appears that a piece of steel fell and sheared off one of the ties holding the crane to the building, causing it to detach and topple, said Stephen Kaplan, an owner of the Reliance Construction Group.
"It was an absolute freak accident," Kaplan said. "All the piece of steel had to do was fall slightly left or right, and nothing would have happened."
Kaplan said the company had subcontracted the work to different companies and was not in charge of the crane. Phone messages and an e-mail left for the crane's owner, New York Crane & Equipment Corp., were not immediately returned.
"Once that fell down, the entire thing just went backwards," said construction worker, Gregory Jones. "You could literally see people hanging on and it was wow, wow."
Four construction workers were killed Saturday. They were crews either on the crane or directly in the construction zone.
Killed were construction workers Wayne Bleidner, 51, of Pelham; Brad Cohen, 54, of Farmingdale; Anthony Mazza, 39, of Staten Island; and Aaron Stephens, 45, of New York City, police said. Family who answered at the homes of Bleidner and Mazza had no comment Sunday.
On Sunday, the Reliance Construction Group, the project's contractor released a statement expressing sympathy to the families of the dead and the injured and said it was cooperating with government investigators.
Reliance said it had subcontracted different parts of the job and that New York Crane owned the crane. A telephone message left with New York Crane Sunday wasn't returned.
The missing woman had come from Miami to celebrate St. Patrick's Day and visit a friend who lived in the brownstone, said John LaGreco, owner of Fubar, a saloon on the ground floor.
She was in her friend's second-floor apartment at the time of the collapse, he said. Her friend was rescued, he said.
Mayor Bloomberg, Fire Commissioner Nicholas Scoppetta and Soon to be Gobernor David Paterson arrived to the scene Saturday and offered their condolences to those affected by this disaster.
"This construction accident is one of the worst the city has had," said Mayor Bloomberg. "And our hearts go out to all the victims and their families."
"It's a horrible situation, very gory. There's blood in the street," said Lt. Gov. David Paterson. "But we are very lucky to have brave police and firefighters here."
James Kennelly, the lead partner at East 51st Development Company, which owns the property, issued a written statement expressing the group's dismay over the accident.
People who live in and around the collapse zone were evacuated. One of temporary shelters are located outside of High School of Art and Design.
Witnesses say they heard the rumble and described the rushing right into the street after the collapse.
"All of a sudden, I heard two or three loud explosions and then lobby of my building blew up," said evacuee, Pamela Mohr.
Pamela Mohr was in her apartment building next to the construction site taking a shower when the giant crane collapsed.
She along with many other residents in neighboring buildings gathered up their loved ones and ran away from the huge cloud of smoky debris.
"All the debris from that corner kind a fell down on our roof," said evacuee Stefan Pokorny. "And if we're like five feet to the left...and we all would have been pancakes."
Neighbors say the force of the crane hitting the building rumbled and shook the entire neighborhood.
"The whole place started trembling and I was like, what's going on? I ran into the window and I saw from the window that the whole crane was coming down into our garden," said Stefan Pokorny. "At that moment, I thought that's it, I'm dying."
The Red Cross set up a reception area for displaced residents at the Art and Design High School on East 57th Street where many evacuees sought out more information on the safety of their buildings.
"I don't know how much damage was sustains our building, they are supposedly checking all the buildings out in the area," said evacuee Andrew Cenpa.
Before residents can return home, inspectors have to determine that the buildings are structurally sound. In the meantime, temporary shelters will remain open.
Just hours after the collapse, the property owner released a statement.
"There are no words to describe the level of devastation we feel today as a result of this tragic event. Our heart and prayers are with the families of those who died in this horrible accident. We are also praying for a full recovery for the individuals who have been injured today. Our first priority is to support the police and fire rescue operation to ensure that all victims are accounted for and the site is rendered safe as quickly as possible.
We hired Reliance Construction Group (RCG) as our Construction Manager because not only do they have a strong reputation as quality builders, but also for their outstanding safety record. In addition, RCG has hired subcontractors of similar record and reputation. New York Crane, to the best of our knowledge, has earned a reputation as the pre-eminent crane company in the region.
We expect RCG and New York Crane to cooperate fully with any and all investigations into this tragic accident. We must find out why and how this happened as soon as possible."