Mayor Bloomberg and Governor Paterson are in attendance for the service.
The pope entered the stadium via popemobile and circled around the warning track, before heading into the Yankees dugout to dress for the mass.
The mass began shortly after 2:30 p.m.
The visit by Benedict to ground zero was a poignant moment in a trip marked by unexpectedly festive crowds anxious to see the former academic who for three years has led the world's Roman Catholics.
Benedict was driven in the popemobile part-way down a ramp now used mostly by construction trucks to a spot by the north tower's footprint. He walked the final steps, knelt in silent prayer for a few moments, then rose to light a memorial candle.
Addressing a group that including survivors, clergy and public officials, he acknowledged the many faiths of the victims at the "scene of incredible violence and pain."
The pope also prayed for "those who suffered death, injury and loss" in the attacks at the Pentagon and in the crash of United Airlines Flight 93 in Shanksville, Pa. More than 2,900 people were killed in the four crashes of the airliners hijacked by al-Qaida.
"God of peace, bring your peace to our violent world," the pope prayed on a chilly, overcast morning. "Turn to your way of love those whose hearts and minds are consumed with hatred."
Benedict invited 24 people with ties to ground zero to join him: survivors, relatives of victims and four rescue workers. He greeted each member of the group individually as a string quartet played in the background. In his prayer, he also remembered those who, "because of their presence here that day, suffer from injuries and illness."
New York deputy fire chief James Riches, father of a fallen Sept. 11 firefighter, said the pope's visit was important and gave him "a little consolation."
Hundreds of people stood just outside the site, behind police barricades, hoping for a glimpse of the pope.
The site where the World Trade Center was destroyed is normally filled with hundreds of workers building a 102-story skyscraper, a memorial and transit hub. It bears little resemblance to the debris-filled pit where crews toiled to remove twisted steel and victims' remains.
The remains of more than 1,100 people have never been identified.
Benedict was joined by New York Cardinal Edward Egan, along Mayor Michael Bloomberg, New York Gov. David Paterson and New Jersey Gov. Jon Corzine. The land is owned and managed by the Port Authority of New York & New Jersey.
Benedict has addressed terrorism several times during his six-day visit.
In a private meeting with President Bush, the two leaders "touched on the need to confront terrorism with appropriate means that respect the human person and his or her rights," according to a joint U.S.-Holy See statement.
Benedict has been critical of harsh interrogation methods, telling a meeting of the Vatican's office for social justice last September that, while a country has an obligation to keep its citizens safe, prisoners must never be demeaned or tortured.
Addressing the United Nations on Friday, Benedict warned diplomats that international cooperation needed to solve urgent problems is "in crisis" because decisions rest in the hands of a few powerful nations.
The pope also insisted that the way to peace was by ensuring respect for human dignity.
"The promotion of human rights remains the most effective strategy for eliminating inequalities between countries and social groups, and increasing security," the pope said.
Those whose rights are trampled, he said, "become easy prey to the call to violence and they then become violators of peace."