Zuri C. Towns, 45, was identified as the person allegedly responsible, and was arrested by police.
Police say they were notified Saturday about the vandalism overnight at Church of God in Christ for All Saints; Bethel A.M.E. Church; Calvary Baptist Church; and Union Baptist Church, all in Morristown; and Morristown Church of Christ in Morris Township.
The pastor of Bethel A.M.E. Church says one of the churches captured surveillance video, and the footage was turned over to police.
Police say Towns used to go to one of the churches as a kid. The acts of vandalism is not being considered a bias crime.
Surveillance images show the moment the man walked up to Church of God in Christ for All Saints, pulled a rock from his pocket and threw it into a stained-glass window.
Bethel A.M.E. Pastor Sidney Williams said the news left some of his members shaken.
"Several of our members said they're not coming to church this morning because they're fearful for their lives," Williams said.
Each pastor said they had already stepped up security due to recent violence at churches around the country.
The pastors worked to clean up the damage before Sunday's service.
"We still preach forgiveness but we also have to preach safety and understanding," said Williams.
New Jersey Governor-elect Phil Murphy tweeted that he was "disgusted" by the vandalism spree.
Disgusted by reported vandalism of African-American churches across Morristown. These cowardly acts will not shake our faith. They will renew our determination to come together as one NJ. -PM— Phil Murphy (@PhilMurphyNJ) November 26, 2017