Holder, testifying to the House Judiciary Committee, was asked what criminal charges could be pursued against IRS employees or officials. Holder announced on Tuesday that the Justice Department was the investigating the IRS after the agency acknowledged that agents had singled out conservative groups for extra scrutiny when they applied for tax-exempt status.
"I can assure you and the American people that we will take a dispassionate view of this," Holder said. "This will not be about parties, this will not be about ideological persuasions. Anybody who has broken the law will be held accountable."
Holder also says it will take time to determine if there was criminal wrongdoing.
As the investigation widened, House Speaker John Boehner told reporters he had this question: "Who's going to jail over this scandal?"
"There are laws in place to prevent this type of abuse. Someone made a conscious decision to harass and to hold up these requests for tax exempt status," Boehner said. "I think we need to know who they are and whether they violated the law. Clearly someone violated the law."
Wednesday's hearing was the first of several in Congress that will focus on the issue.
The House oversight committee announced Wednesday that it will hold a hearing May 22, featuring Lois Lerner, the head of the IRS division that oversees tax exempt organizations, and former IRS Commissioner Douglas Shulman, whose five-year term ended in November.
The House Ways and Means Committee will hold a hearing Friday, featuring the acting IRS commissioner, Steven Miller, and the Treasury inspector general for tax administration, J. Russell George.
Three congressional committees are investigating the IRS for singling out tea party and other conservative groups during the 2010 congressional elections and the 2012 presidential election. But Holder's announcement would take the matter to another level if investigators are able to prove that laws were broken.
Ineffective management at the IRS allowed agents to improperly target tea party groups for more than 18 months, said a report released Tuesday by George's office.
The report said that while their applications for tax exempt status languished, tea party groups were asked a host of inappropriate questions, including: Who are your donors? What are the political affiliations of officers? What issues are important to the organization, and what are your positions on those issues? Will any officers in the group run for public office? Where do you work?
The report lays much of the blame on IRS supervisors in Washington who oversaw a group of specialists in Cincinnati who screened applications for tax exempt status.
It does not indicate that Washington initiated the targeting of conservative groups. But it does say a top supervisor in Washington did not adequately supervise agents in the field even after she learned the agents were acting improperly.
"The report's findings are intolerable and inexcusable," President Barack Obama said in a statement. "The federal government must conduct itself in a way that's worthy of the public's trust, and that's especially true for the IRS. The IRS must apply the law in a fair and impartial way, and its employees must act with utmost integrity. This report shows that some of its employees failed that test."
The agency started targeting groups with "Tea Party," ''Patriots" or "9/12 Project" in their applications for tax exempt status in March 2010, the inspector general's report said. By August 2010, it was part of the written criteria used to flag groups for additional scrutiny.
Lerner had been briefed on the matter in June 2011. She ordered the initial tea party criteria to be scrapped, but it later evolved to include groups that promoted the Constitution and the Bill of Rights. The practice was ended in May 2012, the report said.
IRS agents were trying to determine whether the political activities of such groups disqualified them for tax exempt status. These groups were claiming tax exempt status as organizations promoting social welfare. Unlike other charitable groups, they can engage in political activity. But politics cannot be their primary mission.
It is up to the IRS to make the determination.
But by using improper criteria, the IRS targeted some groups, even though there were no indications that they engaged in significant political activities, the report said. Other non-tea party groups that had significant political activities were not screened, the report said.
"The criteria developed by the Determinations Unit gives the appearance that the IRS is not impartial in conducting its mission," the report said.
"Unfortunately, the report raises more questions than it answers," said House Oversight Committee Chairman Darrell Issa, R-Calif. "What we do know for sure is that the IRS personnel responsible for granting tax exemptions systematically targeted conservative groups for extra scrutiny, and that officials in Washington, D.C., were aware of this practice, even while publicly claiming that it never happened."
The IRS on Friday apologized for singling out tea party and other conservative groups.
On Tuesday, the agency said, "After seeing issues with particular cases, inappropriate shortcuts were used to determine which cases may be engaging in political activities. It is important to note that the vast majority of these cases would still have been centralized based on the general criteria used for other cases."
Associated Press reporter Andrew Miga contributed to this report.