Holder said the FBI is coordinating with the Department of Justice to see if any laws were broken.
At a news conference Tuesday, Holder called the practice, "outrageous and unacceptable."
Numerous congressional committees already are investigating the IRS for singling out tea party and other conservative groups during the 2010 congressional elections and the 2012 presidential election. But Holder's announcement takes the matter to another level, if investigators are able to prove that laws were broken.
Holder's comments came a day after President Barack Obama said that if the agency intentionally targeted such groups, "that's outrageous and there's no place for it."
Steven Miller, the IRS acting chief, acknowledged "a lack of sensitivity" in the agency's screenings of political groups seeking tax-exempt status and insisted those mistakes won't be repeated.
The IRS apologized Friday for what it acknowledged was "inappropriate" targeting of conservative political groups. The IRS admission was made at an American Bar Association conference.
The agency started targeting groups with "Tea Party," ''Patriots" or "9/12 Project" in their applications in March 2010. The criteria later evolved to include groups that promoted the Constitution and the Bill of Rights. The practice ended in May 2012, according to a draft of an upcoming report by the Treasury inspector general for tax administration.
In some cases, the IRS acknowledged, agents inappropriately asked for lists of donors. The agency blamed low-level employees in a Cincinnati office, saying no high-level officials were aware.
Miller has emerged as a key figure in the controversy. In his first public comment on the case, Miller said there was "a shortcut taken in our processes" for determining which groups needed special screening.
In an opinion piece in Tuesday's editions of USA Today, Miller conceded that the agency demonstrated "a lack of sensitivity to the implications of some of the decisions that were made." He said screening of advocacy groups is "factually complex, and it's challenging to separate out political issues from those involving education or social welfare."
"The mistakes we made were due to the absence of a sufficient process for working the increase in cases and a lack of sensitivity to the implications of some of the decisions that were made," Miller wrote.
Miller said the agency has implemented new procedures that will "ensure the mistakes won't be repeated."
On Monday, the IRS said Miller was first informed on May, 3, 2012, that applications for tax-exempt status by tea party groups were inappropriately singled out for extra scrutiny.
At least twice after the briefing, Miller wrote letters to members of Congress to explain the process of reviewing applications for tax-exempt status without disclosing that tea party groups had been targeted. On July 25, 2012, Miller testified before the House Ways and Means oversight subcommittee, but again did not mention the additional scrutiny - despite being asked about it.
Miller's op-ed did not address why he did not inform Congress after he was briefed.
Miller was a deputy commissioner at the time. He became acting commissioner in November, after Commissioner Douglas Shulman completed his five-year term. Shulman had been appointed by President George W. Bush.
Sen. Charles Grassley, R-Iowa, questioned the timing of the IRS admission. It came days before the expected release of the inspector general's report.
"This timing is curious," Grassley says in a letter Tuesday to Miller. "The IRS chose not to fully answer long-standing congressional questions on the issue, even though they had been posed months before this particular question was asked at the conference."
Grassley asks Miller to provide records relating to the agency's decision to disclose the targeting of tea party groups at a Friday conference rather than to members of Congress who had been asking about it for more than a year. He also asks for any communications on the issue between the IRS and the White House.
At least three congressional committees have made similar requests. The House Ways and Means Committee, chaired by GOP Rep. Dave Camp of Michigan, is holding a hearing on the issue Friday and Miller is scheduled to testify.
Meanwhile, two Republican governors urged Obama to appoint a special prosecutor to investigate the case. Louisiana Gov. Bobby Jindal and Wisconsin Gov. Scott Walker call the allegations "Big Brother come to life." They want a special prosecutor to find out if any laws were broken and say Obama should fire any IRS employees responsible for the situation.
Sen. Roy Blunt, R-Mo., called Tuesday for Miller to step down or for Obama to replace him.
"If the reports are accurate that Steven Miller knew about the IRS' egregious targeting of conservative groups last year and misled members of Congress about those actions, he should step down or be removed immediately," Blunt said Tuesday.