However, Mueller claimed that this was partly the fault of telecoms companies which had provided the FBI with "too much information".
"We are committed to ensuring that we not only get this right, but maintain the vital trust of the American people," he said.
The problems stemmed from the FBI's use of information request letters which was made much simpler after the passage of the Patriot Act.
"Everybody wants to stop terrorists. But Americans believe in our privacy rights and we want those protected," said Senate Judiciary chairman Patrick Leahy.
"There has to be a better chain of command for this. You cannot just have an FBI agent who decides he'd like to obtain Americans' records, bank records or anything else and do it just because they want to."
Leahy added that the FBI had now reformed its practices since March 2007 and would stay within the law.