The companies join PayPal
"Visa has decided not to join the Libra Association at this time," the company said in a statement.
"We will continue to evaluate and our ultimate decision will be determined by a number of factors, including the Association's ability to fully satisfy all requisite regulatory expectations."
The latest exodus could be particularly problematic, as it includes all major payments companies that were founding members of the project.
Facebook announced plans to launch the digital currency in June 2020 in partnership with other Libra Association members.
Libra will press ahead with plans to formally charter the association, Dante Disparte, its head of policy and communication, said in a statement on Friday.
"We look forward to the inaugural Libra Association Council meeting in just three days and announcing the initial members of the Libra Association."
Facebook CEO Mark Zuckerberg is expected to discuss the project when he testifies before the US House Financial Services Committee on October 23.
France and Germany last month pledged to block Libra from operating in Europe and backed the development of a public cryptocurrency instead.
US Federal Reserve chairman Jerome Powell has cited serious privacy, money laundering, consumer protection and financial stability concerns that must be addressed before the project can advance.