Investors who piled into anything and everything in the junk-debt market in recent years have begun to run in the other direction at the first sign of trouble.

The turnabout has caught Wall Street's biggest banks off guard and is increasingly leaving them on the hook for funding takeovers that investors want little part of. On Tuesday, Bank of America Corp. and Morgan Stanley were forced to shelve the debt package backing the year's largest leveraged buyout—US$5.5 billion meant to fund Carlyle Group LP's purchase of Veritas, Symantec Corp.'s data-storage business, according to two people familiar with the matter.

"There's a risk of this happening more," said Jamie Farnham, who manages about $6 billion of high-yield bonds and leveraged loans for Los Angeles-based TCW Group Inc. "The amount of the market that is un-financeable is getting larger."