Regulators' efforts to rein in Wall Street's biggest banks are in danger of backfiring.

Guidelines aimed at strengthening lending standards are shifting the market for high-yield credit to less-supervised loan funds, raising alarm this week from the Financial Stability Oversight Council. Because the funds don't have depositors, some of their money comes from Wall Street banks, leaving systemically important institutions exposed to risks regulators hoped to avoid.

Loans by nonbanks such as KKR & Co. and Apollo Global Management LLC affiliates are a small part of the market, but they're growing. Direct-lending funds, which raise money from institutional investors such as pension funds and insurance companies, surged to a global record last year of $29.9 billion, according to financial data firm Preqin. Loans by U.S. business development companies, or BDCs, many of which have credit lines with banks, jumped to $55 billion last year from $17 billion in 2010, according to Deloitte LLP.