Washington rewrote the rulebook for Wall Street after the 2008 financial crisis, but dangerous lending is still eluding regulators.

Take Bomgar Corp., which just lined up $439 million in loans. The deal marked the software company's third trip to the debt markets this year. By one estimate, Bomgar's leverage could soon spike to 15 times its earnings, raising questions about whether the firm could ever pay it off.

These kinds of transactions are increasingly common in the U.S.'s more than $2 trillion market for leveraged loans and junk bonds, and agencies including the Federal Reserve can't do much about it. That's because some of the most aggressive financing is being done outside the traditional banking sector.