Dealer revenue from negotiating interest-rate swap transactions is poised to plunge about 45 percent as new rules boost trading costs, pressures that may prompt banks to participate less in the $633 trillion over-the-counter derivatives market, Tabb Group LLC estimates.

Banks collect about $3.25 billion a year from trading rate swaps with their customers, Tabb said. That revenue will shrink to $1.8 billion in 2014 as most transactions shift to public markets, according to a research report meant for Tabb customers that Bloomberg News obtained. Dealers will also need to hold more capital to back trades, boosting expenses, said Will Rhode, who wrote the report.

"There probably will be some painful conversations at the end of 2013 about how much this business costs to run," Rhode said in a phone interview. While increased electronic trading will reduce transaction sizes and lead to more trades, it won't be enough to offset lost profits, he added. "There's no way the turnover increase can be sufficient enough to make up for the revenue shortfall," he said. "These products that were basically free have just become much more expensive."