Swords have a constant multiplier when stabbing and slashing, which Brandish involves both one slash and one stab. Axes, however, have a different multiplier for stab and slash. Unfortunately, Axes do lower damage on average due to Brandish having both a stab and slash. Here are the multipliers for Swords and Axes.
Based on calculations that I've done previously (which you should be able to find with the search function) an axe user will likely be between 12 and 20% weaker than a sword user given that their funding level is equal and that you're at least approaching endgame. Endgame in this sense meaning level 140+, with enough money to have a couple 30%s on your weapon, a decent glove and a scrolled (though not necessarily double digit) cape/shoe. Prior to this point the gap is smaller, but the further you go towards really really endgame, the bigger the gap becomes.