A part of why fist weapons are difficult to use is their incredibly low poise damage combined with the lack of some major redeeming factor. In terms of poise damage, fist weapons compete with thrusting swords, daggers, and I believe whips -- all these groups have the lowest poise damage in the game. Daggers have high critical modifiers to go with their low weight and fast attacks, though, and thrusting swords can attack from behind shields, tend to have excellent range for their weight class, and deal piercing damage.
The other issue is that highly effective characters have synchronicity between their poise and the poise damage of their weapons. Remember, an enemy cannot attack you if they're staggered, so dealing high poise damage is a defensive trait as well as an offensive one. If you have 80 poise and the claws, but your adversary has 40 poise and a claymore, they're going to stagger you first because of the synchronicity between their poise value and their weapon; the claws do such little poise damage that they allow adversaries to trade with you freely, meaning that the trade will be less a case of poise and more of poise damage.
Fist weapons lack any redeeming features that make up for their lack of poise damage, whereas daggers and thrusting swords have good criticals, and the latter group also has a range of other advantages. But fist weapons require you to get so close that you're almost certainly opening yourself up to attack, and you're always going to lose the trade or the poise contest in general.