You might want to doublecheck if the fail2ban version you're using has IPv6 support -- just sayin' :-)
Conversation
It should really be about fully disabling legacy SSH password authentication. Presence of fail2ban is a strong sign of poor security.
Worth noting it's standard to be given a /64 or /56 IPv6 block. Some providers still only give a /128 but that's just poor setup on their part.
1) You are assuming fail2ban is only usable with SSH, but it's not.
2) It is possible to ban IP ranges, as opposed to single IP addresses.
3) It's not about A 'xor' B -- nobody is implying fail2ban should be your one and only single line of defense.
2
3
When it comes to SSH, it's an administrative service for which the regular folks on the Internet shouldn't have access to. i.e., there should be ACLs blocking access from the general public.