Yes, I can see that and given that my work starts to spread beyond my personal bubble it might hit people that actually find it too disturbing. In this case I can't help to suspect it is more a block by association since I tend to follow people from "both" sides.
this problem might be exacerbated now with Twitters random retweet functionality, aka the heart button. Even worse with popular accounts, as their contents might seep into your timeline far more often (which is why I blocked Kanye for example)