Generally, it's normal to see fluctuations in your subscriber count. This is because we want to make sure the numbers are 100% accurate, so we perform additional verification and spam reviews. Also, it's likely folks are subscribing and unsubscribing as well.
People are being forcibly unsubscribed from channels. If they are being unsubscribed because of inactivity on the channel (which is fairly reasonable) perhaps notify them that they have been unsubbed so if they decide they want to watch the channel they can subscribe again