Generally, it's normal to see fluctuations in your sub count because we want to make sure the numbers are 100% accurate, so we perform additional verification & spam reviews. Also, it's likely folks are subscribing & unsubscribing as well. More info: https://goo.gl/6B1Eb6
Just to confirm, are your subs reporting that they are getting unsubscribed from your channel? We use technology to identify spam accounts and we don't remove legit subscribers from the sub count. Let us know.