BBC comedian Nish Kumar was met with boos and bread rolls after making Brexit jokes at a charity cricket lunch.