The Lib Dem leader was criticised for suggesting some Leave voters were nostalgic for a white Britain.