Opposition to freedom of movement is driven by racism, a former Labour frontbencher and ally of Jeremy Corbyn has said.