Dominic Cummings should not be permitted to roam freely around the Palace of Westminster, according to a senior Tory peer.