Cabinet Ministers are allegedly among those calling for the Home Secretary to go after being reassigned to the post