A federal judge refused Thursday to delay the start of the prison sentence for I. Lewis Libby Jr. in the CIA leak case while he appeals his conviction, meaning he could be ordered to surrender within two months.