Middle Age Hollywood

Randy Jackson is the only judge returning to "American Idol" next season ... but that almost wasn't the case.

Sources close to production tell TMZ there was "serious talk" of a clean sweep of the judge's panel ... but we're told Idol brass wanted to keep a connection to the original judge's table, so Randy was spared.

As we previously reported, Simon Cowell left for "The X Factor," Ellen DeGeneres stepped down after one season and Kara DioGuardi was let go -- leaving Randy as the only remaining judge.