According to reports, 55-year-old Simon has got his eye on superstars including Dolly Parton and Rod Stewart – and even wants to sign up Sir Tom Jones, following his axing from BBC rival The Voice.
A show insider told the Mirror: “There have been some discussion about introducing a fifth guest judge on the panel for the live shows to keep the format fresh.
“The idea would be to have a music legend to join Simon, Cheryl, Rita and Nick as an impartial judge each week.”
But the addition of an extra judge, although an impartial one, could put an end to the controversial deadlock.
The insider continued: “A fish judge means we have a majority decision for anything that happens, it would mean an end to deadlock too in sing offs which some viewers see as a cop out.”
While Simon is keen to keep attracting new, younger viewers, he also wants to keep older people entertained – and thinks Tom Jones is the perfect way to do this.
Another source added: “This new plan might get also get viewers who are much older and might tune in to see their idols on screen.
“Tom Jones was a big hit with older viewers on The Voice so it is that kind of theory.”
What do you think of the proposed changes to The X Factor? Let us know on Facebook and Twitter.