The 54-year-old media mogul is a judge on the singing contest - which is his brainchild - however he could be planning to leave the show since it is pulling in fewer viewers than in previous years - with the live show on Tuesday night attracting only 4.8 million viewers.
The drop in numbers could force bosses to lower the budget on the series, meaning Simon will walk away and back onto the judging panel of the UK version of the talent show.
An insider told The Sun newspaper: "People have talked as if it is all over for The X Factor USA but that's not the case at all.
"It is true that ratings are down but it still does well with younger viewers if it was to come back next year it would have to have a lower budget.
"Simon leaving the show would allow a cheaper panel to be introduced - and give him the time to be reunited with the British show."
The X Factor USA is currently in its third season with the winner awarded a $1m recording contract with Simon's label Syco, however, in the previous two series' it was worth $5m.
Simon - who is expecting his first child with New York socialite Lauren Silverman - is currently on the judging panel alongside Demi Lovato, Kelly Rowland and Paulina Rubio, and he is mentoring the groups category.
24.com publishes all comments posted on articles provided that they adhere to our Comments Policy. Should you wish to report a comment for editorial review, please do so by clicking the 'Report Comment' button to the right of each comment.
Shailene Woodley has written an essay detailing her arrest for trespassing and what people can do to help protest the Dakota Access Pipeline. Read More »
Brad Pitt has met up with his oldest son Maddox, separately from the rest of his children, in a supervised visit. Read More »
10 epic pics.
All the deets.
South AfricaCity Press
Johannesburg CBDResourcing Solutions
HousesR 2 495 000
HousesR 3 220 000
HousesR 3 300 000