Cowell confirmed in a Twitter message that Spears, 30, the world's biggest pop phenomenon of the 2000s, will join him on X Factor, which searches for a new pop star from tens of thousands of ordinary Americans.
Spears will be joined by singer and actress Lovato, 19, to fill the judge's seats left by Paula Abdul and Nicole Scherzinger, who were both fired by Cowell after a disappointing first season in 2011.
Cowell announced the news on Twitter, sending a photo of himself with record producer Antonio "LA" Reid, Spears and Lovato and saying "Can't believe it's finally happening. Very exciting!"
*Which celeb would you want to see on the judging panel? E-mail us your suggestions.