Madonna, 56, has reportedly refused to appear on the ITV singing competition although Simon really wants her to perform.
A source said: "There are very few people in the music world who Simon can't appeal to. He has made no secret of the fact that he is desperate to get Madonna on board but she just won't do it.
"The X Factor final is one of the biggest shows on the planet and a great platform for anyone who wants to sell records. It is a big blow to the final."
Reportedly Simon has called in for some big names – who could you get to replace Madonna though?
The source told The Sun: “They have already got some absolutely brilliant names in One Direction, Sam Smith and Ed Sheeran.
"They are three of the biggest stars of the moment but none of them has that real wow factor and One Direction are already so closely associated with The X Factor.
"A huge American star such as Beyonce, Christina Aguilera or Rihanna makes people's jaws drop to the floor."
Madonna’s representatives and The X Factor have yet to comment.
Will the show’s finalists Fleur East, Ben Haenow and Andrea Faustini be joined by the boy band One Direction, Ed Sheeran and/or Sam Smith?