The 61-year-old singer and Perez, 36, came to blows last week which resulted in Alexander using a homophobic term to describe him.
Alexander and the other housemates claimed Perez was taunting and manipulating him, but he was given a formal warning.
After another argument with the gossip blogger, Alexander made the decision to leave the house. It’s since been reported he left because he was worried he’d hit Perez.
“It was getting to the stage where Alex was going to react. He’s in his 60s now and can’t be on telly hitting someone, no matter how much they deserve it,” a source told The Sun.
Speaking to Michelle Viasge and Katie Hopkins before his exit, Alexander apologised for using the slur in the argument.
He said: “I apologise to the gay community. I have gay cousins, gay friends, I have people in my family that I love with a passion, if anybody hit them or messed with them in the wrong way it’s on. God don’t make no mistakes.
“For that I apologise, but I never apologise for what I said about him to him ever. But I apologise for that statement if someone took it like I’m…”
Following Alexander’s exit, furious fans took to Twitter to brand Perez a bully.
One wrote: “Perez Hilton is the most ridiculous housemate that there EVER has been. Ever. He’s a bully and mentally deranged.”