Being very non-religious I follow several Atheist/Agnostic pages on social media. Very often an anti-Islam thing comes up and sparks a debate. While most irreligious people are left-wingers, there are quite a few right-wing atheists and agnostics just like there are left-wing devout Christians who are far outnumbered by their conservative peers. They often mock left-wingers as we are "soft" on Islam and only focus on "attacking" Christianity.
But the whole concept of not believing in religion is not agreeing on any of the religions. I don't technically hate Christians despite thinking The Bible is nothing but a book of fables. I actually would have a lot of respect for more Christians if most of them embraced a Jimmy Carter-like Christianity. Carter is not a bigoted man, but regardless, he is a very moral person who considers himself a devout Christian. Jimmy Carter helps the poor and unfortunate even in his elderly age for no benefit of his own. But when Christians become racist towards minorities, bigoted towards the LGBT community, hateful to people other viewpoints (including in their own faith), and try to use the government and public schools to endorse and force people to be Christians, well...I just have a few issues that makes me very angry and ready to counter verbally without much sympathy for their feelings.
But I think part of the reason why most left-wingers are "soft" on Islam, is that we also see it the whole issue more being on race than religion in general. Granted Islam is not a race, but most of the time when we think of Muslims we picture someone from the Middle East as the the vast majority of them are Muslims and that's where Islam originated and grew. Right-wing Christians get awkwardly "tolerant" when someone like Trump gets diplomatic with Saudi Arabia, but when most right-wing Christians have no idea what the Quran is even about (they often refuse they worship the same god), it becomes more clear it's really more about xenophobia than religion.
That doesn't mean I don't have criticism of the religion of Islam. For one, most of the stories are straight from The Bible (seriously) so it's not exactly a shocker when you hear some pretty horrid stuff in the Quran. But the way many women, the LGBT community, and non-Muslims are treated in some Islamic-majority countries is despicable.
But that's in other countries, many of which don't have many people who would understand what I'm saying since I only speak in English. In America, I've never had a Muslim try to convert me, nor I have seen a Muslim try to preach their religion in government/school. Nor do any followers of Islam come knocking on my door with recruitment pamphlets. They just leave me be. I rarely ever speak out against Islam for the same reason I'm "silent" on Buddhism, Hinduism, and all other religions, because none of them have affected my personal life, so far anyway.