The flag of St George will not be flying over Radstock any time soon after town councillors decided it was inappropriate.
The town council decided not to buy a St George's flag after a discussion about repairs to the civic flagpole to ensure a Union Flag could be flown on Armistice Day.
Among the concerns expressed were the offence that could be taken by Muslims.
Councillor Eleanor Jackson (Lab, Radstock), a university lecturer and teacher, said its use during the Crusades of the 11th, 12th and 13th centuries could mean the English national flag could be seen by some as offensive.
She added: "My big problem is that it is offensive to some Muslims but even more so that it has been hijacked by the far right. My thoughts are we ought to drop it for 20 years."
She suggested the Union flag was a more inclusive and suitable way of displaying national pride.
The council will also be flying the In Bloom flag at the Miners' Memorial Garden to celebrate the town's achievements in the gardening competition and the rainbow flag, the symbol of lesbian, gay, bisexual, and transgender pride movement, at "appropriate" times of the year.
There are also plans for a flag to be designed specially for Radstock.
Town council chairwoman councillor Lesley Mansell said the main reasons for not buying the flag were practical ones: "We do not have a Union Jack and the discussion was mainly about purchasing one to fly mainly for Armistice Day.
"The list we saw at council included a number of other flags which councils are allowed to display which includes those for the patron saints for England, Wales, Scotland and Ireland.
"The council confirmed its original decision to purchase a Union Jack, to fly the In Bloom one and the rainbow flag at appropriate times. We are working on a new logo and when that is sorted we hope to have our own flag for Radstock."
John Clements, vice-president of national patriotic group the Royal Society of St George, branded the decision "nonsense" and said "censoring" of the national flag played into the hands of the far right.
A spokeswoman for the Muslim Council of Britain, said it encouraged the flying of the St George's flag.
She said: "St George needs to take his rightful place as a national symbol of inclusivity rather than a symbol of hatred. St George actually lived before the birth of Islam and should not be associated with any hatred of Muslims."
In a further statement, the town council said reports that it had "banned" the flag were wide of the mark.
"Radstock Town Council is sorry about reports in the media and wishes to clarify that at no point has a decision been taken to lower the St George's flag. At our April meeting, the council was informed that it does not have a Union Jack and a vote was taken and agreed by all to purchase one.
"The council did not vote to ban the St George's flag nor any other flag.
"The decision taken was made in line with the resources and staffing that we have as a small town council.
"The mechanism on the flagpole is currently broken and we are seeking to have this repaired as soon as possible. The council is proud to fly the Union Jack as part of our heritage."