Most of the causalities killed in the crossfire were women as clashes broke out earlier this week. A large area of Sirte, the hometown of former leader Muammar Gaddafi, fell to IS earlier this year.
Islamic State (IS) is one of among several groups battling for the control of Libya and has made advances across the country.
Fighting broke out in a residential area of Sirte on Monday between Islamic State and a Salafist Muslim group backed by angry armed civilians.
It followed the death of a Muslim cleric who - along with others in the neighbourhood - had refused to comply with IS's order for residents to pledge allegiance or face death, a local journalist told the Associated Press. It is reported that most of the 25 people who had been killed were armed civilians who had joined the fighting.
One report said IS had taken back the neighbourhood, although this has not been confirmed.
IS faced a similar battle in the eastern port city of Derna in June - the first city outside Iraq and Syria to fall to the group - and was pushed back by an al-Qaeda affiliated group.
Since Col Gaddafi's death in Sirte in October 2011, Libya has descended into chaos, with various militias fighting for power. The internationally-recognised government has been forced to leave Tripoli.
In the past year,IS has set up checkpoints and established a presence in cities across Libya.
In January, an IS-affiliated group bombed the Corinthia Hotel in Tripoli, killing 10 people.
In February and April, videos were released showing IS militants in Libya killing two groups of mainly Egyptian and Ethiopian Christians.
Air strikes against IS positions have been conducted by the Egyptian air force and by Libya's internationally-recognised government.
