Accident caused by car that drove off untouched

I recently had an accident caused by a car pulling out in front of a third party causing that car to swerve into my lane. I braked and came to a halt, the third party looked as though he was about to crash head on into me but just managed to get back to his side of the road, but our door mirrors were smashed. He then stopped further down the road behind me. The car that had pulled out wasn't hit by the third party and drove off.
The third party insurance company is now saying that because their driver had to swerve to avoid the other car, it's not his fault had they are refusing to pay for the accident. As a result, my insurance have paid for the damage to be repaired, I've had to pay the excess and have lost my no claims bonus.
What I have trouble understanding is how can I suffer a loss when I haven't done anything? It would have been a lot easier if the third party had crashed into the car that pulled out as I wouldn't have been hit at all.
Anyone able to shed any light on how this is normally dealt with?