One reason why I think the Druids were vilified aside from Cearas' reasonable insight, is that they were often likened to the Pythagorean Cult, who was a pretty large cult in his time. We all think of Pythagoras as being just some guy who had a good formula to figure out an angle, but he was pretty hardcore. I heard a story where he used a man as fish bait for not agreeing with the word of the cult.

I think in Caesar's time, they weren't looked upon as geniuses like we give them credit for. The Romans weren't particularly concerned with numbers, aside from building and warfare, and possibly thought of the Pythagorean cult as a mob, or a bunch of bullies. Basically, you will listen to my word, or you will die.

I need to go back and read my Livy to be sure, but if I remember correctly from the first time, Livy did not speak too highly of the Pythagoreans. It might have been an earlier stereotype, because of similar religious beliefs, that the Romans vilified the Druids.

Of course, Cearas makes the best points I have seen, I just thought I would add a little fuel to the fire.