Why?

In parts of Tamilnadu, Buddhism & Jainism flourished and in fact some key Tamil literature (Silappatikaram and Manimekalai) have significant Buddhist elements in them. In Manimekalai the crux of the story is around the protagonist's pursuit of Buddhism, eschewing the family life or the courtesan life which were the main options available for women in the era.

But the decline had more to do with the patronage of the kings and hence places of worship more going towards Bhakti movement & other religions.