Plastic is cheap but bad for your health. Even if you store it correctly, you should still rotate them every 6-12 months.

Glass is safe, but won't protect water from light, and is heavier very fragile.

Metal(stainless steel) is the best because it's safe, durable and allows you to boil the water, but it's very expensive.