It is fine, our teacher even gave us some to play around. It's just a stone.

The fibre is the dangerous thing.

So when the OP talks about glass fibre I think it's the 'glass-wool'-type fibre, not fibre optics.

So, asbestos is fine, like glass is fine.

Just don't inhale tiny strings made out of it.