I think the weight needs to be on the outside edge 🤔 but I could try washers and see. I have another small spindle with a stone wheel and a similarly small diameter. The problem with it is that as soon as it stops spinning it goes backward with equal speed.
Discussion
a larger diameter whorl will make it spin for longer because it's easier for it to keep spinning. the problem is that the larger the whorl, the harder it is for it to be evenly weighted all the way around. a natural material like stone might not be perfectly weighted or perfectly centered to keep a good spin.
an overall heavier spindle either on the shaft or whorl will make it easier to spin thicker singles.