Can't do TOTP if the device doesn't know what time it is! There's no Raspi Zero clock battery so it's the beginning of time every time you power it up.
Discussion
So another idea could be preparing multiple cheap SD cards and each one is hidden in a 3D printed seal that nobody knows contains the SD card. If you don’t use it for longer period you move on to the next sealed SD card.