At some point you just need to put in a limit for the color change, maybe max it with a relative measure so its always 30% darker/lighter than whatever the text darkness is (or find some specific minimal difference so the text is always readable)

After like 10 threads deep its hard to come up with any way that it isn't at least a little confusing, lol. At that point maybe we just give up 🤣