Sorry if I'm going about this too much, but there is even more to say because this is really quite complex. A big reason for the whole notion of the secular Jew came out of racism.

They didn't call it being "anti-Jew", they called it "antisemitism." Even though technically that is overly broad (muslims are semites too), it was a close enough distinction because the other non-Jewish semites weren't really present in Europe where this was going on.

So if your religious family and friends insist you are still a Jew, and if the Nazis and racists also insist that you are a Jew and you get the downsides of that, what does it matter if you yourself don't believe the religion? In fact, it makes more sense to band together as a team against this hatred and take some of the upsides.

This history didn't happen to Seventh-Day Adventists. But I suppose if people hated me for being SDA, I might feel a closer affinity to the SDA people as we would share the injustice.