We're gonna have to disagree out the gate. A manifold has a precise definition as a surface which is locally Reimanian, something like that. I'd classify it as one of the more elaborate constructs, and less primitive or whatever than a simple set. Afterall, you need set elements to talk about before you can define a manifold. All the formal stuff I've seen looks like "a group is a SET, with the following additional properties...".

Reading the rest I do see how you mean though, so I'll stop getting hung up on technical definitions. Yeah, I'd think patterns or differences come first, then the notion of distinct things. That's how I'm roughly understanding you.