Why people constantly complain that they don't understand pointers?

Pointer is just an index in a global array of bytes that we call "memory". What's the problem?

Pointers are just variables that store memory addresses. And then you dereference the pointer to get the value stored at that memory address. I am not sure what's complicated here.