Yes, I have a terrible rote/short term memory. It makes names hard to remember. Information needs to be connected to other information for it to stick easily.

All this is to say, I don't read much fiction but I had the same problem with Atlas shrugged and Lord of the rings.

Just keep at it. You might lose some connections, but it'll get easier as you read more or reread in the future. You'll still gain a lot :)