Took an neurophysiology class specializing in your special senses in college. Gustatory and olfactory senses are much more primitive in their gating mechanism relative to hearing and sight, which are significantly more complicated.

The ability for olfaction to ring so strongly in your memory is potentially related to how old that neural fabric is within our psyche.