they were likely outposts of the civilization before the flood . the ancient legends down south often speak of tall white bearded gods who appeared and taught building and agriculture... they then ruled these new areas and peoples ... sometime later , pre or post-flood , they simply were absorbed into the local populations.
millennia later , when the spanish arrived , they were able to gain access and more than likely some reverence by virtue of these legends... the Spanish even wrote about and drew illustrations of the then contemporary Incan rulers having a lighter skin tone and greater height, i.e. remnants of the white blood from prehistory, than the masses they ruled...