You're getting perilously close to the insights of David Hume and his epistemology.