That definitely wouldn't be a useful model of "contact" in the sense of "entry in an address book".

A contact should simply be a record of a public key and an identity, perhaps with your signature attesting the extent to which you believe the key is associated with that identity. This would provide a practically useful system similar to PGP's model. Whether that record remains local, or is published/broadcast as in PGP's public web-of-trust model (which is now largely considered societally broken), is a choice for the person that creates/holds that record.