This is an interesting point of view, and we already explored it when WoT entered the scene: if you follow someone that you don't trust, with the current WoT implementations you validate him anyway, since he is present in your social graph. This is a problem, that probably lists could easily solve.

And if this is the path, "contacts" could effectively have a (minimal) trust meaning.