The ledger is and must be a record of every transaction. The "wallets" doing the transactions must be recorded in the ledger. But those "wallets" are simply a long number. The privacy concerns only come in when somebody can connect a specific long number to a specific person/entity.
If you've bought Bitcoin on an exchange and given them all of your personal information now you have a trail. Breaking that trail will require some extra steps and cost.
If you only ever trade in Bitcoin anonymously then there is no trail.