No it is just the fact that Telegram has the keys. So how can one be this naive, to think this keys will not be used to decrypt all chats?

It is always the same. There is just no good rason why to build on trust in things where no trust is needed. Use e2ee services that do not log and save metadata. There is not that much they could share even if they comly fully.