It uses the same "double ratchet" encryption protocol, but with its own implementation. Provided Keychat runs a competent implementation, which I believe they do, it should be as secure as Signal/Molly Messenger. But the encryption protocol isn't the only component you have to consider when judging the security of a messaging platform. How the protocol is implemented is equally as important.

I've used Keychat a bunch of times and I'm really impressed with what they've created. They've created a competent and reliable system with verifiable security. Unfortunately, it seems as though it hasn't been widely adopted as most nostr users seem content to continue using NIP-04 DMs. NIP-17 adoption is apparently a distant 2nd place. As a security advocate, I wish more people would incorporate Keychat into their regular nostr usage. We all benefit when more people use strong communication encryption.