They were getting hashes derived by the device unlock credential. They aren't able to get the keys themselves because of how Pixels use a secure element, and because Android uses filesystem-based encryption rather than the standard full-disk encryption older Android or Desktop solutions like Windows or VeraCrypt would have. There is an in-depth explanation on how disk encryption works on Android and GrapheneOS here: https://grapheneos.org/faq#encryption

Putting the device in AFU and unlocking the device would leave hashes derived from the lock method credentials in memory. They could extract the hash from a memory dump and then make a tool to brute force the hash to attempt to figure out the unlock credential. When a device is powered off or in before first-unlock state then there is nothing in the flash storage or memory.

This issue that made the stock OS get affected by the attack was they do not clear memory when it is freed and that Fastboot could be exploited to dump the memory to another device. The fix Google provided is zeroing memory when switching into Fastboot mode. The actors doing this exploit discussed having no brute-force capability for GrapheneOS at the time in their product changelogs, likely because we sanitize memory on free already. That doesn't mean they could not have figured out another way though, due to the nature of the device being in-use and unlocked

Users should treat AFU smartphones like they'd treat a powered on, unlocked laptop as the data isn't at rest. That's why we've had features like automatic reboot to put data at rest after a set time and USB port controls to disable the data lines (or the port entirely) to make exploitation of devices in that state difficult.

Tools to brute force credentials for encrypted operating systems from memory dumps are trivial in that market, Passware is an example of a product. There are hardware-backed protections against brute forcing, but because of they dump the memory to a PC they aren't touching the phone at all so they could avoid that entirely.

The affair behind this have been quite big for us and it possibly deserves a longer run-down or article in the far future by someone.