Summarizing https://johanwind.github.io/2023/03/23/rwkv_overview.html

Here's my try:

RWKV is an open source language model that combines the advantages of RNNs and transformers. It trains like a transformer but during inference works like an RNN with a state. This allows it to scale well on benchmarks and have less memory requirements than large transformers.