When a crawler requests a page that is managed by an RSL license from your website, it must include a valid RSL License Token for the page in the HTTP header using the new proposed License RFC 7235 HTTP Authentication scheme.’
https://rslstandard.org/guide/web-crawlers

