Imparting "Knowledge" with LoRa / QLoRa has been challenging IME, unless you have *highly* structured data like Q&A with all of the right prompt template tokens for the given model (e.g. https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_1/)

The effect that LoRa training has is kind of noisy, which is why people say that they like them for image models where the results they're looking for are 'thematic' instead of structural or 'domain knowledge', but they are just as effective at imparting thematic/stylistic 'color' on LLM's in my (limited) experience.