Some time ago I wrote about how to create LoRa, but is there any working LoRa for SD 3 at this point?
While I’m sure there are people who has figured out how to create LoRa as well as fine-tuning for Stable Diffusion 3, I haven’t really seen anything official about it. Except for at one place, Swarm UI.
From the information given in the text, it seems that it’s most likely style LoRa you are supposed to be able to create with this. I did some testing to find out
- How to create SD 3 LoRa
- How to use them
- What effect they have (if any)
Attempt to create a working LoRa for SD 3
The LoRa extractor is located in Swarm UI ==> Utilities ==> LoRa Extractor
From the instructions you can read that the LoRa extracts the differences between 2 models, but not really which differences. It also says that the models should have a common base model (i.e. SDXL), so why did I think it was possible to extract SD 3 LoRa with this?
Up until just a few days the Swarm UI was managed by Stability AI, the company behind Stable Diffusion 3, so I thought it was worth a shot at least.
And while trying various combinations it was clear that if the models are incompatible (for example PixArt and SDXL), the extraction will simply stop and give an error message.
I created several LoRas both with SD 3 as base and SD 3 as “other”, with the other model being different SDXL models. LoRas were created using most combinations.
The Loras are used just like any other lora <lora:NameOfLora:1.0> in your text prompt.
The leftmost image is Stable Diffusion 3 with no LoRa, the middle one is Stable Diffusion 3 using a LoRa I created from NightVisionXL and the image to the right is made with NightVisionXL model. It is pretty clear that the LoRa has little to no effect.
Another testing show similar results, none. The leftmost image is created with SD 3 and no LoRa, middle is created with SD 3 and LoRa created from Realism Engine SDXL, and the image to the right is created with the Realism Engine SDXL model.
Unfortunately it seems there’s no readily available way to create LoRa for Stable Diffusion 3 at this moment. I’m still curious to see how SD 3 will continue to evolve, even though it seems to struggle quite a bit right now.
PixArt may or may not overtake Stable Diffusions domination as the go-to Open Source generative AI, but if SD 3 recovers I see no wrong in making use of them both.