This would lead us to believe that said artists images were used in the dataset or not as there is no information about this released by the Black Forest Lab team.

I built a ComfyUI workflow that allowed me to read my formatted prompts from a txt file and run through them one after another, this alone was a fun experiment to build this workflow as nothing like this existed and running a batch that cycles through is not one of the strengths of ComfyUI.

prompt: art by <artist_name> resolution: 1024x1024 px seed: 88888888 image batch: 4

Below is a preview of the study (50 images), because like before with SDXL and Midjourney 4000+ image gallery really strains my web-servers. So if you are interested in grabbing the full study you can get the offline copy.

What you get in Offline version

• 2x ZIP files containing all the files
• 1x PDF overview document with observations

I hope you find our Flux Dev Artist Study beneficial in your experimentation and will find share this resource with others. Let me know if you have any comments or questions below.

Let’s explore the list of LoRAs available, their samples and of course the download link.

In order to install these LoRA files (.safetensors), simply copy them in the models\loras folder of ComfyUI or models\lora folder of Automatic1111.

Make sure you review the usage right for each of the LoRAs provided by the author.

Realism Flux LoRA by XLabs-AI

Adds more realism to your final image with more detail that by default. Finer details like hair, eyes, skin textures are greatly improved with this LoRA by XLabs. It was converted for use with ComfyUI by comfyannonymous. There are no trigger words for this LoRA.

Download Realism Flux Dev LoRA

Antiblur LoRA by Shakker Labs

This LoRA has the ability increase the depth of field controlled by its strength parameter. By default photographic results have very shallow depth of field straight out of Flux Dev. This LoRA increases the depth so you have more details in the background. There is no trigger word and the below samples were at Strength 1.0

Download Antiblur LoRA

Half Illustration by Davisbro

Really cool photo and illustration combined generation. If you have clicked on this post’s thumbnail or some of my YT videos lately then you have seen what this Flux Dev LoRA can do. Trigger word in the style of TOK

Download Half illustration

Frosting Lane by Araminta

A soft pastel illustration style LoRA for Flux Dev by our featured Araminta. She has been putting out some really cool LoRAs for Flux. Keep exploring this post. Trigger word frstingln illustration

Download Frosting Lane

Flux Film Foto by Araminta

35mm Film style images that take the overcooked photographic outputs of Flux Dev to a whole new realism. Trigger word flmft photo style

Download Flux Film Foto

Soft Serve Anime by Araminta

Soft colour anime style images are produced when you use this LoRA. It combines the power of Flux and hones in on a beautiful anime style. Trigger word sftsrv style illustration

Download Soft Serve Anime

Linnea Beta by Araminta

Trained on her own hand drawn sketches, this cute little character emerges with simple hand drawn lines and simple colour scheme. Trigger word linnea teal hair

Download Linnea Beta

Mooniverse by Araminta

Another photographic LoRA but with very nice soft aesthetics (soft focus) and muted colours which produce some very nice results out of Flux Dev.

Download Mooniverse

Sonny Anime Flex by Araminta

create soft gel line anime style images in a sort of cute lofi aesthetic. This is the FLEX version, that has more flexibility and can be pretty successfully pushed away from true anime towards cartoon art and more complex scenes. However, it is recommended to include the style direction in your trigger. Trigger word nm22 [style direction] style

Example prompt: [subject], [colors/aesthetic description], nm22 [style direction] style

Download Soony Anime Flex

Koda Diffusion by Araminta

Koda captures the nostalgic essence of early 1990s photography, evoking memories of disposable cameras and carefree travels. It specializes in creating images with a distinct vintage quality, characterized by slightly washed-out colors, soft focus, and the occasional light leak or film grain. Trigger word flmft style

Download Koda Diffusion 

Midsummer Blues by Jake Dahn

A minmalistic yet colourful illustration style LoRA for Flux Dev that give you a very beautiful vibrant illustrated result, unlike other models with Flux Dev you get the correct number of figures and better compositions. Trigger word is MSMRB but as per the download page it works better when used like illustrated MSMRB style
Download Midsummer Blues

These are some of the coolest looking Flux Dev LoRAs that have been found on the internet so I hope you enjoy using them. They small enough that you can download them all and achieve wonderful results with Flux Dev.

If you have others to suggest, leave the details in the omments.

I’m very happy to share my LoRA Caption workflow for ComfyUI that will let you run a batch of images through two different captioning methods using Florence 2 model or using Clip Interrogator. Both these are paired with WD14 Tagger node which generates some additional tags/keywords for the caption.

Images must be PNG format, JPEGs are not supported currently in the workflow due to custom node.

You can enter the LoRA training “trigger” word which is added to the prompt.

Useful tips

• Caption files generated cannot be overwritten – this a limitation of the custom node, if you want to re-run delete the original TXT file
• Once all images are run through you need to Reset the counter. Use the Reset counter (use once) switch – set to true. Remember to turn if off – set to false.
• List index out of range error – this means you are trying to run it but the txt file caption already exists. Or the counter has reached its limit, you need to reset it.
• I always recommend that you review the captions and finetune them to ensure you get the best result out of your LoRA.

In this post I will share my workflow of this Flux Dev (Image to prompt) with Upscaler workflow. Over the past years I’ve created so many images and with the advancement of tools like IPAdapters we hardly need to write a prompt to generate similar images. So as we don’t have anything like IPAdapters at the moment, I created this workflow which can generate the prompt to feed to the Flux model.

The workflow is running fine with 16GB and uses the Flux Dev Fp8 so it will work with 12GB as well.

Try to play with Task setting in Florence2Run node, i’ve set it to more_detailed_caption but you can scale it back for shorter prompt. Other settings to try would be caption or detailed_caption.

Must Do

In order to make sure these new workflows work for you you must do the following:

• Update ComfyUI so you have the latest version
• Install any Missing nodes using ComfyUI Manager

Workflow

The workflow is provided as PNG and JSON file so you can use either to import the workflow into ComfyUI. Simply drag and drop either of these files on your ComfyUI and you will be up and running in no time, make sure you have completed the Must Do above.

The workflow is easy to follow and comes with some notes to help you understand how to control some settings. The upscaler is set to 2x but works at 4x as well, just takes longer. I’m also embedding the realism LoRA of Flux Dev so you can generate realistic details in the final image.

Of course if you have any questions or comments don’t hesitate to share it below.

My solution for this VRAM limitation (as I run RTX4080 16GB on my local PC) is to use Runpod and use their A40 GPU which only costs $0.35 per hour which is half a cent per minute. I don’t even know if the electricity used by GPU is costing them ore than that or not. It is ridiculously cheap and readily available on their platform.

I am using the ComfyUI Image for Runpod, which provides latest ComfyUI version with standard SD and SDXL models.

You can open the various applications via the links provided.

To download the models I created a JupyterLab Notebook with all the links so you can download all the models, for Flux Dev you do need to use Huggingface access Token which is free so just need to enter it in the notebook before you run it. The way I have built the notebook is that it will download the Models, Clip and Realism LoRA in the required ComfyUI directories.

Simply upload the notebook I provided (IPYNB) in the Workspace folder on Runpod and double click to open. Once launched you can click on the play button for each of the cells with the links (after updating the access token).

Once this is done you can simply import the provided Workflow in ComfyUI and you are good to go. By default the workflow I use utilises all the higher settings of Flux Dev because I built it for RunPod where you will be using higher VRAM GPU. You can modify them as you like.

Add New

The workflow I provided uses a few custom nodes so if you end up with red boxes upon loading the workflow just use the ComfyUI Manager to install missing nodes.

I also have a video walkthrough of my setup of Runpod on my YouTube which is below so feel free to browse through it if you like video format better.

I hope you find this information useful and helpful. If you have any queries or suggestions leave them in the comment section.

A Motion LoRA would hook onto the AnimateDiff Loader in the motion_lora input and is loaded using Load AnimateDiff LoRA node.

The Motion LoRA provide a very nice movement in your animation and generally work really when when a strength of 0.7-1.0 is used. They are designed with v2 version of AnimateDiff so they won’t work with v3 version and were trained with a context window of 16 so ideally suited for Stable Diffusion 1.5 models.

Download

You can grab the various motion LoRAs via the links below. These are hosted by the createor on their huggingface page, links are provided for ease of download below.

• Zooming In
• Zooming In (32 frame)
• Zooming Out
• Pan/Lateral Right
• Pan/Lateral Left
• Crane Up
• Crane Down

Download the basic workflow that will let you test these above motion LoRA in AnimateDiff. You will need to download some custom nodes which you can easily do via ComfyUI Manager.

I also put together a video preview and walk through of my workflows that showcase these basic camera motion LoRA for AnimateDiff. In the video I show a bigger much larger workflow which can be built easily by expanding the above workflow and copying the necessary nodes to review multiple motion LoRAs togther.

Hope you found this post useful and thanks for stopping by. Check out some of the other related posts below and if you can support our blog.

First let’s talk about the Bad

Training AnimateDiff Motion LoRAs when I researched was linked to this repository which is called AnimateDiff-MotionDirector. The repo is about a year old with last commit that was done about 4 months ago (from date of this post). The installation steps are pretty similar to any diffusion based app…do a git clone, create a new env (conda create) and the pip install requirements.txt.

Well that’s where it all turns to SHIT!!

The installation does not run correctly, there are numerous errors and the repo requires a whole 40GB clone of https://huggingface.co/runwayml/stable-diffusion-v1-5 which is never ending. It doesn’t specify if you need .BIN files or .SAFETENSOR so you are already doubling the download and time to download because the repo contains both version of the one file.

So basically after burning some 3-4 hours on Runpod and a day trying locally on my computer, I gave up this repo and method.

The Good and Very Easy

I start to search online and find this wonderful video. After watching about 2 mins I literally could kick myself. The process couldn’t be easier and its all inside ComfyUI which I’ve been running successfully.

The video provides a link to the workflow in its description which gives you a ComfyUI workflow to download. You need to import that into your ComfyUI Install and install any Missing node using ComfyUI Manager

Watch this 3 min quick video on how to fix any missing nodes.

The workflow looks like this, its actually very easy to use and the defaults work pretty damn well.

You need these below model and LoRA:

• v3_sd15_mm.ckpt – Download and move to your \models\animatediff_models folder.
• v3_sd15_adapter.ckpt – Download and move to your \models\lora folder

Make sure you “Refresh” your workflow so the models are now seen by ComfyUI, which you will select next.

Training your Motion LoRA

You need a video file that you will be training the Motion LoRA on so upload that to the Load Video node.

Make sure in the model selections you select the model names as highlighted in the ADMD_AdditionalModelSelect node. In the ADMD_CheckpointLoader you can select your choice of SD1.5 model.

Provide a prompt that describes what your video is about (shown below in red) and also provide a prompt for AnimateDiff to generate a resulting video (shown below in green) with the Motion LoRA. Finally give your Motion LoRA a name eg. morpheousC as demonstrated by the orange arrow.

Queue up your workflow and let ComfyUI do its thing. The default workflow is able designed to train on 16 frame of images but on my computer I was able to push it to 24 frames and when pushed further to 32 frames my GPU ran out of memory (Cuda out of memory). The training time are as below on my specs of my local computer.

• 16 frame training – 20 minutes on RTX4080
• 24 frame training – 25 minutes on RTX4080

When all the execution is done you will find the Motion LoRA in the folder \models\animatediff_motion_lora under folder created with <Date>\<Time>\<Lora_name>. Here you will find the .SAFETENSORS files each created at 50 steps apart.

The workflow will also produce a comparison video which you can use to review and decide which Motion LoRA you are going to use and keep. With the default settings of the workflow your LoRA file should be 128MB each.

Thanks to this method I’m able to generate Motion LoRA for AnimateDiff within minutes and the results are super good. Check out my socials (blog and personal) for some sample results as well as my store if you want to get my Motion LoRAs.

Download the LoRA individually to run locally on your computer or explore and play with them online via Araminta’s Portfolio page.

At the time of publishing this post there were 11 13 LoRAs available that range from soft tones, water color illustration, pop art, sketch art and anime manga style. These LoRAs are designed for use with standard base SDXL model.

• Araminta K Illustration
• Japanese Poster Prints
• Abstract Anime Painting
• Gemini Anime
• Pop Art Anime
• Soft Ones
• Soft Pastel Anime
• Painting Light
• V. Sketch Manga
• Lost and Lonely Anime
• Sketched Out Manga
• Haute Couture Illustration (added 11th June)
• B&W Manga Block (added 11th June)

Exploring the different LoRAs the best way to do this is to lock the Seed and use the same prompt, the only thing that changes is the LoRA. You end up with a collection of images that you can compare easily to get a feel for how the result is influenced by the LoRA.

Araminta uses the keyword/trigger word “daiton style” to invoke the LoRA however I found that even without the trigger word the generation is influenced heavily by the LoRA especially if you are using the Base SDXL model

prompt: happy, a cute girl with black strange hair, red sweater, staring into the camera

seed: 3474888480

model: Base SDXL

Above grid of images showcases some of the aesthetics of the images you can expect to see however they can also be highly influenced by the prompt you use. The last image in the grid is the base SDXL render without any of the LoRA.

I used a specifically designed workflow in ComfyUI that allowed me to test these LoRA which I am going to release tomorrow so you can download it and have fun testing these and other LoRAs.

Of course to get really creative you can combine one or more LoRAs in your workflow and adjust the strength of each to blend the aesthetics together.

Here is a gallery of the images that was able to create using the Base SDXL model and the LoRAs shared by Araminta. Make sure you follow us for more updates and also the artist Araminta.

These are:

• Canny – my favourite
• Openpose
• Scribble

I suggest you rename them after downloading so you know which is which as all of them are exactly the same size 2.5GB and have the same name, so it would be tricky to identify them.

I started with Canny and was easily able to test this in ComfyUI by modifying the standard workflow quite easily. The results with Canny were very nice and with the high quality model like SDXL you can get some high quality results already.

This is using a basic workflow and without any upscaling at this stage. This is only start of the exploration with SDXL Controlnet and I’m sure many of you are going to do some really cool stuff with this.

The author has trained this Canny model on more than 10 Million images as stated in the readme which is commendable as this would take a lot of GPU power and time.

I know many of the people have been asking for these, if they would be released and when they would be released but its exciting to see that we have three ControlNet for SDXL and hopefully we will have more soon. I believe these are not yet available in Automatic1111 but I’m sure they will be added in the near future.

However the power of Steerable Motion is revealed in many forms with a generous collection of workflows available by the community. Each workflow explores new techniques and methods that are unexplored and creates some unique animations.

One of the key components of animation is using Animatediff Motion LoRA which are plugged into the different workflows to help guide the animation being generated. This post is focused on exploring the different Animatediff motion LoRAs and providing you a resource to download them yourself.

I use the below two sample images to illustrate the motion created by the different Animatediff motion LoRAs.

Warning: Some of the LoRAs are hosted on Civitai which is a great resource however it is NSFW (Not Safe for Work) ie. can contain 18+ content so please visit with care. The Motion LoRAs however work really well so no concern with their function, however their authors have chosen to host on Civitai.

Below clip showcases all the Motion LoRAs I experimented with in the single clip. Its a short clip so you can easily watch the whole thing or head over to YouTube to get the chapter links.

LiquidAF, Smoooth & WAS26

Trained by Peter O’Malley these three LoRA provide three motion LoRA for animatediff that provide this fluid and smooth motion to your animated images. You can download these models at Huggingface page.

Download

NeonAbstractsEffects, NeonLightsMotionLora & SubtleRotateML

Neon coloured effects Motion LoRA created by VK and available at Huggingface, these motion LoRAs bring lots of neon elements moving around in the image. Subtle Rotate produces a cool rotation effect in your animate, at times look three dimensional

Download

bubblingRings

Like bubbles moving up to the surface of the liquid this motion LoRA result is quite a fun one to play with. The bubbles move around on the surface before popping. Its one of my favourite ones to play with. You can enhance the result by using the text ‘bubbling rings’ in the prompt.

Download (Civitai)

explosiveAnimatediff

A cool looking motion LoRA that takes your image and creates explosive fountain like result in your animation. You can use the text ‘explosion sparks’ in the prompt to further emphasis the movement.

Download (Civitai)

liquidOilEmulsion

A very fluid motion like when water and oil movement that you might experience when these two different liquids mix. This motion LoRA produces a very fluid effect.

Download (Civitai)

shatterAnimatediff

Like particles shattering and scattering in all different directions, this motion LoRA provides a lot of kinetic energy to the animation.

Download (Civitai)

smokeMotionLoras 1100Steps & 3000Steps

Very cool smoke rising from the flame, this motion LoRA comes into two trained strengths, 1100 step version is milder while 3000 steps produces more exaggerated result.

Download (Civitai)

spinZoomLoop

Making elements of your spin and zooming effect is generated by this motion LoRA. In some cases it appears to be a 3D rotation based on the scene composed.

Download (Civitai)