Want to see your favorite anime character come to life? Or maybe transform yourself into a stunning anime creation? With the power of AI image generation, it’s now easier than ever! In this post, I’ll show you how to use ComfyUI, a powerful and flexible tool, to play around with both real photos and anime images, creating amazing transformations.
What We’ll Cover
- Transforming Anime Characters into Realistic Photos
- Turning Your Photos into Unique Anime Art
- A Step-by-Step Guide Using ComfyUI
- Troubleshooting Common Issues
What is ComfyUI?
ComfyUI is a node-based interface for Stable Diffusion, a leading AI image generation technology. Think of it as a visual way to build custom image creation pipelines. Instead of writing code, you connect “nodes” together to define how your image is processed. It’s incredibly powerful and customizable, allowing you to achieve amazing results with a bit of setup.
What You’ll Need
Before we dive in, here’s what you’ll need to get started:
- ComfyUI: The core software. We’ll provide instructions for installation.
- Gemini 2.0 Flash node: A ComfyUI custom node.
- ComfyUI manager: Makes installing the custom node easier.
- Specific AI Models: These are the brains behind the operation, allowing ComfyUI to understand and generate images. Don’t worry; we’ll provide links to download them. We need:
- Flux.1-dev.safetensors or a smaller alternative flux.1-dev-fp8.safetensors
- ae.safetensors: A VAE (Variational Autoencoder) for better image quality.
- flux1-depth-dev-lora.safetensors
- depth_anything_v2_vitl.pth
- Flux T-5 xxl textencoder
- clip_l.safetensors
- sam_vit_b_01ec64.pth
- face_yolov8m.pt
- Patience and a little bit of tinkering!
Step-by-Step Guide: Anime to Photo with ComfyUI
This section will guide you through the process of setting up ComfyUI and using a pre-made workflow to achieve the Anime to Photo or Photo to Anime transformation.
Prerequisites
- Install ComfyUI (if you are a beginner, you can install ComfyUI desktop app)
- Install Gemini 2.0 Flash node
- Install ComfyUI manager
- Download Necessary Models:
- Download the Anime To Photo workflow from here or or here
- Download Flux.1-dev.safetensors or the smaller flux.1-dev-fp8.safetensors.
- Download the Flux VAE (ae.safetensors).
- Download flux1-depth-dev-lora.safetensors.
- Download depth_anything_v2_vitl.pth.
- Download Flux T-5 xxl textencoder.
- Download clip_l.safetensors.
- Download sam_vit_b_01ec64.pth.
- Download face_yolov8m.pt.
Organizing Your Models
To ensure ComfyUI can find the models, place them in the correct folders:
- Flux.1-dev.safetensors (or Flux.1-dev-fp8.safetensors) goes in: ComfyUI\models\diffusion_models\
- ae.safetensors goes in: ComfyUI\models\VAE\
- flux1-depth-dev-lora.safetensors goes in: ComfyUI\models\Loras\
- depth_anything_v2_vitl.pth goes in: ComfyUI\models\depthanything\
- t5xxl_fp16.safetensors and clip_l.safetensors go in: ComfyUI\models\clip\
- sam_vit_b_01ec64.pth goes in: ComfyUI\models\sams\
- face_yolov8m.pt goes in: ComfyUI\models\opencv\bbox\
Loading the Workflow
The first time you start up ComfyUI, it will open a sample workflow in your web browser. To use our Anime to Photo workflow:
- In the top left corner, click “New” or “Clear”. This will give you an empty canvas.
- Drag and drop the downloaded Flux Anime To Photo.json file into the ComfyUI window.
- You might see a warning about missing nodes. This is normal!
Installing Missing Custom Nodes
To fix the missing node errors:
- Go to the “Manager” tab at the top of the ComfyUI interface.
- Click “Install Missing Custom Nodes.”
- ComfyUI will automatically download and install the required nodes.
- Restart ComfyUI.
Now the workflow should load correctly.
Configuring the Workflow
- Now that the workflow is loaded, let’s make sure everything is set up correctly.
- Ensure that Flux.1-dev.safetensors is selected in the Load Diffusion Model.
- Ensure that flux1-depth-dev-lora.safetensors is selected in the Lora Loader Model Only node.
- Ensure that ae.safetensors is loaded in the Load VAE node.
- Make sure that your two CLIP models are in the correct section as well.
- Depth Anything V2: Check the “Depth Anything V2” node and make sure that depth_anything_v2_vitl.pth is loaded.
- Samplers: Verify that face_yolov8m.pt and sam_vit_b_01ec64.pth are loaded in their respective nodes.
Running the Workflow
- Queue: Click the “Queue” button at the bottom of the ComfyUI interface.
- ComfyUI will now start generating the image. The time it takes will depend on your system’s hardware.
- View Results: Once the process is complete, the generated image will appear in the ComfyUI interface.
How to Turn Your Photos into Anime Characters
Now, let’s try turning your own photos into anime! This involves a slight modification to the workflow.
- Edit the System Prompt: In the “Gemini system Prompt” node (find it by searching in ComfyUI or tracing from the T5 Node), you’ll see a text box. Replace the existing text with a new prompt.
- System Prompt: “You are an AI assistant specializing in crafting professional and effective prompts for the Flux model, suitable for the t5-xxl clip. You are specialized in creating prompts for generating anime characters from realistic looking images. When an image is provided, you should generate a concise and descriptive prompt that will create an anime character based of the traits of the realistic photo that is provided. The prompt should be between 150-300 tokens. The output should only show the final prompt, without any additional comments or instructions.”
2. Upload your own photo: In the workflow, you can replace the reference image of the anime and add your own photo of you, and then run the workflow.
3. Queue: Click the “Queue” button to start the generation process.
Then upload the image you want to turn into an anime image and click on queue.
Troubleshooting
- “Not Enough Memory” Error: Try reducing the image size in the workflow or closing other applications to free up memory.
- “Missing Nodes” Error: Double-check that you have installed all the required custom nodes using the ComfyUI Manager.
- Workflow Not Loading: Ensure that you have placed all the model files in the correct folders.
- Unexpected Results: Experiment with different prompts and settings to fine-tune the image generation process. The results depend on the inputs.
Call to Action
Now it’s your turn! Experiment with different anime styles, character traits, and personal photos to create unique and amazing AI-generated art. Share your creations and tag me on Instagram or TikTok !
Glossary
- VAE (Variational Autoencoder): A type of neural network that helps improve the quality and detail of generated images.
- CLIP (Contrastive Language-Image Pre-training): A model that understands the relationship between images and text, helping the AI generate images that match your prompts.
- Lora: LoRA, which stands for Low-Rank Adaptation, is a technique used in machine learning to reduce the number of trainable parameters when fine-tuning large pre-trained models