Doodle-to-Image Generation with Diffusion Models

This repository contains the implementation of a sketch-to-image generation pipeline developed as part of a Junior Independent Study project at The College of Wooster. The system uses Stable Cascade, ComfyUI, and ControlNet to convert rough user sketches into detailed, high-quality images through diffusion models.

Overview

The primary objective of this project is to enable interactive image generation based on simple line drawings. By integrating edge detection (via Canny), prompt conditioning, and latent diffusion techniques, the model allows users to guide both the structure and style of the generated image.

Features

• Sketch-to-image synthesis using a multi-stage diffusion pipeline
• Canny edge-based conditioning via ControlNet
• Modular, node-based workflow using ComfyUI
• Prompt encoding with CLIP
• Evaluation through SSIM and FID metrics
• Supports training and inference with pretrained models (Stable Cascade)

Shortcuts

Keybind	Explanation
Ctrl + Enter	Queue up current graph for generation
Ctrl + Shift + Enter	Queue up current graph as first for generation
Ctrl + Alt + Enter	Cancel current generation
Ctrl + Z/Ctrl + Y	Undo/Redo
Ctrl + S	Save workflow
Ctrl + O	Load workflow
Ctrl + A	Select all nodes
Alt + C	Collapse/uncollapse selected nodes
Ctrl + M	Mute/unmute selected nodes
Ctrl + B	Bypass selected nodes (acts like the node was removed from the graph and the wires reconnected through)
Delete/Backspace	Delete selected nodes
Ctrl + Backspace	Delete the current graph
Space	Move the canvas around when held and moving the cursor
Ctrl/Shift + Click	Add clicked node to selection
Ctrl + C/Ctrl + V	Copy and paste selected nodes (without maintaining connections to outputs of unselected nodes)
Ctrl + C/Ctrl + Shift + V	Copy and paste selected nodes (maintaining connections from outputs of unselected nodes to inputs of pasted nodes)
Shift + Drag	Move multiple selected nodes at the same time
Ctrl + D	Load default graph
Alt + +	Canvas Zoom in
Alt + -	Canvas Zoom out
Ctrl + Shift + LMB + Vertical drag	Canvas Zoom in/out
P	Pin/Unpin selected nodes
Ctrl + G	Group selected nodes
Q	Toggle visibility of the queue
H	Toggle visibility of history
R	Refresh graph
F	Show/Hide menu
.	Fit view to selection (Whole graph when nothing is selected)
Double-Click LMB	Open node quick search palette
Shift + Drag	Move multiple wires at once
Ctrl + Alt + LMB	Disconnect all wires from clicked slot

Ctrl can also be replaced with Cmd instead for macOS users

Jupyter Notebook

To run it on colab you can use my Jupyter Notebook