This commit introduces `fast_sampler.py`, a new module designed to enhance the performance of sampling and VAE decoding in ComfyUI. It replaces or augments functionality previously handled in `model_management.py`, providing better VRAM management, FP16 support, and tiled decoding for low-memory scenarios. Additionally, it improves the `PreviewImage` node in `nodes.py` for faster and more efficient preview generation. These changes improve efficiency, stability, and usability, particularly for GPU-based workflows.
**Key Changes:**
- Implemented `fast_ksampler` for optimized sampling with improved memory management, FP16 support via `torch.amp.autocast`, and `channels_last` memory format for better GPU performance.
- Added `fast_vae_decode` for efficient VAE decoding, incorporating FP16 support, `channels_last`, and selective VRAM clearing to prevent out-of-memory errors.
- Introduced `fast_vae_tiled_decode` for tiled VAE decoding, enabling processing of large latents on GPUs with limited VRAM by using configurable tile sizes and overlaps.
- Added profiling and debugging utilities (`profile_section`, `profile_cuda_sync`) to track execution times and VRAM usage when `--profile` or `--debug` flags are enabled.
- Improved VRAM management with `clear_vram`, ensuring sufficient free memory before loading models or VAE, with configurable thresholds and minimum free memory requirements.
- Implemented `is_fp16_safe` to check GPU compatibility for FP16 operations, disabling them on unsupported hardware (e.g., GTX 1660/Turing).
- Optimized tensor transfers with `optimized_transfer` and `optimized_conditioning` for synchronous device placement and dtype casting.
- Enhanced model preloading with `preload_model`, which unloads VAE before loading U-Net to conserve VRAM and checks for already-loaded VAE to avoid redundant transfers.
- Integrated `cudnn.benchmark` for for tests, disabled by default.
- VRAM should now be managed efficiently.
- Updated `PreviewImage` node in `nodes.py` to support adaptive resizing of preview images to a maximum dimension of ~512 pixels while preserving aspect ratio, using `Image.LANCZOS` for quality. Increased `compress_level` from 1 to 4 for faster PNG compression, optimizing preview generation.
**Impact:**
- Significantly reduces VRAM usage during sampling and VAE decoding, making ComfyUI more stable on GPUs with limited memory.
- Improves performance for large-scale image generation through tiled decoding and FP16 optimizations.
- Enhances debugging capabilities with detailed profiling and logging, aiding development and optimization.
**Dependencies:**
- Relies on `nodes.py` for integration with `KSampler`, `VAEDecode`, `VAEDecodeTiled`, and `PreviewImage` nodes.
- Assumes compatibility with existing `ModelPatcher` functionality for model patching (e.g., in `LoraLoader`).
**Notes:**
- Users should enable `--profile` or `--debug` flags to access detailed performance logs.
- FP16 support requires compatible GPU hardware (compute capability ≥ 8 or > 7).
- Tiled decoding parameters (`tile_size`, `overlap`, etc.) may need tuning for specific workflows.
- Preview images are now smaller and faster to generate, but users can adjust `max_size` in `PreviewImage` if higher resolution previews are needed.
This is a foundational change to improve ComfyUI's performance and scalability, particularly for resource-constrained environments.
Thanks to Grok @ xAI for help.
* Add basic support for videos as types
This PR adds support for VIDEO as first-class types. In order to avoid
unnecessary costs, VIDEO outputs must implement the `VideoInput` ABC,
but their implementation details can vary. Included are two
implementations of this type which can be returned by other nodes:
* `VideoFromFile` - Created with either a path on disk (as a string) or
a `io.BytesIO` containing the contents of a file in a supported format
(like .mp4). This implementation won't actually load the video unless
necessary. It will also avoid re-encoding when saving if possible.
* `VideoFromComponents` - Created from an image tensor and an optional
audio tensor.
Currently, only h264 encoded videos in .mp4 containers are supported for
saving, but the plan is to add additional encodings/containers in the
near future (particularly .webm).
* Add optimization to avoid parsing entire video
* Improve type declarations to reduce warnings
* Make sure bytesIO objects can be read many times
* Fix a potential issue when saving long videos
* Fix incorrect type annotation
* Add a `LoadVideo` node to make testing easier
* Refactor new types out of the base comfy folder
I've created a new `comfy_api` top-level module. The intention is that
anything within this folder would be covered by semver-style versioning
that would allow custom nodes to rely on them not introducing breaking
changes.
* Fix linting issue
* support 3d model filtering
* fix lint error: blank line contains whitespace
* add model extensions to test runner mimetype cache manually
* use unittest.mock.patch
* remove mtl file from testcase (actually plaintext support file)
Somehow managed to drop a file called "nul" into a windows checkpoints subdirectory. This caused all sorts of havoc with many nodes that needed the list of checkpoints.
* add 'is_default' to model paths config
including impl and doc in example file
* update weirdly overspecific test expectations
* oh there's two
* sigh
* Override user directory.
* Use overridden user directory.
* Remove prints.
* Remove references to global user_files.
* Remove unused replace_folder function.
* Remove newline.
* Remove global during get_user_directory.
* Add validation.
- Implemented a `try-except` block in the `recursive_search` function to handle `FileNotFoundError` gracefully.
- When encountering a file or directory path that cannot be accessed (causing `FileNotFoundError`), the code now logs a warning and skips processing for that specific path instead of crashing the node (CheckpointLoaderSimple was usually the first to break). This allows the rest of the directory traversal to proceed without interruption.
* wip per user data
* Rename, hide menu
* better error
rework default user
* store pretty
* Add userdata endpoints
Change nodetemplates to userdata
* add multi user message
* make normal arg
* Fix tests
* Ignore user dir
* user tests
* Changed to default to browser storage and add server-storage arg
* fix crash on empty templates
* fix settings added before load
* ignore parse errors
