Jedrzej Kosinski
0a86b5bc74
Merge branch 'master' into attention-select
2025-09-12 14:44:39 -07:00
comfyanonymous
33bd9ed9cb
Implement hunyuan image refiner model. ( #9817 )
2025-09-12 00:43:20 -04:00
Jedrzej Kosinski
f71feac11c
Merge branch 'master' into attention-select
2025-09-11 20:59:23 -07:00
comfyanonymous
b288fb0db8
Small refactor of some vae code. ( #9787 )
2025-09-09 18:09:56 -04:00
Jedrzej Kosinski
dd0a5093f6
Satisfy ruff
2025-08-30 14:58:30 -07:00
Jedrzej Kosinski
c092b8a4ac
Remove _register_core_attention_functions, as we wouldn't want someone to call that, just in case
2025-08-30 14:49:04 -07:00
Jedrzej Kosinski
eaa9433ff8
Remove attention logging code
2025-08-30 14:45:12 -07:00
Jedrzej Kosinski
cb959f9669
Add optimized to get_attention_function
2025-08-29 21:48:36 -07:00
Jedrzej Kosinski
2d13bf1c7a
Made SVD work with optimized_attention_override
2025-08-28 22:45:45 -07:00
Jedrzej Kosinski
0ac5c6344f
Made SD3 work with optimized_attention_override
2025-08-28 20:21:14 -07:00
Jedrzej Kosinski
48ed71caf8
Add logs to verify optimized_attention_override is passed all the way into attention function
2025-08-28 19:43:39 -07:00
Jedrzej Kosinski
51a30c2ad7
Make sure wrap_attn doesn't make itself recurse infinitely, attempt to load SageAttention and FlashAttention if not enabled so that they can be marked as available or not, create registry for available attention
2025-08-28 18:53:20 -07:00
Jedrzej Kosinski
669b9ef8e6
Added **kwargs to all attention functions so transformer_options could potentially be passed through
2025-08-28 13:14:41 -07:00
Jedrzej Kosinski
29b7990dc2
Fix memory usage issue with inspect
2025-08-27 17:55:35 -07:00
Jedrzej Kosinski
68b00e9c60
Created logging code for this branch so that it can be used to track down all the code paths where transformer_options would need to be added
2025-08-27 17:13:33 -07:00
Jedrzej Kosinski
b58db6934c
Looking into a @wrap_attn decorator to look for 'optimized_attention_override' entry in transformer_options
2025-08-27 14:18:18 -07:00
contentis
fe31ad0276
Add elementwise fusions ( #9495 )
...
* Add elementwise fusions
* Add addcmul pattern to Qwen
2025-08-22 19:39:15 -04:00
comfyanonymous
9df8792d4b
Make last PR not crash comfy on old pytorch. ( #9324 )
2025-08-13 15:12:41 -04:00
contentis
3da5a07510
SDPA backend priority ( #9299 )
2025-08-13 14:53:27 -04:00
chaObserv
61b08d4ba6
Replace manual x * sigmoid(x) with torch silu in VAE nonlinearity ( #9057 )
2025-07-30 19:25:56 -04:00
comfyanonymous
91d40086db
Fix pytorch warning. ( #8593 )
2025-06-19 11:04:52 -04:00
Kohaku-Blueleaf
520eb77b72
LoRA Trainer: LoRA training node in weight adapter scheme ( #8446 )
2025-06-13 19:25:59 -04:00
comfyanonymous
5a87757ef9
Better error if sageattention is installed but a dependency is missing. ( #8264 )
2025-05-24 06:43:12 -04:00
Raphael Walker
89e4ea0175
Add activations_shape info in UNet models ( #7482 )
...
* Add activations_shape info in UNet models
* activations_shape should be a list
2025-04-04 21:27:54 -04:00
comfyanonymous
e471c726e5
Fallback to pytorch attention if sage attention fails.
2025-03-22 15:45:56 -04:00
FeepingCreature
9c98c6358b
Tolerate missing @torch.library.custom_op ( #7234 )
...
This can happen on Pytorch versions older than 2.4.
2025-03-14 09:51:26 -04:00
FeepingCreature
7aceb9f91c
Add --use-flash-attention flag. ( #7223 )
...
* Add --use-flash-attention flag.
This is useful on AMD systems, as FA builds are still 10% faster than Pytorch cross-attention.
2025-03-14 03:22:41 -04:00
comfyanonymous
96d891cb94
Speedup on some models by not upcasting bfloat16 to float32 on mac.
2025-02-24 05:41:32 -05:00
comfyanonymous
aff16532d4
Remove some useless code.
2025-02-22 04:45:14 -05:00
comfyanonymous
1cd6cd6080
Disable pytorch attention in VAE for AMD.
2025-02-14 05:42:14 -05:00
comfyanonymous
e5ea112a90
Support Lumina 2 model.
2025-02-04 04:16:30 -05:00
Dr.Lt.Data
0a0df5f136
better guide message for sageattention ( #6634 )
2025-02-02 09:26:47 -05:00
comfyanonymous
96e2a45193
Remove useless code.
2025-01-23 05:56:23 -05:00
comfyanonymous
008761166f
Optimize first attention block in cosmos VAE.
2025-01-15 21:48:46 -05:00
comfyanonymous
129d8908f7
Add argument to skip the output reshaping in the attention functions.
2025-01-10 06:27:37 -05:00
comfyanonymous
79eea51a1d
Fix and enforce all ruff W rules.
2025-01-01 03:08:33 -05:00
comfyanonymous
d170292594
Remove some trailing white space.
2024-12-27 18:02:30 -05:00
Chenlei Hu
d7969cb070
Replace print with logging ( #6138 )
...
* Replace print with logging
* nit
* nit
* nit
* nit
* nit
* nit
2024-12-20 16:24:55 -05:00
comfyanonymous
cbbf077593
Small optimizations.
2024-12-18 18:23:28 -05:00
comfyanonymous
4c5c4ddeda
Fix regression in VAE code on old pytorch versions.
2024-12-18 03:08:28 -05:00
comfyanonymous
37e5390f5f
Add: --use-sage-attention to enable SageAttention.
...
You need to have the library installed first.
2024-12-18 01:56:10 -05:00
comfyanonymous
bda1482a27
Basic Hunyuan Video model support.
2024-12-16 19:35:40 -05:00
comfyanonymous
19ee5d9d8b
Don't expand mask when not necessary.
...
Expanding seems to slow down inference.
2024-12-16 18:22:50 -05:00
Raphael Walker
61b50720d0
Add support for attention masking in Flux ( #5942 )
...
* fix attention OOM in xformers
* allow passing attention mask in flux attention
* allow an attn_mask in flux
* attn masks can be done using replace patches instead of a separate dict
* fix return types
* fix return order
* enumerate
* patch the right keys
* arg names
* fix a silly bug
* fix xformers masks
* replace match with if, elif, else
* mask with image_ref_size
* remove unused import
* remove unused import 2
* fix pytorch/xformers attention
This corrects a weird inconsistency with skip_reshape.
It also allows masks of various shapes to be passed, which will be
automtically expanded (in a memory-efficient way) to a size that is
compatible with xformers or pytorch sdpa respectively.
* fix mask shapes
2024-12-16 18:21:17 -05:00
comfyanonymous
e83063bf24
Support conv3d in PatchEmbed.
2024-12-14 05:46:04 -05:00
Chenlei Hu
563291ee51
Enforce all pyflake lint rules ( #6033 )
...
* Enforce F821 undefined-name
* Enforce all pyflake lint rules
2024-12-12 19:29:37 -05:00
Chenlei Hu
2cddbf0821
Lint and fix undefined names (1/N) ( #6028 )
2024-12-12 18:55:26 -05:00
Chenlei Hu
60749f345d
Lint and fix undefined names (3/N) ( #6030 )
2024-12-12 18:49:40 -05:00
Chenlei Hu
d9d7f3c619
Lint all unused variables ( #5989 )
...
* Enable F841
* Autofix
* Remove all unused variable assignment
2024-12-12 17:59:16 -05:00
Chenlei Hu
0fd4e6c778
Lint unused import ( #5973 )
...
* Lint unused import
* nit
* Remove unused imports
* revert fix_torch import
* nit
2024-12-09 15:24:39 -05:00
